> since public cloud became very popular, the profile of infrastructure programmers changed. A lot of knowledge necessary to run own infrastructure disappeared, being replaced by the knowledge of how to operate the service.

It was ever thus. Previously it was CISCO Solution Specialists and HP/UX Most Valuable Engineers and the like, and all the knowledge beyond the basics was completely tied to the vendor. In enterprise IT, it's still like that for Microsoft.

If I'm running on the cloud why do I care how PXE boot works? How is that helping me? Yes it's lost knowledge/skill areas but it's also not needed.

Similar, I forgot how to config HA proxy and setting up Nginx would take me a few days to re-learn/optimize but I don't need those skills anymore. Just like I don't need to know how to install oracle/pgsql from scratch, setup raid on my computer, compile the kernel, etc to do my day job.

I'm also not debugging the assembly coming out of the compiler anymore looking for compiler bugs (the one time I've done that in my 25 year career, I just pinged one of the eng who worked heavily in that area and he filed a bug to sony 10 minutes later and told me to just add some int ++ / -- and that shifted the compiler bug out of the way. Sony fixed it a month later.

Nor am I a wizzard at add/remove drivers from win98 or setting up a token ring lan.

Germane knowledge comes, and goes. My job isn't to know PXE boot or any other specific tech, it's to ship product...

I think you missed the point where they were saying most infra dev might not know how to do on prem and cloud so it becomes less of a choice to use cloud if cloud skills are the only skills in your org.

That's a very good question. Well, maybe PXE specifically isn't such a great example, but still, this is a good question because the answer isn't obvious.

First, cloud vendors don't really invent the infrastructure from zero. They typically adopt existing solutions to their needs. And they aren't infallible. Every now and then there'll be a bug or an edge case they didn't handle. Alternatively, there will be weird restrictions that you, the customer, would have to go fish for in substantial volume of quite opaque documentation in order to protect yourself from encountering those unfortunate edge cases.

I don't know if AWS internally uses PXE boot. But, suppose they did: this would mean that there'd have to be some specific details of the network configuration that enabled the use of PXE. Also, there could be some pathological cases of interaction between "special" (boot, bootloader) partitions and PXE boot. You wouldn't normally care in VM-based deployment about boot partition as some VM players can even offer to boot your kernel w/o you having a bootloader in the image, so, you may go for a long time w/o ever discovering what your bootloader is configured to do and how it may interact with PXE boot.

Suppose at some point AWS upgrades something in their PXE server / code it runs on clients... and it breaks your stuff. Because this would affect most likely all of your VMs, the effect may be devastating...

But... you didn't even know PXE boot existed. You had no idea that you had to look for some subsection X.Y.Z in the appendix to the user manual that warned you against using a particular configuration in your bootloader, which stayed dormant for a long time.

It's not hard to imagine how this might go financially...

Second, any solution purports to sell to many customers -- the more the better, that's how the solution becomes profitable. Naturally, the solution provider wants to find as homogeneous group of customers as possible, so that with as little effort possible they'd be able to cover most customers. But the reality tends to work against this desire by creating customers who aren't like each other. So, some generalization must happen, and it means that very specific, very individual desires of customers will likely not be covered by the service provider. Now, suppose you, the client, know how the underlying technology works, and thus are able to defeat the unwillingness of the provider to accommodate your very specific needs... or, you don't. Typically, this ends up being not a deal-breaker, but an inconvenience. Often times it's the money you'll pay for the service you don't need, or the resource consumption that could've been avoided.

I cannot think about a good example with PXE, but here's something I crossed paths with recently at my kid's birthday party: I met a guy who works at Gitlab, and that reminded me about a grudge I had against Gitlab for some time. So, Gitlab is an example of service that replaces in-house IT / Ops who'd manage company's repository in this case.

Gitlab, at least initially, gained popularity due to CI bundled into service. This CI knows how to run jobs. These jobs produce logs. The logs can be obtained through API, if necessary. But... if you do this at the time the job hasn't finished, there's no API call to enable paging of logs. So, if you want to display logs as they are being generated, you poll, and each subsequent response will get you 0..N characters, with N ever increasing. This is hugely wasteful, and, in part the reason why their logging configuration puts a hard limit on log size... Well, bottom line, their logging API is bad. But, you won't abandon the service because the logging API is bad. You'll suck it up, especially since they give you a bunch of free stuff... You'll just pay a bit extra, if you don't need the free stuff.

With the PXE boot issue... amazon has millions of servers and they roll changes out to hardware on a 5 year cycle or versions/skus of the compute. So anything hardware wise will be detectable when your software doesn't work on the new instance type/sku/class. I've had this happen, multiple times over 10 years, and always just canary out new instance types, in fact I setup our systems to make this simple. But you can easily try it out and hand the hardware back if it doesn't work. We reported the issues every time. I believe AWS scrapped one sku due to an issue we reported. They also pulled back skus from us because another customer realized they were broken in a away that would impact everyone eventually.

When it comes to software amazon again has millions of machines, regions, availability zones. AWS is very very slow and deliberate about how they roll out software. They do things by Sub AZ (shard, cell), AZ, Region... they can and do split on other axes as well. Rolling out a PXE boot change would be caught by increase in failed boot or under utilized instances (they didn't boot right) pretty quickly in a roll out and rolled back.

All of this is aided by many may of AWS's customers treating servers like cows not pets, so whacking and restarting a small % of hosts isn't super detrimental and likely won't even be noticed. AWS can be extra cruel to internal teams actually. We have have a few concepts internally that push users to be even more ephemeral than normal.

As for how do people learn PXE boot? Experience or on the job training. I have no idea if AWS uses PXE boot anymore, I suspect they have a specialized setup, boot vms directly, etc. There's hardware security with the hypervisors you don't get on normal machines. I PXE booted a linux desktop the other day in the office to install linux so the corp side does use it.

I'm not really sure what the point you are getting at is tho. The cloud providers are doing things on a scale most people are not used to thinking of and the problems are often quite different as are the solutions. Are you rolling your own custom hardware with FoxCon (https://www.importgenius.com/suppliers/foxconn-aws)? Do you roll your own custom silicon (https://www.amazon.jobs/en/landing_pages/annapurna%20labs)? Would you be solving PXE boot issues if you had these capabilities?

What happened to all the engineers who knew these things before? Was this so long ago that they have all retired and died off?

Or is it that there are orders of magnitude more internet-based companies today, all competing for the same talent (which, indeed, is a limited skillset because these are not generally applicable skills). How many of these new companies exist _only_ because cloud infra is available now?

The industry grew a lot. I don't have the recent numbers but around the time AWS became a household name the growth in the industry was about 5% yearly. Programming was one of the fastest growing industries for a while (probably not true with recent wave of layoffs, but it's probably still growing). But it grows very non-uniformly. There's a lot more growth in the Web sector. Web is one of the areas with more simplified infra, which can be catered for by the public cloud providers more easily. So, while there's growing demand it's satisfied by services rather than employees knowledge.

Another factor is promotion into management. Programmers have very little to look for in terms of professional growth. Those who "make a career" tend to go into management. It's so to the point that if you were in the industry for 20 years and you aren't a head of a department at least, you are seen as a loser. So, a lot of those who did know their trade decade or two ago are ticking boxes in Excel today.

Yet another aspect is the nature of venture capital investment into startups which encourages very wasteful short-term solutions. So, while it may be a more financially sound strategy to do things on-prem, the investments are rationed in such a way that there's no money to build the infrastructure upfront, even if in the long run this would've been a better financial strategy.

Is there some institutional barrier to training new hires? It seems like the inevitable result if you need an increasingly niche skill. The situation doesn't seem like it'll get better on its own.

I work in an infra department, which used to be its own company before it was bought by a bigger one. Out of about 20 developers, one is younger than 30. Most are in their 40s and some are 50+.

I know that when I was hired, my job was advertised for over a year, if not longer. They pay may not be stellar, but it's still pretty decent...

In my previous position, I was also responsible for screening and hiring, so I also had some insight into what's available on the job market. My impression is that the younger generation simply doesn't even know this kind of job exists. Those who go into programming aspire to be a Web developer, or an ML developer, or a game developer... DevOps is one of those things people have heard about. But, this label was coopted by cloud some time ago. So, those aiming for DevOps role think they need to learn how to use public cloud API...

It's actually quite eye-opening to have access to the hiring process. One of the very surprising things to me was that newer generation of programmers tends to be confused about persistent / volatile storage. So, if you asked them to give a description of what a "file" is, more than once I've heard an answer like "a portion of memory". Which, while true in some sense, is a weird way to put it. But then I realized that people who are entering the field today may not have been exposed to PCs. Naively, if anyone asked me to show where the files in the computer live, I'd get them a picture of a hard drive. But people who only ever used laptops probably don't even know what a hard drive looks like.

Finally, it's also a chicken and egg problem. You cannot really learn about infra outside of a big org that has its own physical infra. Similar to how you'd never be exposed to configuring BGP if all your exposure to computers is your own PC. A popular method of learning today is to learn online. But, if you do that -- you are almost inevitably driven to the public cloud, because that's the way to provide you with all that virtual infrastructure online. You cannot also practice at home, since the hardware to practice on tends to be very expensive and not very useful for individual users.

Other fields get it easier because they have to select from somewhat prepared candidates. When it comes to infra, it's a lot more investment into teaching one.

Train by who? How do you start this process? If you never had the expertise, or more unfortunately had it but lost it, it's very expensive to gain it.

This seems to be a growing problem across the economy. Companies weren't willing to train for so long that now not only is there no one to hire who knows how to do jobs, there's no one left to train people to do them.