Hacker Newsnew | past | comments | ask | show | jobs | submit | fooey's commentslogin

starting to see some explanations for what's going on

https://www.wsj.com/us-news/faa-temporarily-halts-flights-ac...

> The outage appears to be affecting a telecommunications network overseen by L3Harris Technologies, according to people briefed on the matter. The network handles crucial radio frequencies that air-traffic controllers use to communicate with pilots, the people said. It wasn’t immediately clear whether the outage also affected the radar feeds that controllers use to keep aircraft safely separated in the skies.

https://www.cbsnews.com/news/faa-issues-nationwide-ground-st...

> An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage, multiple sources told CBS News. The system is operated by L3Harris, which did not immediately respond to CBS News' request for comment.


Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday? This would be extremely unusual in any software job I have ever had. For commercial aviation specifically which does have pretty well defined peaks and valleys in terms of traffic over the course of a day this seems like an odd time to choose

I did web dev work for an airline and we couldn't even release new updates to the loyalty benefits website during a weekday (even though there was no connection to operations or flights). The fact that this company would reboot a critical comms server at 3 pm on a Friday is indeed bizarre.

I believe not deploying on Friday is an anti-pattern for most software teams.

When the update requires downtime (or has such a high likelihood of downtime) that you'd be knowingly (or almost knowingly) inconveniencing users, do it off-hours (e.g. nights and weekends) and gather availability commitments from all the people you anticipate might be useful. If something goes so terribly sideways that it takes a long while to stabilize given the skeleton crew, at least you have plenty of time before users need it again. Friday evening maximizes that window.

Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.


> Friday evening maximizes that window.

For the FAA it sure as fuck doesn’t.


Unless you have god-tier CI/CD, which most places do not, I do not like Friday changes. I’ve worked at precisely one place where it didn’t bother me, and one place where they didn’t have CD, but shared my belief and didn’t deploy on Friday.

What industries have you seen teams deploy on a Friday afternoon?

We do for clients who have 24/7 users during the week.

I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.

We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.

It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.

Much better to get fast feedback while the deploy could be monitored against prod traffic.


We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.

I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.


or want to have to.

Why wouldn't you be able to rollback on a weekend like any other day?

cause I don't want to work on the weekend. Why would I introduce that risk?

Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.

Assuming they don’t have an actual life and aren’t somewhere touching grass.

I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.

Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.


Because you are patching a critical system that can't be down during the week.

In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.

Friday deployments make no sense if you have teams that work Monday-Friday.


Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend.

Several hours. That number goes up quickly when the bus number for something that broke unexpectedly is unavailable. While your coworker who actually knows what that error means is at the zoo with his girlfriend or with his kids.

The whole point of WLB is that we don’t live like monks.


Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation?

Outages are caused by change. Sometimes user traffic, but more often your coworkers.

Do you know how every feature toggle in your system works? Do you know the potential blast radius of those toggles? Can you speculate on the surprise surface area that might be additionally exposed by impedance mismatches and side effects (ie, unanticipated shared state)? Hell does every engineer who wrote a feature toggle know this about their own code? You know who I’m talking about.

If you’re having problems in production that can be anticipated, why are you still having those problems? How long should those be tolerated before someone gets fired?


We host a private cloud, so the only code we write are occasional shell scripts based on repeatable steps that have already been done manually hundreds or even thousands of times. And yeah, everyone on the team has a pretty thorough knowledge of the applications we host. Documentation is usually enough to fill the blanks.

I'm not saying surprises don't happen, just that they should be extraordinarily rare.


I think when we are discussing a very public outage though, we can take it as read that there are any number of problems with the company and this is just the one they got caught out on. And similarly that a lot of replies are from people who have experienced the same with other businesses and/or employers.

As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.


Thursday is pretty close to the weekend. Too risky on Thursday?

Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.

I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.

Or just have a good rollback plan like a snapshot.

If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.

And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.

Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.

If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.

Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.


So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.

Me and my colleagues don't want to work weekends, so that goes into the planning of deployments. There is always a possibility of failure when releasing software. I don't see the "how much revenue" relevant to what I wrote - if we deploy Friday and it breaks, we need to work weekend.

We don't WANT to work on weekends either. That's why it is extraordinarily rare that we do. Plus the fact that most clients have a Dev environment that we test the changes in first.

Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.


I don't know, it feels like you're arguing based on your knowledge about the system you work on, without taking into account that other systems does not work the same. It's just nonsense. Good luck.

Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.

It’s ok, claude can do it for you.

You can, but it also means you might have to.

Not a family man, I see?

That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.

  >I don't like working weekend [sic] anymore [sic] that [sic] anyone else.
So why the feigned confusion above? It seems you already knew the answer to your own 'question.' :P

My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary.

We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays.

The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.


There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes.

Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS.

It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.


Actually, we ALL work on call. No one gets out of it just because they have seniority. Plus, we have an excellent team of smart people who are able to figure most things out on their own through use of their own knowledge and troubleshooting skills, aided by both internal and external documentation. It is rare that escalation is required.

I never said things don't go to hell sometimes. That's why you build a good team and train your people.


I think it's not just inconvenience for users but also availability of engineers to look at failures.

Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.


So... it's 24/5, then.

Yes pendant

It happens. My company has done a couple dozen production rollouts today, including minutes ago.

In public utility we only did deployments on Fridays after 5PM

There's a bifurcation in best practices.

For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance."

For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert."


Team America: World Police

Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it.

Only if the team wants to work weekends?

If something goes down on Saturday there’s a good chance I’m out of cell service


It is interesting to see this bifurcation in action.

Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off.

If not... Has anyone told y'all you should unionize?


Many people work shifts, it’s quite normal.

What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.

Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!


I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)?

The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.


Did you know that when you use your 2 days off to go shopping, someone has to work at the shops?

Yes. That's why people work staggered schedules. Folks who work weekends at shops usually (and should) have two other days off in the week.

People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us.


And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours.

Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities).

Wanting to have a weekend is smug and privileged now?

People fought hard for the 40 hour work week and 2 day weekend.

You think that should be given up?


Oh, I know all about that history. Having been a volunteer for Bernie Sanders back in 2015, I admired him as being one of the remaining few people who actually remembered this history due to him being part of the Silent Generation and continue to fight for it decades later.

>You think that should be given up?

Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.


I really don’t understand how you got that from my post.

I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage.

Are you saying labor conditions need to get worse for everyone in order to get better?

If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends


IT in corporate is actually more panicked than IT in life safety, because life safety invests more redundant design.

the update caused the reboot but could that mean the reboot happened due to a prior update? or literally the reboot was FOR the update hmm

Windows Update decided to reboot automatically because somebody didn't press the skip button enough time?

> we couldn't even release new updates during a weekday

so you were forced to release updates on weekends? isn't that worse?


It depends on the schedules of the people who'll have to fix it if something goes wrong. I don't mind working weekends, so when I was in the sysadmin game I always tried to make significant changes late friday/early saturday, so if something went wrong I had the weekend to fix it & clean up. But if the people needed to fix a problem take weekends off, pretty much the opposite schedule applies.

Flights generally peak Mon-Thu due to business flights, weekends are usually lighter unless it's a long holiday weekend.

Worse for the employees. Better for the clients.

"3 pm on a Friday is indeed bizarre"

Most likely, they HAD to do it.


Probably was scheduled for Thursday evening but was delayed due to the Outlook issues.

It should be fine to update at any moment. Why? Because systems like this ought to be hot spare or blue/green models that have a full working copy at any given time, and a failover/failback.

A reasonable deployment plan considers the reality that the deployment could fail. One mitigation is to choose ideal conditions for a possible failure. You’re correctly pointing out one of the mitigations, but it doesn’t render the others unnecessary.

You can also have reasonable updates done at reasonable times that don’t show issues until unreasonable hours - or something else fails.

But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.


As opposed to what alternative? Being run by the FAA itself or with contractors? Being run by the DOT?

Well, it worked so far, I think it is great. 100% availability is a dream.

That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.


Yeah either the FAA or the DOT sounds good. Not contractors tho.

you seriously think the government would do a better job?

Yes, actually. They're (until recently...) more likely to be there longer, and have a better understanding of how things work and who to talk to.

No. I think the fact that the government wouldn't do a better job is a skill issue and the government has to get good. But also this government where it does farm out to contractors and monopolizte for every little thing could never build the software itself. Even saying "the government makes it itself" is saying we fundamentally have a government tha is different than right now.

It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.


This is a narrow and shortsighted take. Government would absolutely do a better job if only for the SOPs around redundancy that business is famously flexible on (read: happy to ignore).

the government would do a better job than a private company with incentives to maximize profits.

I mean they also make the CPU for the F-35, alongside kinetic strike and electronic warfare missiles, so they could cause plenty of damage if they wanted to. This isn't some mom-and-pop radio company.

> But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.

Foreign cyber-adversaries (the usual suspects ;) are taking notes.


I mean sure and we have that in finance but there’s no amount of time saved that could make me deploy something during market hours

Airspace certainly has busy and quiet(er) periods, but the skies are open all day every day.

There are obviously crests and troughs in flight volume (as measured in passenger-hours), 3pm on a Friday seems pretty cresty to me.

> 3pm on a Friday seems pretty cresty to me

It wouldn't be my pick, that's for sure.


Well clearly they don't have a hot spare or blue/green models, or working failover.

Which is why deploying at 3pm on a Friday is even more bizarre.

Sure, but if it fails and a fallback takes over, will you be going home and having a nice weekend? Or will you be stuck at work over the weekend to fix it?

I'd much rather the team's weekend be impacted than millions of travelers.

Why not do a deploy on a monday morning, and impact the teams workday instead of their weekend?

What if you could avoid the team impact and not screw millions of travelers.

Mission critical systems that are in use 24/7 are not managed based on week day vs weekend shifts. Or at least shouldn't be.

My (hopeful) assumption is that air traffic control isn't based on the "weekends are the off hours" mindset.


Maybe patching a critical vulnerability? I’m also at a loss for why they chose Friday afternoon.

That could be their least busy day, for all we know. Weekends are busier than weekdays for airline travel.

Friday and Sunday are the busiest days. Business and pleasure overlap. Sat is mostly pleasure. Mon-Thu are mostly business.

Tuesday and Wednesday are apparently the least busy days, at average -10% traffic each.

So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)


But 3am is when a big chunk of planes are parked at gates awaiting the start of the flight day (yes, there are red-eye flights and international but they are a minority.)

Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.


9pm is likely better. if you patch at 3am there's a decent chance no one notices until 6 and then you ruin the next day.

It's also very possible they patched last night since they were possible also updating systems for the big AI rollout and then the issues didn't cascade or cause issues until this afternoon

I'd wager management wanted this by the EoW

Arbitrary day-of-the-week and date-based deadlines piss me off so much.

I worked for a person who had a fixation on "neat" deadines: the "first of the month", "the end of the week", "the start of the quarter", etc. It'd be one thing if the dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case. They were chosen because they were "tidy".

I'd time meetings looking to maximize time for work. You try to meet late in the day Wednesday because an early Tuesday meeting might get an "end of the week" deadline. A late Wednesday meeting was more likely to garner a "end of next week" deadline. Meeting Thursday or Friday guaranteed an "end of next week" but you lost Thursday and Friday work time. Wednesday was a gamble, though, because it was still early enough to get you an "end of the week". Meeting after lunch helped with that.

You always had to be aware of the numeric calendar day, though. Gunning for an "end of next week" deadline when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.

All of this deadline-setting happened with little regard for the amount of time work was going to take.

It was immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.


"The project will land on the first full moon after the vernal equinox"

Weird how all epic stupid decisions come from the people who are supposedly the trained 'decisions makers.'

I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."


RCA could be in just 2 words: incompetent people

"More testing? A more gradual phased rollout? We need to get this out! We need a win!"


A security patch would be a reason not to wait.

And because a security patch can't wait no deployment should wait because it's Friday. That's how you stay ready; you're always ready.

Rolling out updates on a Friday afternoon is still practiced in various industries.

> Rolling out updates on a Friday afternoon is still practiced in various industries.

By people with unreliable systems and a lot of myhts behind.

In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.


They're just now completely phasing out floppy disks. By 2028. Guessing that's not the type of org that deploys frequently !

It does not matter when they did the updates. What matters now is what the source code is for the code that they deployed. We need to see what the documentation says about human reviews, testing, and certification processes.

If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.

This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.


Perhaps.

But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.

My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".

I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.


> It took a few weeks for most Americans to catch on

In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.


My money is on DNS, but if it is enemy action, will the FAA even know that's what it is?

> Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday

As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.

(Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.)


> As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.

Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?


> I would surmise the technical reason for it is that they didn't want to work over the weekend.

Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.


There is also a real possibility they did this many times before and it worked. I don’t really get it: if the thing would have broken on a Monday night and it would take a day to fix would that have been better?

The odds of someone screwing something up just before the weekend are way higher. Both because it’s obviously a rush job (it can’t wait until Monday), and because it’s the end of the week and folks are tired.

It’s just dumb.


That's why you don't deploy on a Friday. :)

Yesterday email went down for half the world's population because MS pushed a faulty update and a day later all US planes are grounded also because of a faulty update, come on what are the chances.

If it was a malicious act, or maybe even two, would they publicly state it was the case ?


It's worse than that. It's 3pm on a Friday before Thanksgiving week.

I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.


No it's not. Thanksgiving is on November 26.

So yes, it is a Friday before thanksgiving week. Two weeks before.

If you don’t let juniors ship to prod on Friday afternoons sometimes they will never get their wings.

Nothing like a good Friday afternoon did lot to ruin a weekend.

This is why you don't push PROD on fridays...

Did anyone question why Microsoft would do an update at 10am on a Thursday?

Seeing L3Harris failing gives me a little bit of anxiety since their systems run some of the backend of some nuclear operations.

These systems fail safely. In the meanwhile, millions of people worldwide die of air-pollution of the fossil fuel industry but they don't even get a 100th of the scrutiny that nuclear industry gets.

This is not a critique of nuclear power. It is simply a (mildly existential) realization that even with high reliability goals, failure is still possible.

I expect the teams developing ATC communications and the team maintaining the CANDU DCC to be distinct and not have many dependencies.


Huh. I thought Collins now ran what used to be Aeronautical Radio, Inc. (ARINC), which is the ground portion of air traffic control radio. L3Harris has the radars and ADS-B.[1]

[1] https://www.l3harris.com/sites/default/files/2021-08/MN_SBSS...


Collins/ARINC is used for oceanic HF radio. They run "San Francisco Radio" and "New York Radio" specifically, which are the HF radio operators that relay oceanic messages to ATC.

(They also own a small number of private VHF stations for relaying company messages to air carrier operations domestically, but these aren't used for ATC purposes.)

Otherwise, the VHF radio links are operated directly by FAA TechOps.

L3Harris owns the telecommunications backhaul between facilities.


I bet it was an issue with BGP.

Rollout new router config Thursday night - seems fine at first - issues compound and then remote admin goes down


Yes, I doubt someone pushed out an update minutes before the outage. The hard bugs are the ones that take a long time to impact a system and don't always appear in testing.

Doesn't L3 Harris use Microsoft Azure Government for most of their mission critical infrastructure? Are we looking at fallout from the Microsoft infrastructure collapse yesterday?

Apparently the L3Harris CEO was ousted in August in a Jensen-like scandal. It's also the company that was modifying Trump's Qatar corruption plane.

https://www.cnbc.com/2026/08/17/l3harris-ceo-chris-kubasik-l...


Didn't they get bought it merge with some other giant company ?

[flagged]


There's no evidence that AI chips were sold to China. The chips were sold to Singapore. I have yet to see confirmation the chips were subsequently smuggled out of Singapore with Jensen's knowledge.

Gamers Nexus had some interesting coverage on the whole situation so far: https://www.youtube.com/watch?v=YYU_geUwQoA

The Nvidia chips that the Russian drones are using are consumer class chips (TE980M-A1) that have no export controls. It's odd that this channel decided to highlight that when it's very clear those are not on export control lists only the high level AI chips are restricted and controlled in this way. Those chips were sold to any country that has open trade with Russia then they could easily make their way to Russia from second-hand markets.

So from that alone I don't trust the integrity of this source's reporting.

You'll find Intel, AMD, and every other name brand of CPU/GPU in all countries on earth, that doesn't make those companies culpable.


Fair enough. That part was pretty short, the rest of the video making connections between the money and the companies was fairly convincing to me that something doesn't smell right.

>You mean he facilitated sale of restricted AI chips to China?

No, they mean "had a sexual relationship with a subordinate".

Which is what the linked article says.

>Sorry, but I am tired of the effort to make it look like Jansen's crime was infidelity.

Strawman. Nobody says that, and nobody cares.

Jensen's "crimes" that we know of are:

1. Having a sexual relationship with a person suspected of being a Chinese spy (who, allegedly, influenced NVIDIA's chip sales to China)

2. Having an undisclosed sexual relationship with a direct report

Both are huge liabilities for the company, as well as national security concerns (someone in a critical position putting themselves into a position for easy blackmail).

None are crimes, all are factors contributing to the firing decision.

>If he just went to whore house he would be fine (well, ignoring a possible expensive divorce).

Same goes for the ousted L3Harris CEO.


The number of creepy men supporting infidelity on HN amazes me.

Not sure who you meant, if it was me, I didn't say infidelity was moral thing.

I'm pointing out that looks like there is an effort to kind of turning this scandal as being about infidelity and then in next step conclude that infidelity is not illegal so "Jensen didn't do anything wrong".

The infidelity is bad and typically is connected with character, so somebody who does that is very likely unreliable in other ways as we can see.

Anyway the thing is that talking about infidelity is muddling what actually happened. The US considers this technology to be important for national security and now we are learning that the chips which supposed to be restricted were sold and are in China.

To illustrate the scandal, let's replace the chips with for example F-35 parts (which ironically there was already another scandal a while ago).


Could you be more specific about the comments you’re referencing? I don’t see anything in this thread, at least at a cursory glance, that could reasonably be described this way.

Did you somehow see defense of infidelity in my comment (hope not, because it wasn't there), or are you reflecting on the general trend?

Who do you feel is creepy here (or a man, for that matter)?

Infidelity (in itself) is not the thing that got either of those 2 men fired.

That statement doesn't say anything about how anyone here feels about infidelity, nor whether it should or shouldn't be enough of a reason to fire a CEO.

I don't see anyone supporting infidelity here.

However, in a thread about Jensen, I've seen multiple people argue with me that the CEO having sex with subordinates isn't problematic at all (with total disregard for the power imbalance and potential for coercion).

That was, indeed, very sad.


How is there not a second backup network???

Cue that line from DS9 where O’Brien says he’d never be caught without a secondary backup during a fight, and the cardassian is like “you have a second backup?” incredulously. https://youtu.be/o7o9ZtipVyo?is=y3for_WXwISmuqpd

I wouldn’t say anything but this is approximately the only time in my entire life this scene will be relevant.


I remember that scene, but I can’t believe that you had it at the front of your memory like that. That’s awesome.

There are people who literally live for this shit, you know. I have friends who have puns lined up for 1000+ situations.

The more time I spend working in tech, the more I realize that everything important is held together with cardboard and duct tape.

That's because everything really important is not just tech but a socio-technical system of tech+humans operating/building/maintaining that thing. A lot of the things can't/shouldn't be automated with tech (see the Irony of Automations paper) – they are up and running thanks to the resilience (socio-technical quality) which appears as a "cardboard and duct tape" for untrained eye.

True for every industry / field I've worked in, unfortunately (which is more than I'd like to admit).

You had duct tape? Lucky!

Any time people freak out about these kinds of things and assume the absolute worse, I just tell them I’m surprised these things don’t happen more often

Why only “one” network at all!?

The Internet was purpose designed to tolerate disruption as major as a nuclear war, which it achieves through decentralisation.

Private networks exist to make idiot managers feel more confident while actually dramatically reducing availability by putting all their eggs into a single basket.


Many aviation systems predate the internet. Depending on the region, some systems do not use Internet Protocol, let alone TCP.

The FAA Telecommunications Infrastructure (FTI) operated by L3harris was built between 2002 to 2007.

That is already well within the era of IPv6, let alone "the Internet".

There's no excuse.


For something so incredibly critical in real time, I’m not sure I would want to trust the Internet.

When was the last time the Internet "went down" globally?

The last time the internet caused flight outages was September 22nd of this year.

Over what area? Localised, nation-wide, or global?

thats like mandating using I-95 for all ambulance & firefighter traffic

Isn't using a private network like mandating using I-95 (since it might be blocked from a freak accident one day), while using the internet, with redundant paths between hosts, like allowing the traffic to find the best path?

The internet is pretty unreliable as it goes, and the internet in your locality is probably only one network (e.g. Comcast's) - what are you gaining by running critical systems over Comcast internet access, instead of Comcast leased line? Saving a few hundred bucks a month, yes, but that's because you're getting less reliability than you actually need.

Also lower MTTR and more robust SLAs in many cases. Business services will usually result in service credits being issued if the SLO is breached. Residential... not so much.

One network can be a fully redundant design with multiple geographically distinct links and so on.

But it does not make it 100% resilient to cascading failures or possible Layer2/3 loops that might happen during a software upgrade, especially in a country wide deployment.

The FAA will give a root cause at some point of why this happenned.


> caused the system to reboot, leading to a crash and a communications outage

why a reboot would cause a crash? or it means that an update irreversibly broke the system and it was rebooted but that didn't help?


Yeah - like, why do they not have an "HA" cluster, where they patch and reboot a subset at a time... Something this critical should have some sort of failover/load-balancing in-play...

Maybe the systems connected to it are not properly designed to handle such a reboot

"What is Microsoft Azure?"

>An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage

And that's why you never deploy to prod on a Friday.


Especially Friday the 13th.

well is there a rollback procedure?

it could only be deployed from the servers that crashed

and they said "wont deploy on friday? thats 20% of the week in lost productivity!"

I have memories of my professor saying something in the lines of

"it's computer science, if you make a mistake nobody will die!

Unless you're dealing with nuclear plants, medicals, or airlines"


Thank you.

I learned more from this post in 15sec than I did in 20min from watching the Sec of DHS and Sec of Travel press conference


> leading to a crash

poor choice of words


This indeed is a poor choice of words…

Leading to a crash is very unfortunate wording, when talking about systems for aeroplanes.

Who updates on Friday?

People who have a testing window Saturday morning, which is how a lot of legacy industries with cyclical schedules work. Weekends are slower air travel days.

Organizations that have good engineering practices.

This woodoo like belief you should not deploy on Fridays, Christmas, Valentine’s Day and other magical days is a disservice to the industry.


But when it goes wrong you’re working on the weekend.

I think it also creates a tension because people are more compelled to "just ship it" to go home. If it's a Monday and it's not quite ready yet or someone has doubts, more likely to just hold off.

I feel like people want to go home on Friday and will make sacrifices they would not make on a Wednesday.


When things go wrong, you work to fix them. Doesn't really help if it is the weekend or a very long Monday.

Things go wrong all the time, it is not a reason to fear doing stuff.


Isn’t Friday usually a light travel day?

No, very heavy travel day for business.

No. Business and pleasure overlap, making it a busy day, along with Sunday.

A junior engineer according to my table: intern: break the build and go home. junior engineer: break the build and go home for the weekend. staff engineer: break the build and go on vacation. senior engineer: break the build and retire.

There are no weekends in the airline industry, in case you didn't know.

Everything operates normally, as any other day.


There are early mornings and nights. Air traffic slows down significantly past midnight.

Air traffic slows, yes, but it doesn't stop. ATC operates 24/7/365. Whether you have one plane or one thousand, the entire system needs to be operable.

It's basic risk mitigation.

I agree that you should do everything possible to make sure there's no downtime. But if there is a time for downtime, it's when there's the least amount of air traffic.


Screwing up big time in the dark might be worse than doing it in daylight.

There are still ways to roll out updates sensibly and safely in a 24/7 industry

no weekends for blue collars, but why would there be no for white collars?

I wonder if the update was signed with a cert chaining to a CA with a private key of "4".

the lesson congress learned from the Hillary Clinton coronation getting derailed by Obama was to absolutely never ever vote on anything important


Can you elaborate what you mean?


In 2002, Hillary Clinton voted for the Iraq War. In 2008 Obama used that fact successfully to [help] win the Democratic nomination and went on to beat McCain.


absolutely nothing in going to happen in the US congress until 2029

we're essentially one year of three into one of the least effective lame duck legislative bodies we'll ever see


I think owning GitHub is a prestige play, not necessarily thought of as a profit center.

Now however, it's competing for resources directly with the corporations core strategic plans.

GitHub having issues is embarrassing, but the shareholders don't really care.


I've always wondered about that. I've suspected the same as you, it's mostly a brand / "ecosystem" thing.


When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all.

The AI boom caught them napping though, and they're slipping well into the too unreliable category


> The AI boom caught them napping though

Microsoft, who invested eyewatering sums into AI, even near to betting the whole company on an AI boom, is surprised about the AI boom?


Microsoft bet everything on possible AGI, no one thought the result would be a bonafide commit printing machine.


That's not what Nadella said 1-2 years ago. Everyone seemed pretty aware of LLMs soon being able to handle a lot of steps of software development.

I bet it's mostly due to Azure being brittle and the AI boom is just a nice cover story (not saying it's not challenging to handle all that new load) to save face and sell more Azure services.


100% if this post (https://isolveproblems.substack.com/p/how-microsoft-vaporize...) is anything to go by, it could explain much of the scaling & reliability issues.Perhaps github not being built to scale this fast + being hosted on azure is closer to the truth.


PRRRR


Microsoft considers it a bonus if a hype lets one department cripple another department.


cloudflares lack of spending caps and and usage alerts is a seriously unnerving approach


I shutdown all my little informational hobby projects this year that I've tinkered with for decades

They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes

I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots


the Tesla standard is 5 seconds, NHTSA says 30 seconds

Luckily, none of the Tesla data is available because they're currently getting away with redacting so much in their reports they're essentially useless


Complete coincidence that Elon cut the funding for the department that was investigating Tesla: https://techcrunch.com/2025/04/10/nhtsas-self-driving-safety...


I expect a flavor of affiliate marketing where you can never trust if the LLM is giving you the best recommendations or the most highly bidded recommendations


"Since we're on the topic of DIY car repairs, did you know Autozone carries a wide variety blorpity slop?"


The proposed build out will consume 9gw of electricity, all LNG fueled

The entire State of Utah, *combined* consumes less than half that much energy.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: