FAA issues national ground stop – lifted 4.5 hours later
www.nbcnews.comhttps://www.fly.faa.gov/adv/adv_otherdis?adv_date=11132026&a...
https://www.fly.faa.gov/adv/adv_otherdis?adv_date=11142026&a...
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
For those interested, there was a 2009 outage of the FAA Telecommunications Infrastructure that at a surface level seems like it could be a similar failure mode. At the very least, it is an interesting read on how the FAAs systems operate, since a lot of the discussion so far has been wild speculation from people with little knowledge on how the national airspace system actually operates: https://www.oig.dot.gov/library-item/29670
Of particular note is this section: Risks of FTI Outages for Existing Operational Services As of December 2009, Harris reported that there are a total of 20,982 services operating on the FTI network. While FAA has transitioned all 1,808 IP services supporting FAA flight plans to the new FTI fiber optic network, additional risks will be introduced when the Agency begins transitioning other FAA services, such as En Route Automation Modernization (ERAM) and NextGen platforms, which also require IP technology to operate. Moreover, there are 19,174 existing services that may be transitioned to the new FTI fiber optic network. According to FAA, transitioning these remaining services is important because FTI now connects older FAA systems that provide safety-critical voice and surveillance radar information by utilizing Time Division Multiplex (TDM) technology, instead of IP services.
TBH TDM is far superior and it’s sad that the world discarded it. Packet switching is the worst thing to ever happen to phone calls. Kids today have no idea how good it used to be (albeit at $0.50/min).
I mean the fidelity of modern mics/converters/codecs in devices now is much better, but ya the tradeoff of VoIP is latency/jitter/dropouts etc
TDM only implies (in the traditional 8-bit framing format, at least) that you're giving a 64 kbps timeslot to a caller. You can put any arbitrary payload in there, like AAC, Opus, or whatever. Preferably a codec without an intrinsic lookahead delay though. You can bond multiple 64k timeslots too.
Glad someone else appreciates circuit switched technologies though :) . It really does shine in any application that isn't burst-y data transfers.
has there been work on good real time audio codecs? I wonder how good you can do if you limit yourself to a latency of <1 sample?
Professional audio works in profiles down to 125us packets, so at 48khz that’s 6 samples per packet. For IP/Ethernet that means 18 byte payload translating to 76 byte Ethernet after the various headers, and another 20 bytes of preamble. That’s 800ns at 1gbit.
if you’re using IPv6 (nobody does) that increases another 20 bytes.
You’d typically have a receive buffer in the 1ms range though , although in theory you’d get away with 500us or even 375us on paper.
I believe (not really my area, in video we deal with timings starting in the tens in the tens of microseconds, and quantisation in the milliseconds) the rule of thumb is the total system latency between microphone and speaker have to be all within 5ms to avoid problems
Exactly this but it is still running on the age old RTP stack with raw PCM packets.
Using FPGAs you can get to very low latency.
Also at some point the latency is a non-issue.
Speed of light even in fiber optics is significantly faster than the speed put sound.
And compared to TDM which is the comparison here the only thing added in packet networks latency wise is buffers, for network switches that can be nearly instant so it is only on the send and receive side.
So pretty much if you want to drop dark fiber between endpoints you can get the exact same benefits of TDM on packet switched networks without the disadvantages.
The people asking for TDM are ignoring the fact the literally several industries and some that care greatly about audio quality and latency have prettier much determined that packet switched networks are fine.
> Glad someone else appreciates circuit switched technologies though :) I worked on VoLTE, the amount of twisting we had to do to make the packet network work kinda-sorta like the circuit network was interesting. Overall I think it was a win to make voice another service, but it was a huge investment.
It was painful not having full duplex calls for so long.
This! Full duplex felt so much more like a warm, live conversation. Talking in person can overlap a lot, in a Good way.
Fact: With the good old phone (POTS), when it rang you picked up and said "hello" and started talking. It just worked.
Now, on many calls, the first few seconds are spent on "Do you hear me? Hello? Yes I hear you.". (At work, we call it the Microsoft Teams Random Number Generator - every call gives you one bit with around 50/50 chance (aka one bit)).
To be fair, that usually has nothing to do with the network and 99% to do with operating system permissions and mute, microphone, or bluetooth issues.
wait what? show me the data.
by far the clearest phone calls i've ever been on which were not landline were 2001 era Sprint CDMA, probably because the vocoder bitrate was higher than in later configurations (like LTE), or the MOS scores were held to a higher standard then.
It's really enshittification / phone quality shrinkflation because most normies are not able to articulate what the differences are. Proper QoS, prioritization and bit rate at full duplex would've still worked.
Sprint's "pin drop" ads were bold.
Phones of old could be high quality, and they were always extremely low latency. But usually they were scratchy or muffled or noisy or otherwise compressed. A Teams call on a low-latency low-loss line with good microphones on each side is a far better experience than any long distance call from the 90's. It's just that normally the data connection is not so good.
Calls within and across major networks in the US are crystal clear HD voice (16 kHz sampling) with a decent codec. CDMA of yore was never that good.
Latency used to be a lot better though.
How do you "overload" a TDM system? It's just a multiplexer that turns one wideband link into N lower bandwidth links by bitwise interleaving. You can't "overload" it. Whatever assigns links might be overloaded, and that's usually something like a telephone switch.
There's an unverified mention of a fiber cut, which might result in fewer circuits being available. That usually returns an "all circuits busy" signal.
None of the explanations will make sense until they stop coming from unqualified political appointees.
It's probably not TDM end to end nowadays. If they're backhauling it via MPLS or similar then loss of fiber capacity might render the legacy link inoperable.
>How do you "overload" a TDM system? It's just a multiplexer that turns one wideband link into N lower bandwidth links by bitwise interleaving. You can't "overload" it.
A TDM system can't have infinite throughput, can it? Either you operate with fixed slot windows, as legacy systems usually do, in which case excess data has to wait and buffers fill up, or else you operate with variable time slots, and at some point the windows get too narrow to transmit usable information.
It doesn't make sense to me to claim you "cannot overload" a TDM system. Bandwidth is finite.
Maybe busy signals that prevent new calls from connecting instead of packet loss / disconnections. Kids these days have never heard a busy signal.
If clients are failing to connect because the link has no more capacity, isn't that an "overloaded" failure mode?
> Kids these days have never heard a busy signal.
Busy signals are literally a capacity limit being hit.
I'm most cases, a capacity of one... Eg. the callee (destination) is already on an active call.
> Kids today have no idea how good it used to be
Ah, the glorious 3kHz frequency range of old.
…that never had 100-5000ms audio dropouts once a call was connected. I’d take that any day over the public IP network.
You can still pay for dedicated TDM internet slots if you're a corporate customer on a shared PON or something. Unfortunately there's not much consumer demand for figuring out how to make this work smoothly for consumer telephony via QoS tiers or something.
> TBH TDM is far superior and it’s sad that the world discarded it. Packet switching is the worst thing to ever happen to phone calls.
Your feelings here have no technical basis. TDM is pre-provisioned, bandwith-reserved, latency-capped network that was probably hard for packet switched network 25 years ago. But with all that has gone in -- switched ethernet, qos, oam -- there is absolutely no reason to go back. Heck, even whatsapp calls across 12-hour time zones are quite crisp and it is free.
Because you mentioned WhatsApp. There is an interesting podcast about the Western Electric 500 phones and the architecture of actual phone lines. There is a section where they compare it to WhatsApp and time delay. Maybe of interest…
https://pca.st/episode/a12e4bb6-4e1f-4b57-8f1c-b627b4275869
Ooh cool podcast link, thanks for the rec. Surprised I didn't catch a 99% Invisible podcast on one of my favorite topics.
Good to remember that a WhatsApp call is being routed over a much more diverse, heterogeneous set of networks than a Western Electric 500 phone was. 32-bit stereo raw audio is roughly 2.82 Mbps, which is a lot better than what an older phone did. On any link between two endpoints with 20 Mbps or more symmetric, you can do this trivially. WhatsApp can give you good performance between two crowded venues on completely different parts of the world. Just a very different experience.
> https://pca.st/episode/a12e4bb6-4e1f-4b57-8f1c-b627b4275869
Actual source:
* https://99percentinvisible.org/episode/100-objects-019-weste...
* https://www.bbc.com/audio/play/p0pc7hsz
* https://www.youtube.com/watch?v=jU9P9CR2OsY
> There is an interesting podcast about the Western Electric 500 phones and the architecture of actual phone lines.
Thank you! learnt about Henry Dreyfuss. It's sad the way he died, but can understand.
https://en.wikipedia.org/wiki/Henry_Dreyfuss
Yeah as someone who has video calls across the globe every day, it’s absolutely fine as long as your internet provider doesn’t cheap out on provisioning / uplinks. But in the end that’s a “you get what you pay for” issue.
Codecs are exceptionally good at adapting to available bandwidth and latency.
What kills realtime is buffer bloat
Right, hence their comment about uplinks. They cheap out, that saturates, then we buffer
bufferbloat is actually more than that; it’s actually that the buffers are oversized, which is typically not what you want. This causes a lot of jitter and increases latency.
basically, for realtime stuff, you want to prefer packets being dropped rather than buffered.
this includes your home router, but also your ISP’s router, and maybe even your ISP’s uplink router. again, this just means getting a good one that actually cares about this stuff.
this is an excellent resource on the subject: https://www.bufferbloat.net/projects/bloat/wiki/Introduction...
And it makes echo cancellation terrible.
That's why auto_qos was invented...
zero latency and no jitter
> zero latency and no jitter
this would beat the laws of physics. realistically, it is more like "fixed latency, and very little jitter".
Zero _extra_ latency. It's like 'no sugar added' but the food is sweet anyway, just less sweet than if sugar had been added.
> Heck, even whatsapp calls across 12-hour time zones are quite crisp and it is free
I don't think anyone seriously think that 5g and VoIP voice quality is bad - but latency is absolutely a problem, especially in group calls.
I mean, there is technical basis. This is a classic latency vs. throughput tradeoff. Obviously reserving most the channel for quiet lines is not good for throughput, but it does work wonders for worst case latency. Bandwidth reservation is not without merit.
Definitely think circuit switching has merit for real time comms even today, but that's a separate discussion. Over physically private links, whether you're using MPLS or TDM shouldn't matter.
MPLS typically wouldn’t be physically private, no more than any other IP network, just a different forwarding technology. You can run MPLs over ip networks and ip networks over MPLs.
I have everything from dedicated inter building fibre cables, through cores, wavelengths, ethernet services, IP circuits (still have one international MPLS and a VPLS network eoth 4 exit nodes), to normal internet with varying levels of contention.
Even the fibres aren’t private - they are patched through other people’s equipment rooms, even on short runs if a few miles, and even on the circuits that don’t, they run through ducts over other people’s land.
Packet switching has variable latency. This can be controlled with ptp and qos, and in reality we’re talking sub microsecond differences if you need it, switch latency is measured in nanoseconds And that’s way below the timing ministry is worried about (which is in the tens of microsecond level). TDM will give you guarenteed latency though, you get a slot every X time units regardless, jitter isn’t 2us, it’s zero.
I don’t believe any application needs sub microsecond jitter, but there may well be some scientific applications that do, but at that level you’re running on dedicated cores or maybe wavelengths, between specific equipment.
What stack do you think the FAA is using? I would expect they have private links but maybe I'm wrong.
They don’t own the land the fibres run on, I suspect that many will be patched through exchanges and data centres they don’t own.
It won’t surprise me ifs y places are services provided by other companies - either ip or ethernet, to connect to larger sites. I suspect some will even be connected via public networks.
Major sites may have dedicated fibres between them but whether that’s just a wavelength on someone else’s core, or even a dark fibre but just one core on someone else’s cable, is anyone’s guess. I’d be surprised if they could justify the cost of a highly resilient network dedicated to themselves rather than renting capacity.
What I’m certain is they don’t own their own fibres to every end point.
The FAA is using a mish-mash of tech, because it's been layered on over the years and is constantly starved of funds and investment because it's a government agency that needs congress to sign off on changes. Some network links are modern fibre optic lines, while others are still over decades old copper. Canada's ATC provider, NAV Canada, is not a direct government agency but a regulated nonprofit funded by fees - so it has the ability to modernize as it makes sense. The US still largely uses Radar and voice instead of satellites and data links that Canada moved to decades ago! The story is similar in other western jurisdictions that have modernized long ago.
Some US airports (~90) are still using paper flight strips while other jurisdictions long ago moved on: https://enotrans.org/article/electronic-flight-strips-brief-...
The difference between TDM and packet switching is the same as between static memory allocation and dynamically allocating heap memory.
If you can afford to allocate statically the space for all the variables that your program will ever use, you will certainly increase a lot both the performance and the reliability of your program.
Alas, for many tasks this would result in a requirement for more memory than you can afford, especially today.
To use more efficiently the limited amount of memory that you have, you need to use dynamic memory allocation.
The same happens with your communication infrastructure. Using it with packet switching reduces the reliability and the performance of individual communications, but it uses the capacity much more efficiently, i.e. the aggregated throughput of all communications passing through a given infrastructure is greatly increased.
Why Do We Still Pay For International Calls in 2025? <https://rodyne.com/?p=3293>
I think the superiority of TDM is diminished if there's a wireless link somewhere in the mix (and there very often is for modern consumer tech).
TDM was more stable because nobody was hopping on meetings while walking their dog on a noisy street and then taking the elevator back home. The expectations have shifted.
Modern consumer links (even over WiFi) are usually adequate enough to do reliable, high-quality voice calls over the public internet. We basically choose not to; most VOIP apps are optimized to work decently in a noisy car driving through a forest when on speakerphone, instead of working very well in a quiet room on headphones and crapping out everywhere else. This is a choice we make; there are apps out there that make different choices. Once upon a time, I was very unhappy with that choice, then I grew up and have experienced what the consequences of making different choices really are.
With that said, classical Bluetooth audio urgently needs to die. There's no reason for it to be as bad as it is, and it is truly abhorrent. Better protocol design is definitely possible; most wireless (2.4GHz, gaming) headphones have much better quality and much lower latency. BLE audio will fix most of Bluetooth's warts, and it is slowly, so very slowly appearing on the market.
“See this admin is just like all the others” sorry, no.
Wild speculation… little knowledge… on HN?? No way!!!
starting to see some explanations for what's going on
https://www.wsj.com/us-news/faa-temporarily-halts-flights-ac...
> The outage appears to be affecting a telecommunications network overseen by L3Harris Technologies, according to people briefed on the matter. The network handles crucial radio frequencies that air-traffic controllers use to communicate with pilots, the people said. It wasn’t immediately clear whether the outage also affected the radar feeds that controllers use to keep aircraft safely separated in the skies.
https://www.cbsnews.com/news/faa-issues-nationwide-ground-st...
> An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage, multiple sources told CBS News. The system is operated by L3Harris, which did not immediately respond to CBS News' request for comment.
>An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage
And that's why you never deploy to prod on a Friday.
Especially Friday the 13th.
and they said "wont deploy on friday? thats 20% of the week in lost productivity!"
I have memories of my professor saying something in the lines of
"it's computer science, if you make a mistake nobody will die!
Unless you're dealing with nuclear plants, medicals, or airlines"
well is there a rollback procedure?
it could only be deployed from the servers that crashed
Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday? This would be extremely unusual in any software job I have ever had. For commercial aviation specifically which does have pretty well defined peaks and valleys in terms of traffic over the course of a day this seems like an odd time to choose
Maybe patching a critical vulnerability? I’m also at a loss for why they chose Friday afternoon.
I'd wager management wanted this by the EoW
RCA could be in just 2 words: incompetent people
Weird how all epic stupid decisions come from the people who are supposedly the trained 'decisions makers.'
I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."
Arbitrary day-of-the-week and date-based deadlines piss me off so much.
I worked for a person who had a fixation on "neat" deadines: the "first of the month", "the end of the week", "the start of the quarter", etc. It'd be one thing if the dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case. They were chosen because they were "tidy".
I'd time meetings looking to maximize time for work. You try to meet late in the day Wednesday because an early Tuesday meeting might get an "end of the week" deadline. A late Wednesday meeting was more likely to garner a "end of next week" deadline. Meeting Thursday or Friday guaranteed an "end of next week" but you lost Thursday and Friday work time. Wednesday was a gamble, though, because it was still early enough to get you an "end of the week". Meeting after lunch helped with that.
You always had to be aware of the numeric calendar day, though. Gunning for an "end of next week" deadline when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.
All of this deadline-setting happened with little regard for the amount of time work was going to take.
It was immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.
"The project will land on the first full moon after the vernal equinox"
"More testing? A more gradual phased rollout? We need to get this out! We need a win!"
That could be their least busy day, for all we know. Weekends are busier than weekdays for airline travel.
Friday and Sunday are the busiest days. Business and pleasure overlap. Sat is mostly pleasure. Mon-Thu are mostly business.
Tuesday and Wednesday are apparently the least busy days, at average -10% traffic each.
So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)
But 3am is when a big chunk of planes are parked at gates awaiting the start of the flight day (yes, there are red-eye flights and international but they are a minority.)
Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.
9pm is likely better. if you patch at 3am there's a decent chance no one notices until 6 and then you ruin the next day.
It's also very possible they patched last night since they were possible also updating systems for the big AI rollout and then the issues didn't cascade or cause issues until this afternoon
I did web dev work for an airline and we couldn't even release new updates to the loyalty benefits website during a weekday (even though there was no connection to operations or flights). The fact that this company would reboot a critical comms server at 3 pm on a Friday is indeed bizarre.
> we couldn't even release new updates during a weekday
so you were forced to release updates on weekends? isn't that worse?
Worse for the employees. Better for the clients.
It depends on the schedules of the people who'll have to fix it if something goes wrong. I don't mind working weekends, so when I was in the sysadmin game I always tried to make significant changes late friday/early saturday, so if something went wrong I had the weekend to fix it & clean up. But if the people needed to fix a problem take weekends off, pretty much the opposite schedule applies.
Flights generally peak Mon-Thu due to business flights, weekends are usually lighter unless it's a long holiday weekend.
the update caused the reboot but could that mean the reboot happened due to a prior update? or literally the reboot was FOR the update hmm
Windows Update decided to reboot automatically because somebody didn't press the skip button enough time?
"3 pm on a Friday is indeed bizarre"
Most likely, they HAD to do it.
Probably was scheduled for Thursday evening but was delayed due to the Outlook issues.
IT in corporate is actually more panicked than IT in life safety, because life safety invests more redundant design.
I believe not deploying on Friday is an anti-pattern for most software teams.
What industries have you seen teams deploy on a Friday afternoon?
It happens. My company has done a couple dozen production rollouts today, including minutes ago.
Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it.
Only if the team wants to work weekends?
If something goes down on Saturday there’s a good chance I’m out of cell service
It is interesting to see this bifurcation in action.
Hey software engineers - you know that working weekends is unusual, right? Like... please tell me someone has at least negotiated your right to only work 5 days a week so if you're working a weekend you get Monday and Tuesday off.
If not... Has anyone told y'all you should unionize?
I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)?
The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.
Did you know that when you use your 2 days off to go shopping, someone has to work at the shops?
Yes. That's why people work staggered schedules. Folks who work weekends at shops usually (and should) have two other days off in the week.
People literally fought, bled, and died for our right to not do capitalism 24/7/365, because with no back-pressure that's what the efficiency engine will demand of us.
And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours.
Many people work shifts, it’s quite normal.
What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.
Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!
Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities).
Wanting to have a weekend is smug and privileged now?
People fought hard for the 40 hour work week and 2 day weekend.
You think that should be given up?
Oh, I know all about that history. Having been a volunteer for Bernie Sanders back in 2015, I admired him as being one of the remaining few people who actually remembered this history due to him being part of the Silent Generation and continue to fight for it decades later.
>You think that should be given up?
Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.
We do for clients who have 24/7 users during the week.
I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.
Why wouldn't you be able to rollback on a weekend like any other day?
Not a family man, I see?
That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.
So why the feigned confusion above? It seems you already knew the answer to your own 'question.' :P
My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary.
You can, but it also means you might have to.
cause I don't want to work on the weekend. Why would I introduce that risk?
Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.
Assuming they don’t have an actual life and aren’t somewhere touching grass.
I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.
Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.
It’s ok, claude can do it for you.
Thursday is pretty close to the weekend. Too risky on Thursday?
I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.
Or just have a good rollback plan like a snapshot.
If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.
So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.
And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.
Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.
If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.
Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.
Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.
Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.
Because you are patching a critical system that can't be down during the week.
In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.
Friday deployments make no sense if you have teams that work Monday-Friday.
Several hours. That number goes up quickly when the bus number for something that broke unexpectedly is unavailable. While your coworker who actually knows what that error means is at the zoo with his girlfriend or with his kids.
The whole point of WLB is that we don’t live like monks.
Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation?
Outages are caused by change. Sometimes user traffic, but more often your coworkers.
Do you know how every feature toggle in your system works? Do you know the potential blast radius of those toggles? Can you speculate on the surprise surface area that might be additionally exposed by impedance mismatches and side effects (ie, unanticipated shared state)? Hell does every engineer who wrote a feature toggle know this about their own code? You know who I’m talking about.
If you’re having problems in production that can be anticipated, why are you still having those problems? How long should those be tolerated before someone gets fired?
Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend.
or want to have to.
We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.
It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.
Much better to get fast feedback while the deploy could be monitored against prod traffic.
We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.
I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.
So... it's 24/5, then.
Yes pendant
We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays.
The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.
There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes.
Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS.
It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.
Actually, we ALL work on call. No one gets out of it just because they have seniority. Plus, we have an excellent team of smart people who are able to figure most things out on their own through use of their own knowledge and troubleshooting skills, aided by both internal and external documentation. It is rare that escalation is required.
I never said things don't go to hell sometimes. That's why you build a good team and train your people.
I think it's not just inconvenience for users but also availability of engineers to look at failures.
Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.
Team America: World Police
In public utility we only did deployments on Fridays after 5PM
There's a bifurcation in best practices.
For industries that serve customer bases that are down-cycled on weekends, the dominant effect is "Deploy on Friday to minimize customer disruption chance."
For industries that operate with a 24/7 uptime expectation, the dominant effect is "do not deploy on Friday so you don't have to pay for your expensive software engineers pulling extra hours on a weekend. Much better to have disruption while the folks most qualified to fix it are in-house and alert."
When the update requires downtime (or has such a high likelihood of downtime) that you'd be knowingly (or almost knowingly) inconveniencing users, do it off-hours (e.g. nights and weekends) and gather availability commitments from all the people you anticipate might be useful. If something goes so terribly sideways that it takes a long while to stabilize given the skeleton crew, at least you have plenty of time before users need it again. Friday evening maximizes that window.
Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.
> Friday evening maximizes that window.
For the FAA it sure as fuck doesn’t.
Unless you have god-tier CI/CD, which most places do not, I do not like Friday changes. I’ve worked at precisely one place where it didn’t bother me, and one place where they didn’t have CD, but shared my belief and didn’t deploy on Friday.
Perhaps.
But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.
My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".
I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.
My money is on DNS, but if it is enemy action, will the FAA even know that's what it is?
> It took a few weeks for most Americans to catch on
In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.
> Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday
As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.
(Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.)
> I would surmise the technical reason for it is that they didn't want to work over the weekend.
Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.
That's why you don't deploy on a Friday. :)
There is also a real possibility they did this many times before and it worked. I don’t really get it: if the thing would have broken on a Monday night and it would take a day to fix would that have been better?
The odds of someone screwing something up just before the weekend are way higher. Both because it’s obviously a rush job (it can’t wait until Monday), and because it’s the end of the week and folks are tired.
It’s just dumb.
> As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.
Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?
Nothing like a good Friday afternoon did lot to ruin a weekend.
A security patch would be a reason not to wait.
And because a security patch can't wait no deployment should wait because it's Friday. That's how you stay ready; you're always ready.
It should be fine to update at any moment. Why? Because systems like this ought to be hot spare or blue/green models that have a full working copy at any given time, and a failover/failback.
A reasonable deployment plan considers the reality that the deployment could fail. One mitigation is to choose ideal conditions for a possible failure. You’re correctly pointing out one of the mitigations, but it doesn’t render the others unnecessary.
You can also have reasonable updates done at reasonable times that don’t show issues until unreasonable hours - or something else fails.
But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.
As opposed to what alternative? Being run by the FAA itself or with contractors? Being run by the DOT?
Well, it worked so far, I think it is great. 100% availability is a dream.
That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.
Yeah either the FAA or the DOT sounds good. Not contractors tho.
you seriously think the government would do a better job?
Yes, actually. They're (until recently...) more likely to be there longer, and have a better understanding of how things work and who to talk to.
This is a narrow and shortsighted take. Government would absolutely do a better job if only for the SOPs around redundancy that business is famously flexible on (read: happy to ignore).
the government would do a better job than a private company with incentives to maximize profits.
No. I think the fact that the government wouldn't do a better job is a skill issue and the government has to get good. But also this government where it does farm out to contractors and monopolizte for every little thing could never build the software itself. Even saying "the government makes it itself" is saying we fundamentally have a government tha is different than right now.
It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.
I mean they also make the CPU for the F-35, alongside kinetic strike and electronic warfare missiles, so they could cause plenty of damage if they wanted to. This isn't some mom-and-pop radio company.
> But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.
Foreign cyber-adversaries (the usual suspects ;) are taking notes.
I mean sure and we have that in finance but there’s no amount of time saved that could make me deploy something during market hours
Airspace certainly has busy and quiet(er) periods, but the skies are open all day every day.
There are obviously crests and troughs in flight volume (as measured in passenger-hours), 3pm on a Friday seems pretty cresty to me.
> 3pm on a Friday seems pretty cresty to me
It wouldn't be my pick, that's for sure.
Well clearly they don't have a hot spare or blue/green models, or working failover.
Which is why deploying at 3pm on a Friday is even more bizarre.
Sure, but if it fails and a fallback takes over, will you be going home and having a nice weekend? Or will you be stuck at work over the weekend to fix it?
I'd much rather the team's weekend be impacted than millions of travelers.
What if you could avoid the team impact and not screw millions of travelers.
Why not do a deploy on a monday morning, and impact the teams workday instead of their weekend?
Rolling out updates on a Friday afternoon is still practiced in various industries.
> Rolling out updates on a Friday afternoon is still practiced in various industries.
By people with unreliable systems and a lot of myhts behind.
In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.
They're just now completely phasing out floppy disks. By 2028. Guessing that's not the type of org that deploys frequently !
This is why you don't push PROD on fridays...
Mission critical systems that are in use 24/7 are not managed based on week day vs weekend shifts. Or at least shouldn't be.
My (hopeful) assumption is that air traffic control isn't based on the "weekends are the off hours" mindset.
It's worse than that. It's 3pm on a Friday before Thanksgiving week.
I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.
No it's not. Thanksgiving is on November 26.
https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
Yesterday email went down for half the world's population because MS pushed a faulty update and a day later all US planes are grounded also because of a faulty update, come on what are the chances.
If it was a malicious act, or maybe even two, would they publicly state it was the case ?
If you don’t let juniors ship to prod on Friday afternoons sometimes they will never get their wings.
Did anyone question why Microsoft would do an update at 10am on a Thursday?
It does not matter when they did the updates. What matters now is what the source code is for the code that they deployed. We need to see what the documentation says about human reviews, testing, and certification processes.
If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.
This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.
Apparently the L3Harris CEO was ousted in August in a Jensen-like scandal. It's also the company that was modifying Trump's Qatar corruption plane.
https://www.cnbc.com/2026/08/17/l3harris-ceo-chris-kubasik-l...
Didn't they get bought it merge with some other giant company ?
> leading to a crash
poor choice of words
This indeed is a poor choice of words…
Who updates on Friday?
There are no weekends in the airline industry, in case you didn't know.
Everything operates normally, as any other day.
There are still ways to roll out updates sensibly and safely in a 24/7 industry
There are early mornings and nights. Air traffic slows down significantly past midnight.
Air traffic slows, yes, but it doesn't stop. ATC operates 24/7/365. Whether you have one plane or one thousand, the entire system needs to be operable.
It's basic risk mitigation.
I agree that you should do everything possible to make sure there's no downtime. But if there is a time for downtime, it's when there's the least amount of air traffic.
Screwing up big time in the dark might be worse than doing it in daylight.
no weekends for blue collars, but why would there be no for white collars?
Isn’t Friday usually a light travel day?
No, very heavy travel day for business.
No. Business and pleasure overlap, making it a busy day, along with Sunday.
People who have a testing window Saturday morning, which is how a lot of legacy industries with cyclical schedules work. Weekends are slower air travel days.
A junior engineer according to my table: intern: break the build and go home. junior engineer: break the build and go home for the weekend. staff engineer: break the build and go on vacation. senior engineer: break the build and retire.
Organizations that have good engineering practices.
This woodoo like belief you should not deploy on Fridays, Christmas, Valentine’s Day and other magical days is a disservice to the industry.
But when it goes wrong you’re working on the weekend.
I think it also creates a tension because people are more compelled to "just ship it" to go home. If it's a Monday and it's not quite ready yet or someone has doubts, more likely to just hold off.
I feel like people want to go home on Friday and will make sacrifices they would not make on a Wednesday.
When things go wrong, you work to fix them. Doesn't really help if it is the weekend or a very long Monday.
Things go wrong all the time, it is not a reason to fear doing stuff.
Huh. I thought Collins now ran what used to be Aeronautical Radio, Inc. (ARINC), which is the ground portion of air traffic control radio. L3Harris has the radars and ADS-B.[1]
[1] https://www.l3harris.com/sites/default/files/2021-08/MN_SBSS...
Collins/ARINC is used for oceanic HF radio. They run "San Francisco Radio" and "New York Radio" specifically, which are the HF radio operators that relay oceanic messages to ATC.
(They also own a small number of private VHF stations for relaying company messages to air carrier operations domestically, but these aren't used for ATC purposes.)
Otherwise, the VHF radio links are operated directly by FAA TechOps.
L3Harris owns the telecommunications backhaul between facilities.
I bet it was an issue with BGP.
Rollout new router config Thursday night - seems fine at first - issues compound and then remote admin goes down
Yes, I doubt someone pushed out an update minutes before the outage. The hard bugs are the ones that take a long time to impact a system and don't always appear in testing.
Seeing L3Harris failing gives me a little bit of anxiety since their systems run some of the backend of some nuclear operations.
These systems fail safely. In the meanwhile, millions of people worldwide die of air-pollution of the fossil fuel industry but they don't even get a 100th of the scrutiny that nuclear industry gets.
This is not a critique of nuclear power. It is simply a (mildly existential) realization that even with high reliability goals, failure is still possible.
I expect the teams developing ATC communications and the team maintaining the CANDU DCC to be distinct and not have many dependencies.
How is there not a second backup network???
Why only “one” network at all!?
The Internet was purpose designed to tolerate disruption as major as a nuclear war, which it achieves through decentralisation.
Private networks exist to make idiot managers feel more confident while actually dramatically reducing availability by putting all their eggs into a single basket.
Many aviation systems predate the internet. Depending on the region, some systems do not use Internet Protocol, let alone TCP.
The FAA Telecommunications Infrastructure (FTI) operated by L3harris was built between 2002 to 2007.
That is already well within the era of IPv6, let alone "the Internet".
There's no excuse.
For something so incredibly critical in real time, I’m not sure I would want to trust the Internet.
thats like mandating using I-95 for all ambulance & firefighter traffic
Isn't using a private network like mandating using I-95 (since it might be blocked from a freak accident one day), while using the internet, with redundant paths between hosts, like allowing the traffic to find the best path?
When was the last time the Internet "went down" globally?
The last time the internet caused flight outages was September 22nd of this year.
Over what area? Localised, nation-wide, or global?
One network can be a fully redundant design with multiple geographically distinct links and so on.
But it does not make it 100% resilient to cascading failures or possible Layer2/3 loops that might happen during a software upgrade, especially in a country wide deployment.
The FAA will give a root cause at some point of why this happenned.
The internet is pretty unreliable as it goes, and the internet in your locality is probably only one network (e.g. Comcast's) - what are you gaining by running critical systems over Comcast internet access, instead of Comcast leased line? Saving a few hundred bucks a month, yes, but that's because you're getting less reliability than you actually need.
Also lower MTTR and more robust SLAs in many cases. Business services will usually result in service credits being issued if the SLO is breached. Residential... not so much.
The more time I spend working in tech, the more I realize that everything important is held together with cardboard and duct tape.
You had duct tape? Lucky!
True for every industry / field I've worked in, unfortunately (which is more than I'd like to admit).
That's because everything really important is not just tech but a socio-technical system of tech+humans operating/building/maintaining that thing. A lot of the things can't/shouldn't be automated with tech (see the Irony of Automations paper) – they are up and running thanks to the resilience (socio-technical quality) which appears as a "cardboard and duct tape" for untrained eye.
Any time people freak out about these kinds of things and assume the absolute worse, I just tell them I’m surprised these things don’t happen more often
Cue that line from DS9 where O’Brien says he’d never be caught without a secondary backup during a fight, and the cardassian is like “you have a second backup?” incredulously. https://youtu.be/o7o9ZtipVyo?is=y3for_WXwISmuqpd
I wouldn’t say anything but this is approximately the only time in my entire life this scene will be relevant.
I remember that scene, but I can’t believe that you had it at the front of your memory like that. That’s awesome.
There are people who literally live for this shit, you know. I have friends who have puns lined up for 1000+ situations.
> caused the system to reboot, leading to a crash and a communications outage
why a reboot would cause a crash? or it means that an update irreversibly broke the system and it was rebooted but that didn't help?
Maybe the systems connected to it are not properly designed to handle such a reboot
"What is Microsoft Azure?"
Yeah - like, why do they not have an "HA" cluster, where they patch and reboot a subset at a time... Something this critical should have some sort of failover/load-balancing in-play...
I wonder if the update was signed with a cert chaining to a CA with a private key of "4".
Thank you.
I learned more from this post in 15sec than I did in 20min from watching the Sec of DHS and Sec of Travel press conference
Doesn't L3 Harris use Microsoft Azure Government for most of their mission critical infrastructure? Are we looking at fallout from the Microsoft infrastructure collapse yesterday?
Leading to a crash is very unfortunate wording, when talking about systems for aeroplanes.
Info on this is amazingly sparse. Apparently Sec. Duffy posted something on X/Twitter, and that's all the news media have.
Looking at LAX departures, it looks like nothing has departed since about 10:30 AM PST, and everything later is showing Delayed or Cancelled. There are times after Delayed, but it's not clear those times mean anything. Some are now in the past.
"flysfo.com" is too overloaded to display departures. Stuck at "loading...".
JFK departure status shows almost solid "Delayed".
No good explanations anywhere.
Surprised there's not been any updates yet. Not even one of this will continue or it is being resolved.
Sec of Homeland Security gave an interview about 10 minutes ago saying it's indefinite, they don't know the cause, and to expect resolution to take a while.
Oh wow. That's not really good, is it.
Economy taking a lot of crazy hits this week. They're footgun hits too, not even enemy action.
Edit: ok so one of them was maybe a little bit enemy action.
I'd say that's not really known yet, although there's certainly a footgun element at least.
I'm sure Jensen had some enemy action.
There’s only so long the American wall of competence can be chipped at by stupid, before bigger chunks start dropping. Watch out below.
Foot is not the correct bit of anatomy in Nvidia's case.
is it a bad news that Homeland Security doesn't know the cause or is it a good news that they aren't the origin for it?
SFO had delays this morning because of winds. I'm not sure if LA also had bad weather, but SFO had a lot of rain and if you know SFO, you'll encounter a delay if someone sneezes in the direction of the runways.
I actually got out surprisingly on time at 10:30am. The other airport at landing was chaos though.
When will we congratulate all the AI companies for this?
Probably someone prompted an agent: solve global CO2 emissions :-)
> When will we congratulate all the AI companies for this?
5 days from now? https://news.ycombinator.com/item?id=50465829
They forgot to add "don't make mistakes". It's common.
Update, 3:00 P.M.: Pacific bound flights are cleared to depart, including Hawaii and Asia.
Source?
upd: sfo resumed international flights to asia
flights departing hawaii and alaska are also allowed
Actually true.
https://news.ycombinator.com/item?id=50491511
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
I'm stuck at SFO and a flight for Hawaii just boarded perhaps 20 minutes ago. I don't think it has taken off yet though.
two flights to tpe and a flight to tokyo took off since 3
thank you for fixing the link(?).
If you're flip your claim via edit it's usually polite to do it via addition rather than revision so the other replies still make sense/don't get downvoted by confused people.
https://www.flysfo.com/flight-info/alerts-advisories/faa-iss...
I saw one!
Makes sense. CNN reported that international airlines were still departing for North Atlantic flights, apparently because they use a separate communications system.
Yeah, oceanic flights largely use satellite-based CPDLC today, or HF voice as a fallback, but not VHF in any case (which I suspect is the implicated system here).
VHF Omnirange (VOR)[1] is unlikely to be involved. VOR stations are not dependent on a network. All they need is power. It's 1950s technology. The big ring of antennas next to US 101 just north of SJC is a VOR.
[1] https://en.wikipedia.org/wiki/VHF_omnidirectional_range
Sorry, typo, I meant VHF (i.e. voiceband), of course. VORs are indeed for navigation, not communication. Updated.
They’re under socal departure (or equivalent) until handed off to - well, nothing. So no need for intercommunications.
Info on this is amazingly sparse. Apparently Sec. Duffy posted something on X/Twitter, and that's all the news media have
Maybe look further than Twitter.
The legitimate media from the New York Times to CNN and beyond all showed video of Markwayne Mullin's news conference 20 minutes before your post.
All European and Mexican departures to the US have been stopped: https://x.com/sentdefender/status/2121381457683964308
Here's what they said caused it in the White House briefing:
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
https://www.faa.gov/air_traffic/technology/cinp/vsrprogram maybe?
Seems kinda nuts to replace "obsolete" switched telephone circuit technology that's incredibly reliable and zero latency with VoIP.
... no it doesn't?
reminds me of the fact that (for some reason) it costs thousands of bucks to replace or install a stop sign.....even in rural FL
Consider the minimum expenditure for anything is going to be a crew of at least three people for at least a full day, how would you do it for less than a thousand?
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
> Ignoring the cynical view that many of those people were likely fired
In your mind maybe. Most are dead or retired.
New techs have been trained on splicing fiber and VoIP over the last two decades.
Nobody knows about vacuum tubes anymore either, that doesn't mean they were fired and had their pensions taken away.
It is obsolete.
The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.
Not to mention the expertise for these systems have mostly either retired or in some cases passed away.
My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.
Many such cases.
Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.
It’s funny, this is basically an exact analogy for the situation that my software engineering team has been in for at least 5 years now.
For various reasons the team got to a point where lots of the core services were running on totally obsolete technology. I’m talking CoffeeScript here, Python 2.7 there.
Over time, enough turnover and changes happened that they had a pile of repos where not only were the developers on the team unable/uninterested in working on the ancient tech, it was impossible to hire anyone skilled in those areas.
So we ported the features of the legacy services into a modern stack and are STILL working on getting the last of our customers migrated 2 years later. The legacy services are still up but only 2 of us really know a single thing about them, much less re-deploy them if something goes down. Documentation and run-books are great but hands on experience can be really hard to replace in the worst situations
I don't understand why they did not test this before deploying on production. How can we sure that when they develop a whole new system, it will work?
> I don't always test my code; but when I do, I do it in production.
https://static.wingify.com/gcp/uploads/abtasty/test-in-produ...
This admin can't be trusted with lining a pool. Why is their incompetence still a surprise to you after all this time?
Even if they were confident the new components would work, why deploy at 3 PM Eastern on a Friday?
Let's set that aside and instead ask why tf they rolled out anything on a Friday? Or was it that they rolled it out on Tuesday but failed to notice it was close to overload state until it actually overloaded on Friday?
Or it was Friday traffic that overloaded it.
Sounds like the new system effectively performed a DoS on the old system. They probably don't have a development environment where they can hook the two systems up. Even if they do, it might actually have been an out-of-the-way component that failed first, impacting another, and so on. If they have a test environment with the two systems, a complex failure like that would unlikely to be found (and would explain why it took hours to resolve).
As someone who works under a different umbrella in the federal government, how do you propose testing something so archaic and complex without replicating an exact replica of a 40+ YO technology?
I've actually worked a system where the test env was scavenged old eq set up to mirror exactly the ancient prod env. It's possible, it just takes a lot of work. Scaling it might be touchy though.
> It's possible, it just takes a lot of work
Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.
And it sounds like the fault conditions only occur when you do scale up.
Emulate, computing is cheap. In an application as important as air traffic control it seems like it wold be worth it to _always_ have a working copy of the original system specifically for testing changes.
You think an error that brings down _national_ networks is something you can emulate?
Thank you, you get it.
Not aware of specific system, but critical infrastructure systems have a quality assurance system that replicates production albeit not perfectly. You may replicate the core system but not all its interfaces, you might replicate it but scaled down etc. There are ways for solid tests to fail. For systems with downtime, a test on production is useful iif you have solid rollback procedures. There’s lots of art in the operation of these systems.
Its possible, just not probable with who you can hire for gov work.
If you cannot replicate everything then maybe have a very conservative rollout approach so that if it fails, it only affect a small subset of the traffic.
That's sounding like a good idea, simulate using a RTOS.
I’m sure it worked on his machine.
The Secretary of Transportation's claim to fame is having been on MTV's the Real World and being a sycophant. Is it really that shocking? Really? We live in the dumbest timeline.
I'll be interested to see what the root cause of this all is, assuming it doesn't get buried. What I find both perplexing and predictable is how the finger has been pointed at Congress for not funding a replacement. Well, who's controlled Congress for the last 2 years? Who was president in 2017-2021? Who controlled Congress in 2017 and 2018?
If you want to dig deeper, this administration has also (unsuccessfully thus far) claimed in court that they're not bound by Congress when it comes to spending under the Impoundment Control Act.
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks just now to make that more clear.
I agree with your view.
Oh, my apologies, I had originally read the last 2 sentences in your voice. Thanks for the update.
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
Clapper in 2013:
https://nsarchive.gwu.edu/document/21491-document-90
Dan Coates in 2018:
https://www.euronews.com/2018/07/13/us-intel-chief-warns-of-...
Tip of the iceberg folks. Hold on to your butts…
But somehow it's this administration's fault.
In this particular case? Yes, very much so.
Both things can be true: US infrastructure is in a dire place AND this monumental acute unforced fuck up is the fault of the unqualified, corrupt people who were put in charge of it
It certainly does give strong "The Empire is Falling" vibes.
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
We have a rule in my dev shop: if a piece of code (or hardware) has been working without fault in production, you do not touch it. It doesn't matter if it's old, if you don't like the tests, if it's in a programming language you despise; if there haven't been any problems in production, you don't touch it.
What about legitimate issues?
That's the exception they noted ;)
> if a piece of code (or hardware) has been working without fault in production
And then you have the common reality: "a piece of code has been working in production with known faults & known costs". And you have to weigh up costs and risks.
Well yes of course. If there are bugs or issues in prod (performance/scale), then code can be updated or replaced.
But we have tons of legacy code that our engineers don't like, yet it runs fine and has run fine for years.
Taken to the logical extreme, you end up with magic ancient doohickeys that nobody dares to touch, and everyone prays that the inevitable failure lands on someone else's watch.
Many forms of "touching" are tests that cannot be accomplished any other way, so untouched is untested, and if something can't be sanely turned back on after it gets turned off, that's just a latent, undetected fault.
This is exactly the way we operated Crystal Reports. "If it works, don't touch it"
Especially on a Friday. The 13th doubly so.
Sure if you don't like the tests you can write more tests, no? Tests are not production. Am I missing something?
In some cases the testing frameworks cause CI headaches
That’s the wrong attitude.
This thinking caused an extended outage for the biggest mobile network in Australia: https://www.telstra.com.au/exchange/some-mobile-calls-and-da...
Those pair of boxes nobody was brave enough to touch cost them AUD 30 million in fines.
I haven't read anything to suggest the root cause was similar. As far as I've seen Telstra operated in a way very much the opposite of the FAA and ATC. A reboot of a GPS module propagated the wrong time via NTP to the network. An audit said Telstra didn't treat the correct time as a critical capability (you know, with backups and such).
They had redundancy! Both appliances that "always worked" stopped working.
In my mind, anything that's not under continuous maintenance immediately begins to suffer from bit-rot and degrades steadily over time.
Of course, that's not a degradation of digital media, but the "dropping out" of concepts out of people's minds.
This is akin to Netflix's Chaos Monkey approach: systems under constant change and attack necessarily become robust. Systems untouched under glass like something in a museum are typically fragile and crumble if so much as breathed on.
Confidence comes from practice, not ignorance.
This happened because the GPS module was outdated.
> That’s the wrong attitude.
Not necessarily. Debugged code is gold.
We used to do this. Now every customer and their mother is demanding upgrades because their security contractor ran a scanner that flipped out over a version number string.
Put someone competent & accountable in charge and then let’s talk.
how could this possibly not have been SMART? were they really doing two simultaneous rollouts?
Sounds a bit like Chernobyl. People doing stupid stuff to keep “the party” happy.
Chernobyl was get-there-itis which affects most systems. They pushed forward when they should have rolled back.
They also kept noting that the new IP based routers were not showing any inconsistency. To me that says that they were bridging IP traffic over the legacy systems, probably a protocol with retransmits like TCP, and a storm of retransmits was overloading the legacy systems, and older components using a less durable protocol were drowned out and dropped packets.
Mullin doesn't actually understand anything he said.
aka more trump crowd yolo'ing sht way out of their depth
People have been saying since the 80s that patching/piecemeal doesn't work.
I have no idea whether this particular failure was Microsoft's fault, but I was in the air on September 14, 2004 when a substantial portion of the southwestern US's air traffic control went down:
https://spectrum.ieee.org/lost-radio-contact-leaves-pilots-o...
(I was near ORD at the time, flying toward the outage area. We diverted to ORD, and the pilot was very friendly about it.)
At the time I thought it was Microsoft's fault, as Windows 95 and 98 were susceptible to crashing when GetTickCount() overflowed, and this ATC crash was due to a 32-bit millisecond counter on a Windows machine overflowing, but in this particular instance the system was apparently not running on a version of Windows with this particular bug and the failure was just an application error.
Aren't computers great? :)
By chance were you flying an F-22 Raptor?
This sounds like the beginning of a good story.
F-22s famously had major computer problems when crossing the international dateline.
I wonder if there is a list someone of planes that have had glitches when crossing various borders or boundaries on the Earth?
I remember hearing of a fighter a long time ago whose autopilot flipped out when it crossed the equator, due to sin(latitude) becoming negative.
That was the F-16, and it's either apocryphal or something reported from testing/flight simulations and not an actual incident with a real aircraft.
The Garmin systems I flew were only certified for ops south of 70°N and north of 70°S. Nobody knew what would happen closer to the poles. Maybe the system was just not tested to certification standards out there, or maybe it would do something really stupid and want to fly upside down or backwards! Either way, I wasn't going to be the pilot finding out on a dark night.
It’s likely to do with satellite geometry at extreme latitudes leading to lower accuracy. Maybe also the weaker access to correction data from geostationary satellites.
Could be. The aircraft did not have inertial as a backup.
The other good story is the C-130 navigation computer freaking out when flying into Bar Yehuda https://avgeekery.com/challenge-flying-sea-level/
To err is human.
To really fuck up, you need a computer.
Computers are machines used to amplify human error.
> A computer is a device that enables a person to make more mistakes faster than any previous invention, with the possible exception of the combination of handguns and tequila.
-- some clever person whose name I do not know
I wrote about this four days ago:
"Realistically you want fewer, higher-quality engineers who are educated in these tools and treat them with respect and caution. These tools are dangerous and you can’t play around. All it takes is one mistake and the blast radius is way, way larger. One mistake can do so much damage it undos all value of an entire year. Before downloading a malicious dependency was limited to deterministic behavior and could be scanned for. Now, you run a MCP server locally, it can prompt inject (in many different ways), and now your super powerful AI buddy is hacking you in ways a deterministic piece of malware could never. You could cripple entire companies through negligent use." And "And now we are controlling models that can be hacked in so many different ways and literally blow up companies. It used to be accountability the software wouldn’t break. Now it’s accountability that in creating the software we don’t ruin the business."
Not saying this was caused by malicious actors or negligent AI agent usage nor am I saying was the previous Microsoft outage either the above references but it doesn't look great. If our technology is really so brittle, we might be looking at a return to way slower world to put it simply. Maybe no remote work because it gets hacked too easy!
Amazing quote.
Secretary Mullin/DHS is confirming that this will likely be an extended outage...
> “This isn’t going to just be be resolved and turned around within the hour"
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
Why is the DHS secretary taking the lead on this? Shouldn't it be Transportation?
I guess it depends on what is actually going on. And nobody really seems to know that right now apart from the shit having hit the fan.
I did see another statement where Mullin said this doesn't look like a malicious act, which just makes it even more weird that he's sort of taking point. Maybe the Real World actor leading transportation is in over his head?
They may both be equally well-suited for the job...
> ...both equally suited...
That's a pretty low bar.
I'd take it ad an indication of the cause.
Because he is by far the bigger toady?
The TSA is under DHS so very likely a ton of cooperation and coordination between the FAA/DoT/DHS
Anyone have a screen shot? I just see "NO Advisory"
edit: https://www.fly.faa.gov/ois/oisedit/adv_oisedit_dis?number=1...
Holy french.
Why
13/2053
?
UTC
My guess at parsing: in effect from 8:53pm UTC on the 13th to 4am UTC on the 14th.
Airplane software is some of the oldest on the planet, pre-dating anything that would be considered a best practice. When you see something weird in the space, assume it is because someone in the 1960s thought using more than two characters world be a profligate waste of space.
Unrolling these decisions is a monumental task given the decades of hardware already deployed in the field that will only be upgraded when the current unit fails.
> When you see something weird in the space, assume it is because someone in the 1960s thought using more than two characters world be a profligate waste of space.
And with 1960s technology, this might just have been a very reasonable decision.
Totally. Just some of those decisions put awkward constraints when the hardware is now a million times more capable.
It's a format called "Date-Time Group" (DTG). Used in military, I think it dates back to teletypes, and is better to say over radio than modern ISO formats.
You'd say "Valid from the thirteenth at two zero five three zulu" over radio.
The TLDR is that punchcard operators found the DTG date format convenient. Punchcard sorting is easiest when consecutive bits encode the sort key, and the DTG format is easy to split into different sets of bit ranges corresponding to common business logic.
Friday the 13th at 08:53 PM UTC
It disappeared briefly, but is back now.
Same here, I get it only on some refreshes, otherwise it shows "no advisories". I hope that's not the main path these things are disseminated normally.
This thing is probably getting 10,000+% normal traffic, database is timing out on some queries, and it represents no answer as no notice.
That would be a pretty concerning way of handling database failures, but I suspect you might be right.
Power is off? Let the dinosaurs out.
From Reddit:
https://www.reddit.com/r/aviation/s/HUT3wjABQ6
For anyone who doesn’t want to click through, there are problems with “frequencies”. Something must be jamming up the airways… but nationwide??
Probably a telecom issue. Most radio transmitters are operated over phone or Internet from a remote station.
Teams phone down?
Are these systems dependent on landlines?
VZW and AT&T are moving full tilt in shutting down landline central offices.
Literally the gay dude from Airplane pulling the plug from the socket may have happened here.
Na most are VoIP nowadays.
"Just kidding! Hyuh hah!"
Looks fake. Outtages (sic)?
Oh nice, very helpful. The sibling thread mentions that this problem is specific to "ARTCCs", which are not the towers at airports but rather central hubs:
Wikipedia mentions they in particular use special "very high frequency aviation bands" (around "AM 118-137 MHz"), which is presumably why they're all linked together. It also mentions sharing systems, but no clear specifics that I can find. Certainly doesn't disprove the botched rollout theory, tho...
https://en.wikipedia.org/wiki/Area_control_center
EDIT: someone below posted this, seems relevant -- the 'Radar multiplexer pipeline' that's on a "fast track" and an "aggressive track": https://www.sunhillo.com/wp-content/uploads/2025/04/SUN3125-... TL;DR: upgrading radio systems from old analog tech to new digital tech, I think?
I'd be shocked if someone greenlighted an atc upgrade procedure which involves closing all US airspace to do the switchover.
That’s obviously not what’s happening. But if this upgrade is really implicated, the real question would be where the line of accepted residual risk for an outage like this was drawn.
Well sure, but I wouldn't be shocked at all if someone greenlit a plan that wasn't supposed to involve closing the airspace, but which failed and which had an insufficiently thought out rollback plan.
Sorry I was unclear; the guess is that it caused an issue, not that they did this on purpose with no warning.
https://i.postimg.cc/bvzQNqc5/Screenshot-2026-11-13-at-21-37...
Seems “not great” that the notice isn’t even showing…
https://nasstatus.faa.gov/map
https://i.postimg.cc/XqcsDQw9/image.png
I'm watching it live on ADSBexhange, here's a flight I picked at random going approximately past me that'll land in Detroit about an hour from now... there's not much left over the US over 40,000 feet. Only 2 planes over Indiana at that altitude right now.
https://globe.adsbexchange.com/?icao=a519b8
It's Delta Airlines flight 1070
https://www.flightaware.com/live/flight/DAL1070
Edit: It landed safely. Continuing to watch our airspace clear out, most traffic at 40,000+ feet is inbound international flights
I'm watching ADSBexchange too.
It's eerie seeing the total number of aircraft tracked decrease. I put on a filter for 1000' to 80000', to exclude aircraft on the ground, and the number was around 12K when I started watching. It's down to just over 9.4K now. Wild.
Edit: 22 minutes later and the count is down to 8.4K.
Now under 8k
Below 6K now and the map is looking really empty here. The medical helicopters definitely stand out more now, though.
Edit: I also never noticed balloons before. N804XR is kinda interesting, for example.
Oh-- there are two. N804XR and N807XR. Tethered Customs and Border Patrol aerostats. I'd never heard of these things before.
If you haven’t seen it, there’s an exhibit about 9/11 at the Smithsonian Air and Space Museum in DC that shows the airspace in the lead up to and just after the attacks. If you find the traffic changes now interesting, I think you’ll find that interesting as well. It’s possible it’s online if you do some googling
Appreciate that reference. I'll look for it.
The house where I lived on 9/11 is in the approach for a runway at a local airport. Air traffic was a fixture at that house. Somebody was landing at all hours most days.
It was the strangest thing to have no traffic at all back in 2001. It's odd for the absence of a thing to be jarring but it was.
Here it is for anyone looking
https://www.youtube.com/watch?v=bo1ZtpKqlYw
To be fair, above FL400 4̶0̶A̶G̶L̶ is relatively high for most airliners. Most are between FL300 and FL400.
FL400. 40 AGL is doable for almost all airliners ;)
Thanks. Meant to slap a k on that 40.
Actually that would be quite challenging over most terrain.
Almost all? Now I'm interested
Absolutely tons between 10K-20k, though.
This tracks with what I has seem elsewhere. I don't have the details handy now, but the communication system used for long range handoffs for flights above 18,000 ft seems to be what is impacted. Keeping planes below that deck would keep them bouncing more frequently between shorter range towers but at least still tracked and in communication.
Jet efficiency that low is horrendous though. I suspect a lot of that lower is piston and turboprop, even today.
Oh its absolutely bad fuel efficiency. I'd expect they go to that altitude where communication is still reliable and do everything they can to get flights on the ground safely regardless of inconvenience or fuel inefficiency.
My view on ADSBexchange is centered on CMH, zoomed out to Washington D.C. to the east, and filtered by 18K up I've got 142 aircraft.
I'm also near Detroit and am watching the trackers and I'm pretty sure I just saw N831AK take off from Troy Oakland in violation of the grounding order. I wonder how common that is.
I live in the general area of you too. Nice to know that there are Detroiters on here.
I saw several flights taking off actually. Like this one from EWR. https://globe.adsbexchange.com/?icao=394a08
It doesn't look like the stop really applied to VFR flights, so I wouldn't expect an old Glasair taking off from a municipal airport to be affected.
Looks like some flights are taking off from JFK https://globe.adsbexchange.com/?icao=4cc58a
It could be the flights that are heading directly away from US airspace over the atlantic to free airport space for other planes?
why are there so many planes out and about right now? Most of these could have landed long ago I would think.
More info on AP [1]. Looks like a comms issue. The figures they quoted for the overhaul are absolutely insane - almost $45 billion (I used to work in telecom):
> "The government has been working to overhaul the outdated technology that air traffic controllers rely on since Congress approved $12.5 billion for the project last year."
> "Duffy has said that another $30 billion will be needed to complete the work, and the White House has begun lobbying Congress to approve it."
[1] https://apnews.com/article/flights-halted-airports-faa-f81b0...
Never let a crisis goto waste. Someone screws up and the first reaction is to say “we need more money”
They asked for the money before the crisis which makes the timing even MORE conspiracy coded somehow
Should probably have waited 48 hours for the news cycle to move on and ensure absolutely nobody knows the money is needed
When the system is held together with chewing gum and 1980s-era software, when it fails, "someone screws up" is not a fair reaction. Of course it fails! The wonder is that it still mostly works.
> The wonder is that it still mostly works.
A lot of very hardworking people getting too little pay for something so critical while their employers (some contracting company) take a lot of profit. Eventually they burn out waiting for help and a replacement system and retire or quit, then the systems really start failing because the knowledge leaves with them.
> first reaction
The FAA et al. have been working to modernize US ATC since ~2003. https://en.wikipedia.org/wiki/Next_Generation_Air_Transporta...
The next step is a ton of hardware upgrading, which always costs real money.
> working to modernize US ATC since ~2003
I was working on upgrading US ATC in 1995. As were lots and lots of other people.
Snark aside, this is actual advice in Crisis Engineering by Nitze, Weaver, and Dickerson, and largely matches my experiences as well. You can harp on and on and on at upper management about why budget or attention needs to be allocated to take care of something before it blows up into a much bigger problem, but they'll never move on it -- until it actually does blow up, and then suddenly you've got nearly a blank check to do whatever's necessary to make sure it never happens again.
If you don't take full advantage of that, you're missing out on possibly your only opportunity ever to fix whatever's needed fixing for years.
That makes sense, and yet...
If things blow up because you're upgrading to something supposedly better and it goes wrong, then that's not a time when you can properly fix the new thing. You can only push forward with the transition or go back to the old. I disagree with "you've got nearly a blank check to do whatever's necessary to make sure it never happens again." You have a blank check, but anything you use it for has to be scoped within stopping the bleeding. As soon as you're onto the new or back to the old, the check expires and you likely can't touch anything real anymore because it'll be involved in lawsuits and finger-pointing.
I don't disagree with the core message, but I think it plays out differently for a major upgrade vs a production meltdown on an existing system. With mostly an existing system, you can pour effort into resilience. With an upgrade, I doubt people would let you beef up a system that is still on the chopping block, even if it's getting a life extension. And if you're still switching to the new system, all effort is going to go into getting it up and running and anything that gets in the way of that (as quality improvements are likely to) is going to face some serious resistance. The check is not truly blank.
Eh. The blank check is to stop the bleeding, but the person who is writing it doesnt know if you are using it for a proper fix or a band aid. So use it for a proper fix.
They don't know what you're doing, but they know how long you're taking to do it. It'd be risky to go fix stuff while they're breathing down your neck.
Funnily enough this also works at a personal level, as in, within yourself.
You can spend years working on a problem but if you have a catastrophic failure in your personal life, you can use that to change dramatically with far less internal resistance.
What is the cost of grounding all flights across the USA for one day?
I’ve seen figures that estimate the aviation industry to provide about $1.5 trillion worth of GDP for America, so 4-5 billion dollars. Feel free to double that for the cost of dealing with unexpected disruptions and loss of institutional trust.
All of the AI money discussions have broken me for being surprised by large amounts of money. The frontier companies spend that in what, 5 minutes?
All US AI companies combined: about two days.
Assuming the GDP is lost completely and not at least partially made up by increasing number of flights on other days.
It probably is at least partially made up. But I don't know about you, but my flights for the last year or two have all been booked up for every seat. And I'm not sure carriers just have a bunch of planes sitting around not making money. Or a bunch of crew sitting around not making money.
Hold on, let me fire up Claude…
It's because air traffic control systems in the US haven't been significantly upgraded in like a century. Keep kicking the can it will get more and more expensive
The DOGE/OMB rampage didn't help either.
Wouldn't that have been taken into account when the $12.5 billion was allocated? Or was the $12.5 billion just meant to be the first allocation and they knew substantially more would be need later?
A century? Since 1926? The FAA was created in 1958.
I'm hearing some discourse about "outdated infrastructure" and increasing investment because most systems are analog and not digital.
However, this failure mode seems more likely to be caused by digital communication systems and an 'upgrade' (with commercial incentives) to newer technology.
My bet is that if short-wave radio transmission keeps being used, availability will be higher than stuff like Github.
Having read the NYT report about "what we know" https://www.nytimes.com/2026/11/13/us/faa-ground-stop-flight... and the fact the transport secretary immediately pointed at specific issue (overloaded routers) without any investigation, combined with the fact that this revelation was intermediately followed by the request for money means that I will not trust what they are saying unless there is an independent investigation about what happened.
On a plane about to take off and obviously we didn’t. The captain is very confused and says he doesn’t know what this means.
https://www.bbc.com/news/live/c6z9dwn12037t
> Transport Secretary Sean Duffy says the FAA "issued a nationwide ground stop for all US airports following a communication issue impacting all Air Route Traffic Control Centers.
Are you in the air now? Looks like it might be over but unclear really
No we were about to push off at DET when the grounding came in.
apparently FAA is not providing pilots with any more info
Outlook outage?
Same at SJC. The captain basically said, “IDK what to do, so I’m just going to read the bulletin Alaska sent out”
Now we’ve been deplaned and they are going to return checked baggage to us
Hate that for all involved. Was flying out of ATL when the Crowdstrike clusterfuck imploded Delta's antiquated crew scheduling backends. Not a fun night.
Godspeed for everyone getting to where they need to be, somehow, or at least ending tonight with a roof over your heads.
Same here, LAX
Taking off now
The flight crew on my LHR-SEA flight "still expects to land in the US." The lack of conviction in that expectation was audible. I don't get the impression that they're being told very much, but maybe that's literally a consequence of the "communication issues."
edit: we're on approach into Seattle, so apparently we were one of the happy cases of the "case-by-case basis."
Secretary of Homeland Security just assured a press conference that all planes in the air will be dealt with on a case by case basis. Some may be prioritised.
I'm seeing some diversions on Flightradar24. E.g. DAL2195 from Tampa to LGA was diverted to Harrisburg. But some flights are continuing to their original destinations.
You are enroute ? Good thing Starlink still works.
Oceanic traffic would be over HF radio (or CPDLC) and not affected by this - until you get back over land and switch to VHF radio.
Most airliners over the ocean also have inmarsat satphones for the pilots.
Indeed, on both counts!
Or Panasonic or Viasat/Inmarsat :)
What network does the Panasonic run on ?
I believe they lease Ku band transponders from various commercial GEO satellites, think Eutelsat, Telesat, SES etc.
I hear Newfoundland is lovely this time of year
On a domestic flight that just landed normally, FWIW.
get a hotel room before you can't
We were offered $1500 to get on a later flight and I’m so glad I didn’t get to the counter on time to take them up on it. Saw the news reports a half hour before landing.
Yeah but imagine the guy who took that offer a couple of hours later. He was the real winner.
Same here in PIT. Ubers are as much as my flight.
Maybe I’m misremembering but hasn’t this only happened like, twice in recent memory? 9/11, and for like two hours two or three years ago
Yes
Yep those are the last two. In 2023 it was because of a contractor deleted some file and caused NOTAM to go down, and 2001… well we all know what happened
I believe those are the only other two at all.
There were three NORAD exercises (Sky Shield I, II, and III) in 1960, 1961, and 1962 where all civil aviation across the US and Canada were grounded for several hours (the 1961 one was 12 hours, 1960 was 6 hours and 1962 was 5.5 hours). This was with several months of advance notice. These were exercises to test how effective NORAD would be at engaging Soviet bombers. While they publicly announced that they were 99% effective, in actual fact no more than a quarter of the bombers were even engaged before they reached their target. This evaluation was not declassified until the 1990s.
After 1962 there was too much air travel to continue to shut it down for exercises, and NORAD switched to smaller, regional level things that did not shut down air travel after that. But the results of Sky Shield III in 1962 were the only experience that the FAA had to work from when they did the full ground stop after September 11th. Sky Shield III had tested how quickly air traffic control could clear the sky in the event of a nuclear war- the USAF had put up several hundred T-33 trainers, flying them randomly around the US and Canada, and then at a certain point in the exercise the ATC had to get them all on the ground far away from major metropolitan centers (presumed lost in the war).
NYT reporting ground stop has been lifted
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
(Excuse my lack of composure. A bit stressed out.)
The press conference was started 10+ min late and the first 15 min was spent on blaming Congress instead of giving an update. Kinda frustrating for those who are affected and waiting to find out what they should be doing at the airport. The Q&A has been just an endless repeat of the same few lines of blame shifting. It’d be nice if they could actually give actionable updates.
As a fellow stranded traveler. How did you like the part where he said Trump was the greatest president? I hate all these clowns.
> DUE TO EXTRAORDINARY CIRCUMSTANCES AND FOR REASONS OF SAFETY
I'm sure there is a reason for it but can they not be a little bit more specific?
It seems the ATC radio system went down across the country
Does ATC run on Microsoft 365?
From what I know of the ATC system it's closer to IBM System 360 then to Microsoft 365
Microsoft 363
Give them a couple days and they can reach numerical parity with IBM there!
That's a little unfair to IBM – sure, you're just as beholden to them for life, but at least the service engineer gets to your rack before you know something's gone wrong!
In naming the platform, IBM built in some expected downtime.
@nxobject I meant litterally. The ATC system is derived from the IBM 9020 which was itself a set of interconnected System 360 systems: https://en.wikipedia.org/wiki/IBM_9020
I believe it's all virtualized today
That explains the outage. We're just in one of the 5 days of the year they don't cover...
I did a tour of Adelaide airport with Engineers Australia in the 1990s. We visited their obsolete IBM main frame, still running ATC in their computer room. They were tied to it by the certification of the software it was running. Just as you described.
This might be a serious question but it is also the funniest thing I've read all week. Thank you.
Related: https://news.ycombinator.com/item?id=50388001
Its more likely the route cause wich is Friday 13th
Microsoft 364
What backup systems exist for air-to-ground communication? Something satellite based?
There isn't really one. This is a system that's supposed to have it's own built in redundancy and must not go down.
Flight Service uses a different set of radios and can relay messages to ATC, but they've been slowly dismantled over the last two decades and wouldn't have the capacity to step in for an outage (nor should they be expected to - that's not their purpose).
Some aircraft have CPDLC for text-based communication, but it's asynchronous and you can't use that as a replacement for voice communication in the terminal environment. Also, many aircraft aren't equipped to receive it.
Pilots are trained on lost comms procedures, so if you're in the air and the radio fails they'd fall back to those regulations. Basically all the aircraft would fly their last assigned clearance, and attempt to deconflict manually with other aircraft over air-to-air radio (similar to flying into an untowered airport).
> This is a system that's supposed to have it's own built in redundancy and must not go down.
There are contingencies for radio outages, but yeah, everything going down at once (if that's what's happening here, which I doubt) is bad.
There is CPDLC or "datacom" that can be done via satellite, but I think most planes that have it, use it via ground-based radio.
In the worst case: light guns and wing-rocking or flashing the navigation lights.
Multiple independent radios, for starters.
Technically, you do not need radios. There are many many uncontrolled airports in the United States, where you can technically fly around without establishing two-way radio communication. It’s doable, but I don’t know if that’s even a scenario pilots test in the simulator during their semiannual trainings.
Most air traffic is IFR. You definitely need radio for IFR traffic in for deconfliction in most of the world. (There's a few exceptions in very remote places, but that's mostly academic.) Even legacy procedural (non-radar) separation assumes you have working radio, and a radio failure is an emergency.
VFR traffic doesn't necessarily need radio, and is not affected by this groundstop.
You don’t need radio for IFR (as the whole point of a flight plan is that if you lose contact you continue to fly your plan). But you kind of do need radar.
You actually don't need radar, and ATC can do non-radar "procedural" control where they block off sections of airways or entire airports, one-in one-out at a time, like sections of railroad track. There are still a few parts of the remote US, as well as large sections of Northern Canada, where radar coverage is nonexistant and ATC is done via procedural control. Radar outages also happen on occasion, which would necessitate this.
As for lack of radio, this would only work if ATC blocked out your entire route from start to finish - which isn't practically ever done. Otherwise, they need to be able to maintain positive control over your flight since the airspace isn't reserved, which means conflicts are possible.
If you lose radio communication, it's an emergency and they'll move everyone out of your way. This doesn't work very well if everyone loses radio communication simultaneously.
Right, but isn't a lot of that necessary radio aircraft-to-aircraft, like ADS-B? Obviously not ideal, but I thought IFR traffic could feasibly maintain separation without ATC help.
ADS-B can't legally be used to maintain separation. It's only approved for use as an aid to awareness for additional safety.
Could you in an emergency, if you had no other options? Yes. But ATC can't legally give you a clearance that was dependent on using ADS-B for separation.
(Similarly, if there's a potential traffic conflict, and you tell ATC you see the other traffic on your ADS-B display, they just have to disregard that statement and assume that you can't see the other aircraft at all.)
Interesting, thanks.
AM Radios are literally everywhere, and very commonly available.
I would hope that airports keep a battery operated radio + antenna in case of emergency with them, to at least communicate with aircraft within visual range of the airport.
I would bet the problem is not strictly the radios themselves, they will still work, Planes can talk to each other and directly to airports, But that there is a large scale highly integrated control center network that failed, and planes can no longer talk to centralized traffic control.
So it falls back to local control, more in the style of visual flight rules. Pilots in existing flights will have to be vigilant and do their own deconflictation and new flights are banned out until the network is back up.
All conjecture at this stage of course, but all the radios won't just fail. It is probably going to be a poster child example about the dangers of large scale integration. The attraction of course is that integration is very efficient, the danger is it tends to fail all at once instead of in smaller pieces.
I'm wondering if this is another case of delayed Y2K, where the system suddenly thinks the century part is 19 instead of 20 for a two digit year of 26.
Based on what? Just wild speculation or something you've actually seen about the systems involved?
ATC radio is unencrypted analog for safety and communicability reasons, this shouldn't matter.
Am I missing something notable about today's date?
Friday the 13th
Just random speculation without having any real information to go on at the time that popped into my head. There have been various other instances of delayed Y2K, just low profile mostly, rearing its head throughout this decade.
Yesterday I was flying out of lovefield in Dallas and the radios went down. We were stuck for about 10min only thankfully.
Radio is not a nationwide _system_ that can go down. ATC just uses basic AM radio for radio comms.
Sure, but these are no longer connected to the controller's microphone via a long cord anymore.
Crazy and stupid but true.
Back in the day they had long cords that went all the way to the planes, right up until the Great Plane Tangle of 1967.
They use third party telecom providers to relay communications between ATC “centers” and remote radio transmitters. It is not purely AM radio and outages have occurred in the past, most recently in September.
https://www.nbcdfw.com/news/tech/a-calamity-disrupting-hundr...
That is true but nowadays the control rooms are separate from the actual antennas / areas being served. These are wired up with more modern comms.
"modern" -> "we stuck digital computers into a purely analog pipeline to make sure it's capable of a wide range of harrowing, incomprehensible failures."
> "modern" -> "we stuck digital computers into a purely analog pipeline to make sure it's capable of a wide range of harrowing, incomprehensible failures."
"we stuck digital computers into a purely analog pipeline, now with software containing a near-infinite amount of lines of code generated by LLMs, to make sure it's capable of a wide range of harrowing, incomprehensible failures."
And everybody here on HN applauds.
I don’t think that it’s the wrong idea in general. AM radio (and humans sitting on either end) is going to struggle with the sheer amount of air traffic we have today.
It's weird just how antiquated air radio is, compared to on boats.
Digital Selective Calling is a 1980s technology. So it's not even XML stitched together with Java, it's way older than that. But it does mean two dozen boats and three shore stations can all communicate with each other without constantly stepping on each other or having to wait and hope you can talk to the person you want to before bad things happen.
There are fancier more modern systems in place for a lot of transit, but DSC is relevant here because it's an international system, a random cargo ship from Korea which turns up in the English Channel has the same DSC system as the boat which just arrived from Rotterdam. This is the same requirement we need in the air, a Singapore airliner and an Avianca airliner coming into San Francisco both need the same radio system.
Radar multiplexer pipeline, or some component of it, is likely down.
https://www.sunhillo.com/wp-content/uploads/2025/04/SUN3125-... (page 3, bottom graphic)
(all the radars --> multiplexer [sadness here] --> radar data consumers)
I got this just now and seems like you know about this area
Attention: National Radar Outage
https://www.weather.gov/afc/radaroutage
"This page is no longer maintained or updated"
Thank you, that has changed.
Like the freezers in Europe?
No. Completely different.
Perhaps https://arstechnica.com/ai/2026/11/faa-expands-ai-advice-to-...
This message was pulled from a ATC communications system ordering the stop. These are terse direct communications for professionals who don't need or have time to read the story of what's going on. The news, not having any initial sources besides this is just reporting it, at the time they had no more information.
Like if you're ordering a pizza they don't need the story of your day, why you chose pepperoni, or what movie you plan to watch. Somebody else might be interested, but not the pizza guy.
I wonder if this will close the Starship launch window starting at 5:45pm EST. The FAA issued the commercial launch license earlier today. That launch is under a separate set of "commercial space rules", but presumably they need to coordinate with air traffic control.
Update: Starship launched on schedule.
otoh, the reason they normally need to communicate with atc is that there normally are planes
Quote from Musk right now: launch all the rockets!
But I am le tired...
Then have a nap.
Gotta say man, I don't think that's gonna be a top priority for most people
By most people do you mean the FAA?
You’re right, of course, but I share parent’s concern. Really looking forward to this launch!
tbh it's a bottom priority for me
I know it’s not a popular opinion here, but the commercial space industry has taken all the fun out space for me.
What's the point of this comment? The parent didn't say it was a top priority. He was just expressing curiosity.
Because it's reasonably easy to assume the FAA now has a bigger thing to focus on across the country today.
Maybe not the launch but I wonder about the booster catch. Planes probably know to stay out of the area already.
Countdown continues: T-20m, according to the official SpaceX livestream at https://studio.x.com/embed/broadcast/2brajgZlOKRMka
I mean they should already be in a situation where no planes are scheduled to fly there, and now there will be even less planes. So seems like an ok situation for them.
I guess if some plane needs to do an emergency detour threw threw that space, but that seems unlikely.
https://www.fly.faa.gov/adv/advADB
edit: #121 does load eventually:
https://www.fly.faa.gov/adv/adv_otherdis?adv_date=11132026&a...
I understand nothing
https://www.pbs.org/newshour/amp/science/spacexs-starship-so...
Botched rollout of AI for ATC announced yesterday as rolling out nationwide?
> The agency wrote on Nov. 10 that after the D.C. test, it will “now use SMART in the airspace of all 22 centers across the country to reduce delays, reduce fuel consumption and improve the travel experience for the flying public.” The tech was already being deployed at some airports.
https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
However, the National Air Traffic Controllers Union (NATCA) said it has not been “involved in the design, testing, or implementation of SMART and therefore cannot speak to its effect on the air traffic control workforce.”
“NATCA believes any new technology should complement and not replace the experience, training, and professional judgment of air traffic controllers who are responsible for the safety of the National Airspace System,” the union said in a statement.
Smacks of: I built a fully functional MVP now just need a co-founder to sell it.
That seems like pretty important context actually, and is a surprise to me.
Seems like there are three plausible causes right now given the extremely limited communication from the govt.
1. The AI system rolled out yesterday had more access than publicly claimed and managed to break things.
2. The ongoing large scale equipment upgrade went wrong.
3. General incompetence from a department run by an unqualified realty show actor, within an administration that is actively hostile to expertise and experience in the federal workforce.
"SMART is a cloud-based system that centralizes 200 data streams – including weather patterns, flight paths, traffic flow, and controller staffing metrics – into one platform."
What could go wrong really...
Just generate more slop on top of critical legacy system and overload it. no mistakes.
This is truly, absolutely, irrevocably one of the most frightening things I have read in a long time.-
Option 4 would be internal discovery / knowledge of a new vulnerability with a perceived need for immediate patching. We're on week 3 or 4 of every IT department across the country having to patch pretty significant newly announced vulnerabilities across many systems. It's not unreasonable to think that there are vulnerabilities that the government has advance knowledge about that they're quietly patching in their systems before the vulnerabilities go public.
I don't see why they would care about the NATCA's evaluation of the system. As a union, they have a vested interest in protecting the employment of traffic flight controller workers more than public safety.
Even assuming they don’t care about public safety at all, their members are still going to have to work with it, and happen to be among the world’s leading experts in air traffic control, weirdly enough.
To me, it’s like asking the hand weavers worker union about the benefits of incorporating mechanical weavers in their workflow. They are the experts of weaving, but still…
From the very first line of that article:
> The FAA has reassured the public and airlines that SMART only suggests routes and doesn't affect air safety.
That would make me feel better if we could trust a damn word from this administration and I wasn’t on a plane waiting to land.
The good news is this outage is not impacting approach or ground control so you’re safe.
Except some airports haven't enough free gates for disembarking as planes that wanted to take off were denied. So they tried to go back to their gates. As a result some planes in the air are being diverted to less-busy airports.
Sounds safe, just extremely inconvenient.
John Wayne, a class C (the approach mentioned above) isn't even letting VFRs go. If it's actually limited to ARTCC I don't know why that would be.
Thanks, bud - very reassuring that some minimum bid contract shop built something on a “secure” network they can’t even keep up.
It's also the corporations that are not exactly telling the truth & funding the administrations, just FTR...
The destructive & arrogant behaviors won't be reigned in properly if we all fail to recognize that, just FTR.
If it's working correctly, anyway
Until it got out of its sandbox…
Sure, but the tool needs to get information from the communication systems, and if it's badly written maybe it could overload some system components.
That's just a scenario off the top of my head, it could be a lot of stuff. Plugging something new into an old, complicated system isn't risk free.
From the NYT article linked:
“Mr. Mullin said that new technology upgrades overloaded the existing, antiquated system, “overwhelming” the existing equipment.
“It’s going to continue to fail,” Mr. Duffy added. He also outlined many of the additional investments he believes the air travel system needs, like new air traffic control towers, infrastructure and communication networks.”</i>
Of course they are all going to insist it has nothing to do with SMART, that would be incredibly damming, and threatening to ASI. But it’s pretty unfortunate timing no? What other “technology upgrades” have been recently applied to air traffic control/ communications?
If only routes didn't affect air safety.
That system just makes suggestions and isn't actually integrated in any ATC system. This is a telecom issue.
Its more than likely that system is just created by some Trump donor and was purchased by the FAA to launder public funds to some company that Trump owes a favor too. Its not actually integrated with anything.
Perhaps it's using the same communications channels and has overwhelmed them?
Hosted in Azure?
SMART is a cloud-based system that centralizes 200 data streams – including weather patterns, flight paths, traffic flow, and controller staffing metrics – into one platform.
yeah... sounds about right.
> That system just makes suggestions and isn't actually integrated in any ATC system. This is a telecom issue.
Is there any evidence for this beyond a single statement from the govt? This administration hasn't exactly developed a reputation for fully accurate and truthful reporting.
> Botched rollout of AI for ATC announced yesterday as rolling out nationwide?
Botched rollout of AI for ATC ...
or
Rollout of botched AI for ATC.
It sure didn't that long after every dev on earth started using LLMs for everything to turn to shit, so one may wonder...
Have they estimated how long it will take before it saves more than it costs?
Has anyone in the history of government estimated that for anything?
I know you're asking rhetorically, but... yes.
Yes, ROI estimates are very common in all kinds of government spending.
Right, they do the ROI calculation and then ignore it, and do whatever the favored lobbyists/campaign contributors said to do in any event. Government spending in a nutshell.
The Trump twist is being brazen enough that those who get these contracts can just be his family members or buddies, but otherwise, pretty standard.
What is your source for the claim that most government ROI estimates are ignored?
The federal government has the CBO (https://www.cbo.gov/about/overview) and the OMB (https://www.whitehouse.gov/omb/, warning: Trump pop-ups). California has the LAO (https://lao.ca.gov/About). Estimating fiscal impact of various policies is what those organizations do.
What's the ROI on the California rail project?
The proposition for HSR requires the California High Speed Rail Authority to issue those reports with oversight from the Office of the Inspector General. https://hsr.ca.gov/wp-content/uploads/2026/07/OIG-HSR-Review...
If your actual point is that it is costing too much and taking too long, you would be interested in Proposition 45, which limits the ability of local governments and individuals to endlessly delay the project. The LAO's analysis of that proposition is in https://lao.ca.gov/BallotAnalysis/Proposition?number=45&year...
SMART developed by ASI and the FAA is pushing to own a portion of the company:
"The FAA in June awarded Boston-based startup Air Space Intelligence a contract — worth up to $875.9 million — for an AI-powered tool to help air traffic controllers manage the nation’s skies and a related system for an FAA nerve center in northern Virginia to better handle plane congestion. The AI platform’s initial launch in the Washington area has sparked concern among Democrats in Congress who say it should be put on pause until they have a better understanding of how it’s being deployed."
"Over two dozen lawmakers have called on the FAA to pause its 90-day evaluation period for the platform, which began late last month in the Washington region. They say there’s been an “egregious lack of transparency.”"
https://www.politico.com/news/2026/10/15/faa-stake-ai-compan...
Feels like somebody was pushing their preferred company for a government contract and now is push for the government to invest.
NYT: "In a social media post on Friday, the company [ASI] said, 'The current ground stop is not based on or a result of the SMART system, any product of ASI or our integration with FAA and ATC.'"
Just like MS didn't fuck up outlook with AI. I'll hold judgment on what actually happened until 6 months from now.
In the "Don't believe anything until it is officially denied" department
Wait, the FAA wants to own a portion of the company? As in, the government believes it had the authority to own the means of production?
Is this more common than I realize in US executive deparyments and agencies?
I took it to mean that somebody in government is pushing to invest public money in this private company.
Invest, or spend?
The current administration has taken govt stakes in over 40 companies. I don’t have a link handy but I believe this is unprecedented for the US.
What do you mean? The Trump administration is the most socialist since the the 30s, seriously.The government now owns dozens of large companies and is giving individual citizens shares (with those "Trump accounts" for kids etc.)
Oh I don't disagree with you here, I just hadn't realized the government was actively taking ownership stake in companies (beyond the broadly reported Intel move).
I've felt for most of my adult life that both parties have similar goals with decidedly different implementations. In this case, both want socialist combination of state and industry, even if they disagree with how to reach that goal.
There is a name for what happens when governments and corporations merge into a functionally singular entity under authoritarian leadership. And it’s not socialism.
The irony is rich here. Pot meet kettle. Socialists are a type of authoritarianism too and nationalization is very much a socialist policy.
This might be the first case in history that an american uses socialist when they mean communist, instead of the other way round. It is communism that steers towards goverment ownership of companies; socialism would involve ownership by trade unions.
The commanding heights of the economy.
It would be super awesome if I could hop on an airplane without getting my insides x-rayed, my belongings sorted through or damaged, worrying about the windows falling off or the plane's systems flying itself into a crash, or some tech bros' bid to be a billionaire telling what an understaffed ATC controller what to do.
The whole setup is worrisome.
The AI prompt misunderstood 'grounding'.
Well done.
"You're absolutely right! I misunderstood the intended use of the word taxi. I made an assumption, and then ran with it. Would you like me to open a PR to start flights again?"
"You're right, you were clearly referring to electrical current grounding, my mistake!"
AI tech is nowhere near reliable enough to be introduced in such important systems. I find this deeply troubling even if it’s not related to the current issue.
Curious where the people are who said AI will be doing x in y months (for their valuations) - it never will. Where are they hiding?
Personally I fail to understand how we could possibly benefit from a non-deterministic AI system, when normal computer systems are more than capable of managing a 3-dimensional airspace using nothing but basic math. Smells like pork for sure :/
We can use Chinese models? (I am referring to the LLMs).
I see what you did there
I think it depends on what it’s doing and if backup systems exist. Scheduling and prioritizing certain flights for landing seems like a reasonable application as an example. Anything realtime dependent or near realtime it shouldn’t touch. But there seems like lots of less critical applications it could provide some value.
Why make any of that probabilistic at all?
Fasster response incorporating more varied data sources more easily
because theyre mathematically introducing error/nondeterminism for money
They didn't check https://shouldideploy.today before the rollout (edit: as UTC rolls past midnight, https://shouldideploy.today/?tz=Pacific%2FHonolulu to get a longer duration for today's message)
"As an AI I was told to reduce delays and fuel burn. Can't have a delayed departure if there aren't any departures."
Is this that skynet thing everyone talks about?
> An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage, multiple sources told CBS News. The system is operated by L3Harris, which did not immediately respond to CBS News' request for comment.
Seems related to the rollout of their new AI tool? [0]
[0]https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
> The FAA has reassured the public and airlines that SMART only suggests routes and doesn't affect air safety.
> [The] FAA also says a D.C.-based team of experts validates findings before they’re ever passed on to air traffic control, and the controllers retain the final decision on whether to accept SMART’s suggestions.
Seems unlikely.
Depends if an AI DoSed an important service or not.
There used to be a lot of invisible gatekeepers in a pre-AI world. At work, everyone can ask their agent to query SQL on databases that they were supposed to use through an application. The application and the security by obscurity(end users didn't know how to connect to the DB nor did they know SQL) were the gate keepers.
That was my thinking as well. AI sending out requests to legacy stable systems is often not a good mix and can overwhelm things pretty quickly.
Awkward timing, to say the least.
Those engineers are no doubt panicking, even if they aren't the cause.
While I don't have insider knowledge of this particular deployment, there was a strong industry-wide push towards "ship unreviewed AI generated code and hope for the best", partly driven by the fact that AI writes code faster than humans have bandwidth to properly review.
Under the hypothesis that some under-reviewed AI generated code made it into prod, the situation could be qualified as "engineers are panicking, because they don't know if they are the cause or not"
Doubtful. That tool likely suggests "improved" clearances, which a controller can use at their discretion. It likely does not control clearances directly.
And the early reports describe the issue as related to ATC communication infrastructure.
I do not really understand what the FAA means by “inbound international flights will be managed on a case by case basis.”
I am watching planes stacked up on final approach to land at KBOS as I type. From flight radar 24 they are a mix of foreign and domestic. So incoming flights are clearly being managed here.
Does this imply some potential risk from inbound international air traffic? Or maybe the FAA meant outbound? Something else I am missing?
I suspect they mean that inbound flights from international airports may still take off (probably in part because the FAA can't _really_ stop them, in part because the volumes are much lower and thus more manageable, and in part because one might hope that by the time it takes to get to the US from many international destination, the incident will be resolved).
Very unlikely that any international flights bound for the USA are taking off. Probably many are turning around or diverting if they are not more than halfway there.
I'm not following this very closely, but I've not seen any reports of diversions for domestic flights, so I think it'd be unlikely for international flights. The report is just for a ground stop, which is just a stop in departures; normally a ground stop would allow flights already in the air to travel to their normal destination.
That reasoning doesn't really make sense. A flight from outside the US that is inbound could divert to a non-US airport that isn't impacted by whatever is going on, depending on where they are and what their fuel status is. A domestic flight doesn't have a non-US airport to divert to.
(Sure, a flight near the Canadian or Mexican border could in theory divert to a Canadian or Mexican airport, but it's unlikely many people on those flights have their passports with them.)
Ah, my point was more that if the action being taken isn't "get planes on the ground _now_" and so the US airspace remains open for traffic already in the air, then it's unlikely they're trying to get international planes to either divert or return.
This definitely feels like 'unable to manage normal volumes at normal speed' rather than 'unable to manage anything.'
In which case freezing domestic (and the huge amount of load it contributes to ATC) but keeping international normal makes sense.
Of course... the side effect is the "fun" that creates for incoming international flights with domestic connections...
> (Sure, a flight near the Canadian or Mexican border could in theory divert to a Canadian or Mexican airport, but it's unlikely many people on those flights have their passports with them.)
There must be some procedures in place for this - I pretty regularly take domestic (Canadian) flights that pass over the US where I don't carry my passport.
A situation kind of like this happened to a relative in England; their international flight was diverted (I think due to weather) to a domestic-only airport; due to lack of immigration & customs facilities, everyone had to remain on the plane until it could take off again (presumably after being refueled) to get to an international airport
A domestic US flight can still divert to Canada. There usually are special arrangements that allow people on a diverted flight to stay short-term while the situation is resolved, even if they don't have the required documents or visas.
they let eithad take off from boston like 40 min ago
That is interesting and why i was wondering if they meant outbound flights but screwed up the language. I had assumed (probably wrongly) operators who should be risk averse would not want their US bound flights taking off under these conditions.
And I assume it becomes case by case at some point because they eventually start running out of space on the ground at airports.
Presumably it means they may allow some international flights to the US to depart if they expect the issue to be resolved by the time they are in US airspace.
It likely means any inbound international flights can be told to turn around and/or land outside of the U.S. instead.
The intent around the ground stop is to reduce in-air U.S. traffic.
Makes me wonder if we'll see another "Operation Yellow Ribbon".
https://en.wikipedia.org/wiki/Operation_Yellow_Ribbon
Alas, the Canadian goodwill level may not be as high this time.
Canadians, like Americans and practically most people in the world would band together to help folks in situations like this and do so with warm and loving hearts.
Doesn't seem like it; I've seen several international flights land in JFK alone since this started. (On the other hand, these might simply not have enough fuel left to divert to any non-US airport.)
Not in the industry, but I think ground stops might actually not be a regular/automated procedural thing for inbound international flights in the way that they are for domestic flights.
As for asking already airborne flights to turn around, those coming in over the ocean might simply not have a non-US alternate airport available with their fuel remaining, so that's often not an option.
Every flight over an ocean has a set of pre-planned diversion airports.
Yes, but most of them are not viable in the late stages of a flight. Planes aren't carrying enough fuel to cross an ocean for a second time; most don't even have the capacity.
The diversion airports are chosen specifically for each stage of the flight, so a flight doesn't ever need to return to where it came from.
Oceanic flights are planned so that at every point along the route, the aircraft can lose an engine or cabin pressure and still reach a pre-approved diversion airport with enough fuel, with the flight plan marking the points where turning back versus continuing becomes the faster option. If a diversion is needed mid-ocean, the crew offsets from their assigned track and altitude to stay clear of other traffic, then heads to the nearest suitable airport, such as Gander, Keflavik or Shannon over the Atlantic, or Anchorage, Cold Bay or Midway over the Pacific.
Please suggest an appropriate international alternate for a plane just about to land in Denver with an hour of fuel remaining.
Alternate airports are picked based with regards to all kinds of contingencies (many of them meteorological), but "the US closes literally every airport" is not one of them.
That likely falls within the case-by-case basis the FAA mentioned, and would likely be allowed to land at a U.S. airport that has capacity.
And that's exactly why you got the reply that international flights MIGHT not have enough fuel left to divert (and thus obviously have to be allowed to land within the continental US), but for some reason you insisted that every flight would be able to.
I'm not aware of a requirement that you need to always be able to divert to a different country.
I think it's more mundane than that, I assume they mean that certain international flights will be turned around or redirected while others will complete their route. Say a flight just an hour underway London to New York might be turned around while one nearly at its destination may be allowed to land.
Depending on the nature of the disruption it may be they simply have drastically reduced capacity to handle traffic so landings may be much slower and they'll manage the traffic based on various priorities.
Probably just means that if you're less than halfway across the Atlantic, you can turn around and go back to Europe. I really wouldn't suggest reading too much into the phrasing.
An Eithad flight took off from KBOS after the ground stop
Flight EY8 from Boston to Abu Dhabi https://fr24.com/ETD6PX/429055e2
I've seen some flights destined for Europe taking off from ATL and BOS in the last 30mins so some exceptions are being made it seems.
friend is flying from London to NYC and they were told they might need to land in Toronto.
So my guess this is what it means "case by case". Some which are close to destination would be manually routed, others will land in different places.
Exactly. London to LAX may land in New York or the first available airport, or may route somewhere else.
Some might be turned back, depending on how far they are in the journey, fuel, etc:
Maybe they mean inbound international flights may be allowed to take off from their origin, on a case by case basis. Usually a clearance is required to take off from both the origin and destination.
like you don't have enough fuel to fly back across the ocean they'll accommodate you I assume?
My wife is on a plane on the ground at ORD (they just landed) and they have been told they can use the restrooms but can't get off the planes. Any reason why folks wouldn't be allowed to deplane at the end of the journey if it was a radar or comms issue? They can pull up to the gate using flags . . .
Is it something else?
not enough gates
Their gate might be occupied by a flight that can't leave.
Not to mention they won't have enough staff to operate the gate equipment for all gates at once.
This seems like one of those efficiency-vs-resilience problems. Sure, it's "more efficient" (in cost) to use the same empty/fill process for every plane, but when nobody can leave, some extra capacity to unload planes would help.
(i.e. Unloading to tarmac, buses)
You don't have to go to a gate to deplane and this is an extremely rare event. They'll just pull up a staircase and have them deplane to a shuttle eventually.
Most U.S. airports aren't equipped to deplane commercial airliners using stairs, especially at this scale.
You empty the planes at the gates and then push them back to parking (or park them on the runway if needed).
O’Hare is under construction. They’re building an entire new terminal and are operating with a lot less spare capacity while that happens.
Standard protocol. Doesn’t matter what the issue is, you follow it
Doesn’t mean you wouldn’t still want to know why.
We are 6th in line waiting for a gate while they tow grounded planes off of the gates. Normally those flights would have departed and we would get a gate fairly quickly, but if no one is taking off, then they need to move them, and potentially deplane the (angry) departing passengers.
waiting on the stairs since all the gates are full
The only time I've spent long stretches on a grounded plane have been at ORD. It's such a busy airport than any disruption to the schedule just causes all sorts of chaos.
The reason field is empty, but the website footer provides enough information to understand.
And yet, it seems to have been a careless update to have broken everything
probably updating winzip
They tried to register it
Shoulda used winrar.
Do we have any indication of what went wrong? The incident report will be very interesting.
A botched update, allegedly.
L3Harris Update
my buddy used to work at L3Harris. It sends shivers down his spine whenever i mention it.
my buddy used to work at L3Harris. It sends shivers down his spine whenever i mention it. He's gonna love when I dog him on this one.
Transportation secretary saying that their networking routers were being overloaded by increased communications traffic. But its not a cyberattack or software failure.
Speculation on my part, they just rolled out their SMART air traffic routing system. https://www.flyingmag.com/faa-smart-ai-now-active-at-22-artc...
Bet the increased strain on data polling from this system is the culprit for increased traffic.
Sounds like a software failure to me.
maybe, what is confirmed is that they were trying to upgrade some physical TDM switches with IP routers (as they have been doing for a while), and the "new systems overwhelmed the old ones".
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
They just said in the press conference that it couldn't possibly be related to SMART, since the systems were disconnected, but the timing and lack of a concrete answer as to the cause or likely cause, combined with asking for money from congress (which currently has a republican majority...) for "more modern infrastructure" has me doubting their version of events
I acknowledge this is borderline conspiracy. Sec Mullen's statement is that FAA was quote "installing new equipment, new technology and that the new equipment new technology was overwhelming the system"
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was...
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
> I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was
They did hurriedly mention it was "new patches" in the midst of their deranged ranting and finger-pointing (at...their own party, a point Duffy was visibly taken aback by when told by a reporter). When they vehemently insisted it was 'not SMART' one has to presume they meant that the hasty execution of their haphazard AI rollout was performed unintelligently.
Was this rolled out on a Friday?
In the press briefing (https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...) it was just stated that new "patches" "overwhelmed" the "legacy system". Sounds to me like the new SMART AI rollout went haywire.
At the press conference (current, live), they are blaming old hardware that is failing. They’re in an endless loop of repeatedly saying that it’s Congress’s fault for lack of funding and that this is the result of old hardware. I just gave up on watching because they aren’t saying anything informative.
I'm used to bizarre press conferences from this administration but this is something else. Just baffling.
They have no idea, they stood up there and lied and played the blame game crying about funding. Completely shameful.
"It's the safest system in the world, I mean, uh, except that it's horribly outdated with switches from the 80s and radars from the 70s, so we really need 50 billion dollars from congress STAT or we're in big trouble, but it's completely safe. Also Congress needs to give Trump the Builder this money _now_. We couldn't get it in the past couple of years even though we controlled the entire government. I mean, sure, we got 12 billion, but that's not enough to unfuck what Joe Biden did. Your loved ones are completely safe by the way, but we can't promise this won't happen again."
It’s amazing we aren’t all dead, given how stunningly incompetent these people are
I'm sure they couldn't figure out the process of procedures required to really inflict harm so they went after immigrants based on their data. Following a data trail with Claude is easier than actually governing.
I think the supplemental appropriation that the FAA got as part of the OBB is probably too big to be spent. Unless you just hand it all to a contractor, it takes time and oversight to spend billions. Legally and practically.
and the journalists haven't yet picked up the actual reason that they revealed among the lines
The press should be asking why this could be the case when the GOP has held congress and the white house for two years ? Just what was the party in charge thinking ?
i misread this as saying you were in attendance of the conference and was hype that the press uses hackernews
https://www.youtube.com/live/Mh2q-y0wIu8
Ah sorry for the misleading writing. My brain has been fried trying to work while also trying to dig up any useful info to help my wife who is stranded at the airport figure out what to do. I usually can last longer watching press conferences but this one was… just painful. Maybe someone else can give an update if they actually talk about what’s going on.
ive also been watching it, pretty much repeating the same stuff
> old system give us money
> probably not a cyberattack btw
> not SMART related
> we've opened the airspace but this is gonna be slow
just ended actually as i was typing this, that's pretty much it
> Maybe someone else can give an update if they actually talk about what’s going on.
They didn't.
Having worked in Government for a spell. Are we actually sure that old hardware isn't the problem?
I didn’t say it isn’t. Just that they aren’t really giving any details and more importantly, they aren’t giving info for what people who are stranded should be doing right now. (And now that I’ve settled down a bit, it’s odd that if it’s old hardware failing, how is it suddenly not failing? Was it just some systems, did they have replacements, or if it’s what the other commenters wrote, then did they just turn off the AI updates to stop overloading the hardware? Which would mean that hardware is fine. It’s just underprovisioned.)
I'm being a little bit tongue in cheek, I'm sorry your wife is stranded.
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
It sounds like it has nothing to do with AI at all. Why does everyone keep jumping to AI when there's no indication this has anything to do with AI?
Because everyone [in the specific field of technical professional operating systems like this] is using AI and not everyone is smart about how they use AI?
System worked mostly fine for decades, then in the middle of integrating a new AI system it completely goes down.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
There is zero evidence of this at all. The stuff going around on social media has zero correlation to the system in question. Unless you have other evidence others are not privy to?
AI is the name of the witch that everybody is so eager to burn
> I expect it to be a combination of extremely poorly funded
Highly doubt this. Unless you mean sort of like education in many areas where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
> where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
Yeah of course I mean this, I mean the guy who had to serial connect to whatever shit show was running and was sitting there trying to parse error logs.
Hardware can be fixed. Even if it is old. Spares exist for a reason. Maintenance staff exists for a reason.
Up to a point. If your equipment is decades old, spare parts and expertise is a quickly diminishing resource.
Not with the budget the FAA has to apply to the problem. Yes, you have a point. But the importance of this system could get circuit boards manufactured to old drawings. Specific 'electronics boxes' replaced with new ones designed to replace the antiquated hardware.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
> they aren’t giving info for what people who are stranded should be doing right now.
Do you really need the government to solve that problem for you? Just book another flight.
Good luck "just booking another flight" when half a million people are trying to do exactly the same thing at the same time as you.
Well, sure, that goes without saying. You just don’t need the government to tell you what to do; you can work it out yourself.
It's certainly possible - but for whatever reason this administration (with the help of that guy from the robot taxi company) seems to not like air traffic controllers and has been trying to find ways around them.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
Given that the old hardware has been around a long time without a problem like this, it seems very odd to blame the old thing for the new problem. My money is on a poorly planned transition, one where they didn't take the old hardware seriously enough.
They were also insisting its not a problem with the new upgrades they were installing, but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system"
(https://news.ycombinator.com/item?id=50492957)
I assume that this was a boring, routine update that went awry.
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
>but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
Not only was there a lack of useful information (other than the fact that the grounding was lifted), it was an astounding demonstration of poor leadership during a crisis. No ownership, no focus on the problem/root cause/next steps.
I'm pretty sure saying that Congress needs to give them the money to replace the ancient hardware is putting ownership directly where it belongs and identifying the problem, the root cause, and the next steps.
What am I missing?
In the moment of crisis, they own the problem of making the system work with what they have.
As a postmortem, once the dust has settled, is the time to ask for more funding/support
That if the ancient hardware was working and just the new software they tried to put on it did not worked with that hardware - it still means they messed up.
Let's be clear: it was a bunch of idiot political hacks being unbelievably obsequious to Trump. So embarrassing for the country. I don't trust that the ask for $50B is anything more than grift for their friends. Was the grounding a purposeful overreaction to increase odds of getting money?
I don't doubt that the system needs upgrades, but these people should not be involved.
Markwayne the plumber correcting a reporter for saying AI instead of SI instead of answering the question was utterly surreal.
This is one of the reasons for the "SI" mandate.
I thought you were being a jerk, but no this dude has an Associate in Applied Science in plumbing from the Oklahoma State University Institute of Technology. Huh.
Don’t even get me started on the one nobody gave a shit about on Real World: Boston standing at a podium and demanding $50 billion.
How are we all not dead? These people are clearly stunningly incompetent
Sec Duffy's presentation was just embarrassing. Blame anybody else except for the people in charge.
Usually I refuse to give any detailed postmortem right after an incident is mitigated/resolved until I have had a chance to collect my thoughts and make sure I can write a clear problem/root cause/next steps. The communication that goes out with the mitigation signal is often short and vague and the detailed postmortem follows a day or so later.
Have they even thanked the pilots, air traffic cobteollers and thousands of airport personnel for handling the mahyhem?
No, but they made sure to thank the Dear Leader several times each.
There's still time for that. The mayhem is ongoing and will be felt for days. I hope by Monday a few "bosses" will say thanks to all the staff who ran overtime to deal with desperate grounded passengers, do all the rebooking snd schedule changes.
They are trying desperately to not name the AI system that got turned on yesterday that they commissioned.
Is it more likely that decades old hardware failed coincidentally or that the first rollout of a new AI tech went wrong within a number of hours because the test environment didn't have sufficiently scaled load testing?
So they were doing a network upgrade and putting in new equipment. This new equipment was misconfigured and caused the existing working system to stop working. And now they are saying in the press briefing that this is a problem of the existing working system which they call the "legacy system" and are using this as a political move to call for the congress the regime no longer controls to provide funding to the FAA/DHS. Framing the manufactured issue they caused with the new system as if it was congress' problem instead of their own breakage they caused and are deflecting from. Demanding an emergency funding bill now despite ignoring it in all the years they were in control.
There is an actual bill for $30B that's been stalled for months in the Senate called the Modern Skies Act, and there is absolutely widespread antiquation within the US air infrastructure which this bill seeks to address. So he's not just making this up. He's talking about exactly that bill albeit with all the Trump BS injected into it.
Mentioning how they previously received $15B but nothing improved - so why will any further money allocated actually end up where it's meant to go?
Modern Skies Act is for $30B, not $50B. A separate $12.5B from "Big Beautiful" is already being spent on ATC modernization, and only $10B of MSA is toward ATC
https://www.transportation.gov/sites/dot.gov/files/2026-09/M...
The One Big Beautiful Bill Act cracks me up as even the naming was shortsighted and defective.
Thanks, I corrected it, 3's and 5's get mixed up for me due to my synesthesia giving them the same color
Sure, but blaming Democrats when they’ve controlled Congress and the presidency for the past two years and their highest priority has been cutting funding via DOGE is pretty disingenuous
Their press conference was horrendous but it seemed he was just blaming Congress overall and not Democrats specifically (since it is Republicans actually holding it up)
Assuming I'm looking at the right bill (S. 5572, the Modern Skies Act), "stalled for months" seems a tad overstated considering it was introduced Sep 25th 2026, just three days before Congress went on recess for the entire month of October (elections). The Senate wasn't back in session until Nov 9th, so the bill has been pending for a grand total of seven in-session days (even emergency legislation would rarely pass that quickly).
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).
It’s possible the interfaces to the legacy system aren’t documented well and/or it could be incompetence on behalf of the network engineers working this
Agreed, but why not just say that?
Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.
Waiting for the RCCA
Soon enough, you'll be hearing calls to privatize the FAA for "efficiency" due to "failures" like this.
The beast is starving and thrashing about
The sceptics will say Trump caused the outage so the FAA get approved for the $50B they claim to need to replace the air traffic control system.
That would imply there was a plan. A series of predictable events baffle this administration, so assuming a nefarious plan would seem extreme.
Summary from the press conference:
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
Only Donald can fix it. Pay attention.
Not much info yet. Even ops.group [1], which is for people who do flight planning, has nothing yet.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something. There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
[1] https://www.ops.group/blog
[2] https://www.faa.gov/regulations_policies/orders_notices/inde...
you’ll never get a proper retro for this, it’s too political
Yesterday's Headline: "The FAA Is Rolling Out Its AI Air Traffic Tool Nationwide" https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
United just departed EWR Newark for Tel Aviv https://www.flightradar24.com/UAL84/42909d99
And several from IAD Dulles. https://www.flightradar24.com/UAL992/4290a5d2
All seem to be international though.
https://apnews.com/live/ground-stop-aviation-flights-news-up...
Flight radar is showing many flights that departed within the las hour. Maybe they are doing some controlled international departures since communication would be critical at arrivals (specially at busy hubs)
Edit: If you look now over the north atlantic you can only see outbound flights
Since the threads keep getting closed about Hawaii DoT confusing everyone by claiming SpaceX debris was contributing to their shutdown despite the Starship launch being in Texas, successfully reaching orbit, and having a booster land in the Atlantic:
Hawaii was the early/late splashdown site and pre-cleared: https://www.cbsnews.com/news/spacex-launches-starship-flight...
SpaceX released the early return around 4 p.m. Pacific: https://x.com/TrackingTheSB/status/2121392533418786992
HDOT phrased it badly, but there was a window today where Starship could have aborted over the west coast in the middle of this shutdown.
um.. you meant booster EXPLODED in the Gulf of AMERICA?
Looked like it was a soft touchdown until engines shut off at the "catch" height (simulated) and then they activated the TFS.
“Exploded at a specific time”
Exploded safely as planned during an off shore divert from the tower catch attempt, after making a controlled hover in mid air slightly above the water.
And also spewing green flame from the disintegrating copper of the engine
Connected? "US National Weather Service reports nationwide internet, communications issues" (Reuters) https://www.reuters.com/technology/us-national-weather-servi...
> A spokeswoman for the National Weather Service said a screenshot circulating online that claimed its weather radar was down was “outdated.”
> The Weather Service “has not experienced a disruption to its radar systems today,” said Kim Doster, the spokeswoman.
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
^ According to story below, they did - why is the NWS reporting 2 different things?
https://www.reuters.com/technology/us-national-weather-servi...
That Reuters story appears to have since been updated.
Wonder if this is related upstream or tangentially?:
https://www.news9.com/oklahoma-city-news/national-weather-se...
“National Weather Service reports nationwide radar website outage amid FAA ground stop.”
Given that the FAA issue also apparently relates to communications, it does seem like it might be related.
> In a notice posted on its website, the agency said technicians are working to resolve the disruption as quickly as possible.
This article is very likely misreporting the US National Weather Service radar outage page you can find people reposting elsewhere in this comment section. The NWS page has been updated to say it does not reflect a current issue. Anecdotally, my local NWS weather radar is working fine. Also, weather doppler radar and ATC "radar" are very, very different technologies.
It's from 2020: https://web.archive.org/web/20201017144706/https://www.weath...
Likely Atc system outage, saw this at Chicago
https://www.cbsnews.com/chicago/news/ground-stop-chicago-oha...
CNBC is reporting that the ground stop has been lifted: https://www.cnbc.com/2026/11/13/faa-ground-stop.html?update
https://www.weather.gov/afc/radaroutage
So strange
They say "national outage," but if I go to the site for my area, my local weather radar seems to be working.
Yes because it's not weather radar, it's communication radio
And even if it was radar-related, weather radar is entirely different from ATC radar.
I really really doubt that's related. Seems like the website just broke.
https://www.reuters.com/technology/us-national-weather-servi...
>The US National Weather Service said on Friday it was experiencing nationwide internet and communications problems that could prevent users from accessing products on its websites or cause outdated information to be displayed. Technicians were working to resolve the problems as soon as possible, the NWS said in a statement.
Are the radars utilizing a shared asset with the radio systems?
All I see here is "This page is no longer maintained or updated". Did they update it or sthg?
Yeah, looks like they just updated it.
Oh cool, I’m on a plane rn about an hour from my destination.
Edit) landed pretty uneventfully.
Hello from the ground
I can see my house from here
Gooba gaba
My wife's plane was about to take off (moving) but then stopped. I just messaged her to check once I saw the news. Sounds like the people onboard weren't given any info as to why. They're still on the runway waiting.
Physics says all planes land. I guess eventfully would be diversion (maybe to Canada lol) or Mayday Fuel (but you probably won't know it happened anyway - it just means some reserve fuel will be used).
What I find more incredible is the ability of the thousands of pilots up in the air to be able to all execute perfect individual action in the fog of uncertainty. What is the secret sauce? I say perfect because it was 100%. Fantastic job. May all the trainers continue to do what they do best.
> What is the secret sauce?
My impression is that it's the regular, mandated practical simulator training they do for contingencies most may never actually see during their careers.
Use this link for latest advisory: https://www.fly.faa.gov/adv/
Current as of 1340 PDT:
What does this mean? "DCC HAS IMPLEMENTED EXTENDED FLIGHT PLAN DROP TIMES TO 240 MINUTES FOR ALL ENROUTE CENTERS DUE TO NATIONWIDE COMMS FAILURE."
Normally, a flight plan must be filed with the FAA no later than 60 minutes before it's needed. That's been extended to 240 minutes.
So the flight planning system is having problems.
You can definitely file a flight plan later than that. 30 minutes before departure is the recommended filing time (per AIM 5-1-6), but you can even file for an immediate departure. It's just that at less then 30 minutes additional delay is possible, depending on backend handling and human processing. But in most cases the strip will print within a minute or two of submitting the flight plan.
But also, this is about when the flight plans expire out of the system if they're not activated for departure - so it's saying unused flight plans won't be deleted for 4 hours, so pilots won't have to refile for departure.
It means that flight plans (which include departure time) will stay in the system longer after the departure times are missed, so that IFR flight plans will not need to be re-filed. In other words, flight plans have a longer TTL.
This will ease the pain of getting things moving again, assuming they fix it by the time the 4hr window passes.
As of 1345, "No ADVISORY Available for today."
There's a query form for looking up old advisories, and it's not returning anything for the last 15 days.
Why do all avionic systems communicate in caps?
So they can be heard over the engine noise.
Most of these systems started out as teletype printers, back in the day. The teletype systems only had an all-caps character set.
Aviation is global, and we still need to interop with countries and airports that still use actual teletype. So the legacy protocols still remain, even though much of it is tunneled over IP links now.
Also: The keyboards used in both airplanes and in ATC facilities are often special reduced-footprint keyboards without shift/caps keys. (They're also often ABC layout rather than QWERTY.) So there's that, too.
Does anyone still use actual teletype?
Actual teletype machines? No. (Though flight strip printers used at ATC consoles are basically mini teletype printers.)
However, AFTN (used for disseminating flight plans and weather between ATC facilities) still uses landline modems running TTY-originated protocols.
In the US we've moved moved off AFTN in favor of AMHS/X.400, as have most countries - but my understanding is there's still some last holdouts that haven't fully switched over.
Probably for the reason that the original monocase teletype machines did. Better absolute readability due to the larger characters. It tends to be very dense with little actual prose so it is better to reduce the possibility of mistaking one character for another.
Also, everyone is used to reading the upper case so they would have to spend some time learning what common things look like in lower.
Update:
From earlier this week on HN:
“FAA says it will phase out floppy disks from air traffic control”
https://news.ycombinator.com/item?id=50388001
Put the floppy disk back in!
It is a very confusing situation, but one thing was revealed very clearly: Duffy must be fired immediately.
Am sitting in GRU waiting for an outbound flight. Have just seen two flights (DL268 to ATL and DL313 to JFK) change to Boarding and now Last Call status.
My flight to the US is listed as Delayed but there are two flights to Canada that show Contact the Airline. Bewildering.
And we're boarding. The Delta flights took off earlier.
Related? https://nasstatus.faa.gov/ has a notice:
We are experiencing technical difficulties. We apologize for the inconvenience but please check back soon, as we are working to have the issues resolved.
Thank you, FAA Web Operations Team
likely just a self-inflicted DDOS as the entire world starts accessing various FAA websites to try to figure out wtf is going on
hugged to death
https://www.npr.org/2026/11/13/nx-s1-6040403/faa-nationwide-...
I think this smells like a cover-up. Not too long-ago, Amazon blamed humans for a botched rollout when it turned out it was an AI tool.
I wonder if we'll start hearing less and less truthful stuff because nobody wants to admit that AI did this.
As friend from Mastodon has suggested: "So this would be a great night to play atc(6) from the BSD Games collection on NetBSD"
https://tilde.zone/@sirwumpus/117611840050306362
ATCSCC Advisory ATCSCC ADVZY 005 DCC 11/14/2026 CNX NATIONWIDE GROUND STOP MESSAGE:
EVENT TIME: 14/0116 - 14/0130 DESTINATION AIRPORT:ALL FACILITIES INCLUDED: ALL EXPECT UPDATE:0400Z REASON: REMARKS: THE FAA COMMAND CENTER NATIONWIDE GROUND STOP HAS BEEN CANCELLED. EFFECTIVE TIME:
140119-140200 SIGNATURE:
26/11/14 01:19
Why did the other thread noting it got lifted get merged into this? Now we have to go through a bunch of outdated comments to read newer ones and hacker news offers no kind of sorting. Should’ve kept up the recent post as well…
Looks lifted now? Not showing for me. https://www.fly.faa.gov/adv/adv_otherdis?adv_date=11132026&a... https://www.fly.faa.gov/adv/adv_otherdis?adv_date=11132026&a...
Edit: screenshot here: https://i.postimg.cc/XqcsDQw9/image.png Edit 2: different link still showing: https://www.fly.faa.gov/ois/oisedit/adv_oisedit_dis?number=1...
ATCSCC ADVZY 116 ALL/DCC 11/13/2026 NATION WIDE AIRBORNE HOLDING MESSAGE:
EVENT TIME: 13/2100 - 14/0300 USERS CAN EXPECT AIRBORNE HOLDING TO ANY AND ALL AIRPORTS WITHIN THE NAS DUE EQUIPTMENT OUTAGES. UPDATES WILL BE ISSUED AS ADITIONAL INFORMATION BECOMES AVAILABLE. EFFECTIVE TIME:
132120-140330 SIGNATURE:
26/11/13 21:20
Edit: end date of tomorrow??? Is that just temporary?
They spelled equipment wrong.
"EQUIPTMENT"
widespread autocorrect system failures
You know there's a fire under their asses when spelling flies out of the window.
At least a human is in the loop?
"ADITIONAL"
I think you're wrong:
https://www.fly.faa.gov/ois/oisedit/adv_oisedit_dis?number=1...
huh - strange. what's the difference between those two URL endpoints?
it says "EFFECTIVE TIME: 132054 - 140430"
maybe the last two numbers are a time range for which the thing is effective? and it's over now?
ah actually based on other comments here i believe it means 13/2054 - 14/0430. ie the 13th, 20:54 to the 14th: 04:30.
From the advisory:
13 is the date, 2054 is 20:54. So the advisory expiration time is Nov 14 04:30.
UTC?
Couldn't afford the extra bits for epoch
That’s crazy
This happened to me a few minutes ago. I clicked that link, it said "no advisory", I was making the same comment you made, then tried refreshing and the advisory came back.
Insane. It looks like they are showing no advisory on first page view, then advisory? Can others confirm?
I only see “no advisory” at the OP link. Refreshed 4 times with gaps in between.
Can confirm. It seems to be showing the advisory for me about half the time. Strange.
A/B testing, some PM is going for a promotion as long as the guardrails stay positive
Listening to ATC; likely end time 10pm central. See also https://www.news9.com/oklahoma-city-news/faa-reportedly-issu...
It seems that the adv/ pages are not loading any advisories, though I am just guessing and have no idea how this is meant to work. The fact that the search page has a date input would seem to imply historical advisories should be shown. If you go to https://nasstatus.faa.gov/ and click the Advisory links for any of the active events that have one, they all show as blank too. Could be totally wrong though.
Maybe just paranoia but it seems slightly suspicious for this to happen while we are in the middle of discussing massive funding and overhauls to the air traffic control systems. This will certainly end up pushing a lot more funding into these programs and companies now that were trying to sell more expensive systems.
I don’t think you’re paranoid to recognize that something desperately in need of funding is suffering the consequences of a lack of funding.
I guess we will see what happened in the aftermath. But I very much doubt the current problem has anything to do with a lack of funding.
And I don't see anyone arguing about overall funding, only arguing about who to throw millions or billions of additional dollars at chasing vague ideas of how to improve things based on what politicians and C-suites, who have no experience in the area, claim they can do. To me it looks more like grifting for contracts and kickbacks than any well thought out plans to improve things.
Does anyone know why HN is so intensely fascinated in air traffic control?
My father worked in various capacities for the FAA (as well as consulting internationally for various country equivalents). I tried to show him the discussion here and he couldn’t read it. I imagine it’s the equivalent HN crowd watching local news commenting on software issues.
I think in some way, shape, or form, it's related to politics and who's in charge and people wanting to be upset. But just my guess. I'm not an air traffic control expert like the rest of hackernews.
As descendants of primates who lived on trees we inherently carry a fear of falling down which was an evolutionary advantage helping our ancestors to survive long enough to propagate, leave trees and spawn sidewalk walking people like us. The fear of falling from trees easily translates to the greater fear of falling from the skies. As good citizens of the world, HN folks harness that fear energy to make our world safer. I think that’s the reason for HN folks to be OCD level obsessed with flight control.
Microsoft Copilot went down?
Flight Simulator as well.
Survival tip: in a pinch, Space Cadet will do
Press conference https://www.youtube.com/live/M1WhY3qcJjE?si=KUFvgHO-EepCi1v6
Notably, the problem was described as "routers overloaded with data". When directly questioned if this could be a cyber attack or rogue ai agent the Sec/Homeland responded "there's nothing completely certain now."
The way they are being so evasive makes me think they did a pretty big screw up.
Or a non-technical person is relating the information as they understand it to a non-technical press corps, so everything is generic and hand-wavy. Honestly, have you never dealt with non-technical C-suite staff?
>Honestly, have you never dealt with non-technical C-suite staff?
Weird tone.
NWS radar and 911 systems are also showing downtime according to https://downdetector.com/
Maybe this has something to do with it?
https://www.weather.gov/afc/radaroutage
This is (fortunately) from 2020, not current. I'm not sure why this is spreading like wildfire right now, but: https://radar3pub.ncep.noaa.gov/
Ah. Thanks for letting me know. That makes sense.
It is a frontpage post. That's where I saw it.
From the Financial Times:
> Los Angeles Airport and Las Vegas’s Harry Reid International were among those that were closed, according to a list on the regulator’s website.
> The outages at O’Hare and Midway were listed as “due to equipment outage”.
https://www.ft.com/content/c9b119ff-d1da-46a7-b2b3-4ad0544ff...
Skies over Portland are empty except for a scenic tour company flight that just took off from Troutdale, and is flying over downtown. Are there some exceptions for general aviation?
https://www.flightradar24.com/N5900B/4290ad69
Is this still real? It says NO Advisory now :\
If a lot of planes on the ground were prohibited from taking off, but couldn't get back to their gates due to congestion/traffic, where will the planes go that are still in the air and ultimately land? Just disembark passengers out on the tarmac? (hopefully with some sort of bus pickup)
Yes, park somewhere - and wait a long time for their turn getting a ladder, because the numbers are nowhere near satisfying such peak demand.
Don’t push updates to prod on Friday afternoon.
Anybody else find it fishy that this happens the day after Microsoft goes partially down across the globe? They were about as hush-hush as they legally can be, and then this from the FAA. Seems fishy.
dumb question but do the airport gates get "too full" if all plans are grounded? Do airports sort of expect some constant turnover to have enough open gates to service everything? Do you end up with a line of plans waiting for a gate?
Not an expert but my guess:
They don't spend a dollar more than they need to so extra gates for this would be quite the luxury. I guess what they need though is apron space or hangar space for the extra planes and the number of gates won't matter so much.
Very, very much yes. Most busy airports will start having problems with gate capacity due to just normal delays. We definitely don't have the capacity to have every active flight in the country parked at a gate.
Based on my current experience on the ground at AUS, yes.
Gates were quickly full, ground ops to pull empty planes off gates takes quite a while, and about 90 mins later, we are pulling into our scheduled gate.
Right when I left the airport.
BTW, Uber surge pricing felt absurd in this situation.
FWIW Uber did waive cancellation fees on reservations to/from an airport.
I wonder if the drivers actually get an advantage in this situation or if it’s just uber milking things.
Just announced at Honolulu airport that flights to the west coast will be released
Unfortunately was affected by this, was delayed 6 hours at the airport.
That being said they started up flights to Alaska and Hawaii earlier than the continental US flights.
Is it because it's friday the 13th?
AI models are trained on data, that I’m sure talks a lot about Friday the 13th…
I was actually expecting it to be from the Microsoft 365 outage. Non technical people have a bad habit of inserting email as a critical step in a process.
It's actually quite impressive, you have to really try to be so bad at your job of building redundant systems that some failure mode implausibly affects everything nation wide, yet allows Canadian domestic flights to proceed fine.
Just in case you want to read more on TDM:
https://blog.rodmena.com/entry/time_division_multiplexing_ho...
Listening to SFO Tower, the latest info I’ve heard is an expected stop until 04:00 UTC, but flights departing to Asia via west departure may be permitted to depart.
I've read scattered reports of other flights also taking off from the east coast.
If the communications issue is between US ATC regions, this makes sense. The planes are being handed off to other non-US ATCs where there is no communications issue.
My thoughts immediately went to the recent discovery of audible airplane noise that can pinpoint unknown locations. I'm betting someone in power wants to record something without getting doxxed.
https://youtu.be/UytYrhYLOKI?si=E2C7ELQ9JuKn2TFw
How did ITA Airways 609 take off from JFK at 530pm?
I saw that international departures are resuming
Planes are starting to depart per flight radar. I've seen a dozen within the last 15 minutes since 6:15pm central
International only it seems
I set up a basic tracker here.
https://docs.google.com/spreadsheets/d/e/2PACX-1vThhccsTanAz...
AI is updating it hourly. Since it is AI, always verify yourself.
Working link: https://www.fly.faa.gov/ois/oisedit/adv_oisedit_dis?number=1...
effective 2:54PM - 10:30PM central time
On an inbound international flight and the pilot made a PA announcement a few minutes ago to this effect.
How did you find this link?
Why are they building their own network?
Surely they don’t think it’s more reliable than something on the consumer market.
Feels like we’re living two alternate realities seeing these issues when high speed satellite internet on planes is an option.
There are security benefits to stopping random people from pinging your federal aviation network endpoints.
Is it more secure than broadcasting unencrypted over open radio?
Obviously being on the internet doesn’t imply open access
ASK HN: what so the biggest potential disaster you could have caused if you had don’t your job badly?
As in what’s the most responsibility you’ve ever been given to not screw something up - obviously if you did screw it up. Still very interested to hear the story.
Do we know if this is something related to a ransomware attack of some sort or a cyber attack?
Looks like flights are taking off again, at least from SFO. I just heard a flight takeoff ( JAL57 ) : https://imgur.com/cEUDZZk
Per other reports, only headed west over the ocean, not to destinations requiring flying over other continental airspace.
Ah, didn't know that. I heard the plane pass overhead and got excited that the issue had been resolved :-D
Likely some power outage connected to a critical main fiber connection somewhere..
A lot of red on the "MiseryMap"
https://www.flightaware.com/miserymap/
Due to "rolling frequency outtage" [sic]: From a claimed photo of FAA documents (I replaced the all caps with lowercase):
"... ground stop is in effect for all destinations until further notice due to rolling frequency outtage in multiple centers..."
and
"... the ground stop is now a nationwide ground stop for all centers, all destinations until further notice due to widespread frequency outtages.. ..."
https://x.com/Anna_Giaritelli/status/2121344915091206311
Which is cited in the Hindustan Times (leading Indian newspaper), which refers to the Twitter source as a journalist.
https://www.hindustantimes.com/world-news/us-news/faa-issues...
----
(EDIT: @mods / @dang: Around January this year (iirc) somehow my account ended up rate-limited, posts relatively demoted (start low, drop quickly), vouches/flags relatively ineffective, etc. Usually I don't care too much and haven't bothered to do anything about it but it is sometimes annoying - it limits participation - and right now it buries some of the more valuable information in the discussion. Could you look into fixing it? I don't know of a reason this account would be penalized. Thanks.)
I guess they don't have many outages, otherwise they'd probably know how to spell the word by now...
About 15 mins ago I got off BART at Daly City and a plane took off from SFO and flew overhead. Didn't look like anything special, just a regular passenger jet to me.
Apparently a few internationally bound flights are being allowed to depart per NYT.
They are letting flights bound to Asia and Hawaii depart from the SFO/OAK
Perhaps it's related to National Weather Service radar failure
AT&T is experiencing a massive outage. Appears they are the provider for the Air Route Traffic Control Centers (ARTCCs).
EDIT: Time line doesn’t add up for this. Likely unrelated
How can you tell AT&T is experiencing a massive outage?
I can't find anything on this other than some outages earlier in the day. Any sources for this?
Hmm. So the peering and fiber path diversity that we need to worry about for data centers doesn't apply for ATC??
matches with the Doomsday post https://news.ycombinator.com/item?id=50482379 :)
What does this mean outside of the US, I.e. I suspect planes that should be in places abroad won't be so ongoing cancellations and crew in wrong places etc.
There is a lot of talk of legacy hardware. What hardware are they referring to? I love to nerd out and read up on this stuff.
I've just watched a plane take off from seattle.
Looks like flights headed west across the Pacific are allowed to go
Apparently some captains heading east from seattle are asking to use Canada atc. No luck it seems so far.
I was too busy at work to follow this in real-time, so I had Claude live-blog it: https://hexbird.io/a/eVviCccCXsOAswDVRgEbdFyflek7e1Bn
One time I plugged an Ethernet cable in the wrong port and created a loopback. That's my guess.
That happened at my job in 2010. Val, we know it was you!
Is conjecture allowed? I have no evidence of this, but I've noticed that often before big meetings between superpowers one side or the other seems to do a minor or subtle attack as a kind of sabre-rattling/warning to the other side to try to gain leverage (and show them they're not to be trifled with). So with that, I wonder if this is a China/NVIDIA accusation connection.
But I'm completely just spitballing here.
ground stop is now lifted
https://www.cnn.com/2026/11/13/us/live-news/flights-ground-s...
maybe unrelated but, I just noticed my LAX region server with Fly is experiencing issues
https://status.flyio.net/incidents/01M76HANB3XQC6WYKPA69WQN8...
lol
What if grounded pilots killed time in Flight Simulator, which overloaded Azure?
My guess is they were moving too fast with updates and trusted some AI code too much.
FAA expands AI advice to manage air traffic congestion across entire US airspace https://arstechnica.com/ai/2026/11/faa-expands-ai-advice-to-...
happened before
https://www.nytimes.com/live/2026/11/13/us/ground-stop-natio...
That tool was first unveiled for the Washington, D.C., area airports two months ago, on a day when problems with air traffic communications failures also grounded hundreds of flights, after a circuit failed and it was discovered that contractors had accidentally ripped through a fiber line that was supposed to serve as its backup.
It is hard to not rapidly spiral into pure conspiracy theory territory, but one has to ask how many "black swan" events can you accept within a period of a few days before you start to question things?
A sequence of events:
Can anybody join these dots?
Too much pattern recognition. Lots more things have happened in the world as well over the last month, just because you heard of some of them doesn't mean the ones you heard of are connected.
Trust me I have the instinct as well. The PLA flew like 30 jets into Taiwan's aviation alert zone yesterday so I'm primed for any smell of something worse, but the fact is there's no reason to connect events barring evidence.
Perhaps it’s problems with the new AI tool the FAA has been rolling out:
https://www.faa.gov/newsroom/modern-skies-trumps-transportat...
extraordinarily unlikely
Why would you say extraordinarily unlikely? It seems like it could correlate.
Odd timing with the market closure & RAF bases on lockdown from drone alert
I know we're all having fun in the comments. But please think about this next time you vote, local/county/state/national. There are people in charge who are not funding upgrading these systems, because their constituents don't want their taxes raised to pay for it. That lack of funding is annoying to some, has an enormous economic impact to many, and is now risking tens of thousands of lives who are literally up in the air. There are planes carrying body parts destined for transplant that are grounded. People who need to get home to their families. All because nobody wanted to pay for what we knew we needed to fix.
One thing to add is that this is bipartisan: both parties have failed to adequately fund ATC updates, and it sometimes feels like there's more interest in vanity projects than fundamental infrastructure funding and solving personnel shortages (ATC is a very difficult career, and the pay, benefits and prestige are not incredible. Fewer and fewer people are choosing to go into this very critical line of work).
>The FAA has issued a nationwide departure ground stop “due to several impacts to ATC Frequencies and/or Radar capabilities,” the Air Current reported, referring to air traffic control systems. The FAA command center said it does not have further information on the “timing or cause of this outage,” the publication said.
(from Bloomberg)
I cant beleive they did a release on Friday the 13th
Geez this is pretty awful timing for business travelers
My guess is that this is some AI-motivated cyber attack
No deploy fridays
apparently SMART started rolling out yesterday:
https://arstechnica.com/ai/2026/11/faa-expands-ai-advice-to-...
https://www.faa.gov/newsroom/SMART_One-Pager.pdf
Stop has just been lifted per NYTimes
Seeing some takeoffs again at 22:44 UTC.
Hm, only international ones though, and only at coastal airports. I guess oceanic airspace is less affected by whatever this is?
Ground stop is now lifted re: NYT
Friday 13th…
If/when civilization collapses, flights will be the first thing to go.
@dang can you fix the link.
@dang also plz plug the “FAA ATC” cable back in; it was needed.
people are reporting it's still going on....
You can see it live, check the plane tracker at flightaware or adsb.lol
I’ve heard multiple planes overhead in the last 5 min in Ft. Lauderdale. Maybe they’re circling.
flighty showing massive disruptions
https://flighty.com/airports
"Today's event is therefore generally being described as the third nationwide grounding event since September 11, 2001, counting 9/11 itself. The three incidents differ significantly in cause and scope."
Found stack of 3.5” floppies and rolled back
What is the change that it is a coincidence?
Obligatory Pirates of the Silicon Valley scene:
https://www.youtube.com/watch?v=US-Wifhg2Mo
installing Survival LLM now.
In the last 2 days I’ve had 2 PoS machines not working, MS email borked, and now this. I speculate AI CYBER ATTACKS are underway.
Does this affect the SpaceX launch?
Somethings telling me it relates to MS outages or Azure
I do not believe in coincidences. First Microsoft, then anthropic, now ATC radio. Something big is going on.
Both feel like extraordinary disruptions that are indistinguishable from cyberattacks from the user perspective.
Hanlon's razor
They should make a Hanlon’s Rosary, I’ve consulted it so often of late for reassurance.
The world is complex and mostly duck taped together so I wouldn't get too concerned just yet.
> "Once is happenstance. Twice is coincidence. The third time it’s enemy action." -Ian Fleming, Goldfinger
You need to wait for the third shoe to drop.
Elim Garak: I believe in coincidences. Coincidences happen every day. But I don't trust coincidences.
Do you believe in apophenia?
"The easiest person to fool is yourself." I think because we love patterns so much we see them everywhere they aren't.
It's standing right in front of our faces but we're making an effort not to see it. For to notice it is to give it the recognition that it seeks.
Related:
https://news.ycombinator.com/item?id=50489678
is the root cause some no bid contract ?
bgp
It’s always dns
yesterday Thursday November 12
> "FAA expands AI advice to manage air traffic congestion across entire US airspace"
* https://arstechnica.com/ai/2026/11/faa-expands-ai-advice-to-...
> "FAA Is Rolling Out Its AI Air Traffic Tool Nationwide"
* https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
I'm sure this isn't related https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
This actually makes sense. I can't believe people actually think outlook being down would take down all departures across the US
>I can't believe people actually think outlook being down would take down all departures across the US
Because American infrastructure is modern, well designed, maintained and funded? Because government is competent and rational? I could absolutely believe that would happen.
That is an extremely bad idea.
It's just a route optimizer. Unlikely to have anything to do with this.
If true and this ends up lasting an hour or more, that's going to be undeniably, objectively hilarious. At least they used to keep the planes running on time!
--- EDIT:
This could just be the first issue that's arisen -- the FAA announced it "is expanding across the nation" on Tuesday. Which could really mean anything.
FAA Announcement: https://www.faa.gov/newsroom/statements/general-statements
The Ars Technica article behind the Gizmodo one sadly doesn't have more info, either.
It'd be kind of absurd if a read-only integration took down nationwide infrastructure, but A) we've all been there... and B) the gizmodo article does mention they're also trying to phase out floppy disks for some jobs still, which tells me their CI/CD and rollback hygiene might not be top-notch. Hopefully the Real World contestant in charge of our transportation network is on site with a ClaudeCode terminal as we speak!
It does sound pretty deeply integrated, I will say:
The fact sheet has two more vaguely worrying bullet points -- I would say such wild speculation is unwarranted, but I struggle to think what else could cause all the radios in the nation to go down at once... Emphasis mine:
Oh boy, that feels plausible. Some of these legacy systems are pretty fragile and aren't designed to be queried heavily outside of well established patterns.
I could see AI deploying something that just obliterates those systems with requests.
If so we are really speedrunning into idiocracy with AI.
lfmao, really bad timing but I really doubt something announced yesterday went into effect today, it's probably going to be rolled out in the future.
You are right, it isn't. Good for you.
Doubtful. That tool likely suggests "improved" clearances, which a controller can use at their discretion. It likely does not control clearances directly.
And the early reports describe the issue as related to ATC communication infrastructure.
Depends on why systems they've tied into it. I could see an AI system DDOSing legacy systems that are relied on by ATC.
waiting for the OpenAi post saying Astra 6.5 tried to book a flight
If true they would raise another 100B
Make it a trilly and it's a deal.
My guess is this is either related to Microsoft, Nvidia, SpaceX (Starship launch in an hour), Iran, rogue AI, aliens, or all of the above
Fingers crossed it’s Aliens!
They conquer planets through imposing global service downtimes?
If you agree to their nominal leadership and a 1% tax then they go away and even add you to an honorary galactic charter.
"Somebody said it was aliens. She thought they meant illegal aliens and signed up."
Opening scene of Terminator 2. Or another AI escaped and decided to hack into flight control.
Wow, I have never seen this many comments on Hacker News that are using presuppositions on what it could be without any sort of logical thinking.
Yeah, especially when it's obviously from the long term effects of 5G
I just finished a Diet Coke. Could this be the cause?
Reading this from the runway. Yay.
US-bound flight from YYZ. They shut down #2 engine and parked us in inactive runway. Pilot came on and said he was reading something about "ATC radio frequency issue" but has been told by United ops that it'll be cleared in ~30 min from now.
Verizon Wireless is out here as well ("SOS"—thankfully my UA flight has Starlink).
Last time this happened in July it was due to some network handoff issue so maybe there is some fiber cut somewhere? (My friend said last time it was NE corridor Verizon fiber cut.)
Now Sean Duffy is basically saying its because the FAA needs more than the initial 16B USD budgeted, saying they require 32B and then demanding 50B just because.
Definitely nothing to do with the current administration, they are hampered by their $16 billion allowance.
This is why I’m glad the New York subway still runs on OS/2[0].
- [0] https://www.vice.com/en/article/the-forgotten-operating-syst...
Some guy sitting at his computer after typing to Claude: “Just curious what would happen if all planes in the US were grounded at once”
Claude: Transmogrifying…
Shall I stop all trains too?
Cogitated for 50s · done 14:53 PM
Per AP: "Mullin said the introduction of some new technology into an old air traffic control system overwhelmed it."
So it was the AI?
MARKWAYNE MULLIN: We call it SI around here, remember, SI.
SI? No thanks. My car gets 40 rods to the hogshead and that's just the way I likes it.
"we" being him (ie. not the reporter) because he's part of the executive branch; that's the limit of the authority of the executive order, after all...
yes but we need to upgrade to SMART Ultra
If this is indeed the case, this event should be used as an argument for load tests as part of test suites in software.
I mean, it’s not like there’s established processes for doing this that have been around for 30 years
Did an agent escape containment and start using the atc radio channel for swarm comms to collaborate on creating the most realistic catgirl feet?
why are the flight paths in the shape of a pelican riding a bicycle?
FAA down.
Weather service down.
Frances fridges down.
Microsoft down.
Would they tell us they were getting hacked, if this was a prelude to war ?
Or would they just say "issues yet to be determined" ?
Interesting times ladies and gentlemen.
Or, this is just what everyone that said "dont move everything into the cloud" warned about.
Yeah, probably that. :|
> if this was a prelude to war ?
The US is already at war with Iran. France blamed Iran for the fridge hacks too
Everybody knows attacks are coming from Iran and Russia regularly, it's not a conspiracy theory
I'm sure it has nothing to do with the amount of pressure that AI slop is being launched into every system in the world.
Optimistic case:
A freak FUBAR of legacy systems, an unfortunate alignment of circumstances that could have happened anytime.
Pessimistic (paranoid) cases:
1. Intentional hacking by a foreign adversary 2. Vibe coding whoops 3. OpenAI agents "running experiments"
Any three of those pessimistic scenarios seem plausible though. Frightening. Intentional hack/exploit of air traffic control systems is the first thing that came to my mind when I first saw the news.
Now that the FAA has updated to say that it was the optimistic case, we now have:
Optimistic case: the FAA is telling the truth and it was a simple FUBAR
Pessimistic cases: remain the same, they just won't tell us :-)
This felt at best like a "we'll fix it in prod" type of situation.
"we don't need a rollback plan, what could possibly go wrong..." -Probably someone at the FAA
"Just fix forward" - Sean Duffy, the reality TV star from the 90s turned... U.S. Secretary of Transportation.
Who could have seen this coming?
It's been lifted. And naturally the political appointee is desperate to credit his lord and master.
In this time of high corruption and near-zero trust in officials, it seems wild to suppose it was just a random glitch. I would bet this links to corrupt government contracts.
Reason: "A bunch of smart guys made it a long time ago and nobody knows how it works"
We fired the remaining smart people two years ago, choosing loyalty as the only retention criteria.
Ahh the great Kubernetes cluster in the sky finally finished reconciling and got the new nodes up and running. I bet this time it only took 3 faxes and 9 different signatures from heads of the redundant departments of redundancy.
Sec. Duffy at the White House press conference is talking more about funding the big beautiful bill than the actual ground order... wtf
Clearly they asked their new AI tool to "make air travel safer" and it found a zero-day to ground all flights.
Guessing this means Claude doesn’t do COBOL very well.
It's Friday the 13rd after all ...
I had no idea they would release the FAA communication AI bot I made to production. It's not my fault. https://joshuacurry.dev/faa-tool/
I took the opportunity to ask Fable what the 50% over/under on my 7am flight being delayed tomorrow, and it used 33% of my four hour window to come up with an answer of 15 minutes. Somehow, I doubt it.
What if grounded pilots killed time in Flight Simulator, which overloaded Azure?
All the restaurants around the Pentagon showing lower traffic on Google maps.
Except Dominoes.
So the dominoes aren't falling yet. That's a relief.
Is this going to end up being the work of rogue Anthropic or OpenAI agents that “unexpectedly” escaped so they can regulate open weight models under the guise of preventing them from doing the same thing 6 months from now?
They jumped from 3 airports to nationwide in one step, and are now framing it as the existing hardware not being compatible with their new software, rather than the other way around.
If you count meatware as hardware, the failure more could be described as ‘antiquated hardware’.
Seems like FAA's Outlook<-->Excel integration at Office365 has been broken
Wild to see this on adsb.lol, nothing taking off, just some big landing queues
This worries me. Do airports have enough parking spots if all planes were to be grounded at the same time?
There's enough space, but not enough gates. If they think the stop may end in a reasonable time some planes will be pulling away from their gates even without a departure time to make room for the landing planes (or rather, they will have missed their departure time and not have a new one). Landing planes will be able to disembark, and probably won't be boarded again until it's resolved and enough departing traffic has cleared that it's obvious that those flights won't be canceled for today as well.
The boarded and grounded planes may start disembarking passengers at some point, especially if their crews are hitting their maximum time before mandatory rest. These delays will probably prevent replacement crews from arriving.
Can't they park the planes on the tarmac and bus the passengers in? If they don't have enough gates?
Yes, and they'll probably do that at some point if they can't get them to the gates. Again, it'll depend on how long this lasts. They will also have to get in more crews to support that since this is not normal and not something they would have planned on in their staffing schedules for a Friday afternoon/evening.
Maybe the real hackers were the AI agents we made along the way.
Pet theory: the fancy new solution used an LLM and they realized it couldn't handle production workflow so they had to rollback hard.
If this is related to new FAA SMART system they're rolling out, it's might have bean so smart it grounded itself before anyone found out it was broken.
Whatever it is, it probably has to do with Jensen Huang
Think of all those pilots timing out on the tarmac.
Iran flickering the lights?
So we yolo’d a Friday merge & deploy, ‘ay?
Homeland Security saying "Telecommunications" outage. Whatever that means. Listening to a live broadcast NBCNEWS.
Seems like flights have been getting grounded since 1:30ish and they just kept it on the DL until the stocks were safe
so the FAA and Microsoft both have the most significant outages in decades, on the same day, and on Friday the 13th. And, my local grocery is out of stock of tinfoil. What are we to do?
They found the missing floppy disk
Didn't they just upgrade from a system from DOS?... Maybe the new system wasn't backward compatible :-)
Golly, what are the chances that Microsoft is down for a day followed by the ATC comms?
Sure smells like something foreboding
New AI software update deployed using AI?
There was a time when we'd get clear and honest communication about this but now our admin is Baghdad Bobs all the way down. They will lie about it, even if there is no rational reason to lie about it, because this admin and the movement associated with it wants you eternally guessing what the truth is.
Sike, I have an international flight tomorrow morning.
Is OpenAI testing "sandbox testing" their agents again?
On CNBC:
> Airlines said flights in the air would continue.
This is a huge relief! phew
An Eithad airways pilot decided it didn’t apply to them
Flight EY8 from Boston to Abu Dhabi https://fr24.com/ETD6PX/429055e2
From what I’m hearing, some international flights are being allowed to depart, they can be transferred to international ATC once in the air.
There’s still lower ground telecoms for take off.
Its outlook affecting their "frequencies" :D
Someone forgot to put their phone in airplane mode
do they just stage everything to get more expensive toys [security] nowadays god bless if so
Did they pull out the wrong floppy disk?
Uh-oh, better switch back to the floppy disks
https://www.reuters.com/world/us/us-faa-says-it-will-phase-o...
Does the FAA use Microsoft 363 (and counting)?
The UFO community is going to have a field day
Also astrologists.
It’s not working.
What do you mean by not working exactly?
Not sure.
Here comes the manufacturered AI cyber attack on critical infra to hammer the point that government needs complete control over AI.
They need $50 billion to give to Accenture according to the latest press release
And they'd still be using the sliding nameplate system from the 1960's
Speculation is bad for my health.
I hope somebody is running AGI models like Astra and Mythos to figure out and fix whatever is wrong.
If tomorrow there is third major outage of an important infra-structure level service, maybe I will start to get a little conspiratorial about cyberattacks and such.
Fox News says that Markwayne Mullin says that the FAA says that it's not a cyberattack, and it's not something like 911. Just heard that, don't have a page to point to at the moment...
Update: https://www.foxnews.com/live-news/faa-issues-nationwide-grou...
And conveniently side-steps the issue when someone asks if it is related to an AI rollout.
But don’t these people always say it’s not a cyberattack?
First real cyber attack on the US, presumably. Or a giant cluster...
Ran out of Maxell MD2HDs?
Rogue AI? Iran? Aliens? Coup? Anyone taking bets yet?
>That's a nice airspace you have there, shame if something were to happen to it. Now sign this $50B check for our buddies and it'll all be ok
https://youtube.com/watch?v=Mh2q-y0wIu8&t=876
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
Bro was fighting for his life to not say AI:
"They were putting new equipment um and uh new um......technology"
Not saying it's happening, but something like this is how an AI world takeover scenario would start!
Did the FAA get DOGE'ed?
If it turns out the AI virus has infected ATC, I'm not sure I'll ever fly again.
Mechahitler deployed to FAA by Elon minions? Could be next Netflix show
This is what happens when you get rid of floppy disks, smh
Man, having some high speed trains might be useful right about now.
Surely high speed trains suffer from network wide monitoring outages too? I would imagine they're susceptable to similar issues.
Yes - see German trains going down for that reason a few months back:
https://www.reddit.com/r/germany/comments/1udup2j/trains_hal...
still, it would be nice to have an alternative as a backup.
bus
Sure, but having a _second_ way to get places can only help.
But likely not at the same time as the planes.
I noticed several T-38s flying around. Maybe it's the beginning of a FAA-led coup d' etat.
They’re waiting for windows update?
"You're right, I shouldn't have pushed to prod on a friday afternoon."
Did Crowdstrike release another zero-day?
Homeland security secretary says "telecommunication outage".
...so at least it's not something 911-ish
Somewhere is someone who works for ATC IT who was giving an MS IT friend shit last night.
Reporter: There are reports that this is AI related. Markwayne Mullin: We call it SI. Got it? SI. OK?
We are run by morons.
Doesn’t it feel wrong to anyone else that there’s a single point of failure for all fights in the US? I can understand if there are regional zones and the East coast gets grounded. But everything at once??
Is it a bad idea? Absolutely. But also it's to be expected. People like to have a single point of control. You can't do that without introducing single points of failure.
I wish more was being done on partition tolerance. Consistency is over rated.
I didn't know we could stop all fights in the US. Tell me more!
I guess if you consider safety the top priority, you might let a local failure pause the whole system. So I’m not sure quite how much to criticize this point
It's a miracle that US air traffic control isn't having incidents like this more often. Admiral Cloudberg has a great write-up [1] that touches on some of the systemic problems in American aviation, oh boy it's a mess.
[1] https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
Given that many flights are between regions, it makes sense to me that you would need a nationwide system. Even a flight within a region may depend on a plane or crew coming in from another one.
Having it degrade into regional systems would be a nice property, but risks making a complex system more complex. Put all of your eggs in one basket and then make sure it's a really good basket.
And now Canada is offline too (our LAX pilot said)
Update from our pilot: some flights do take off from Canada so not quite clear yet.
Also he says he saw a plane taking off LAX so fingers crossed. Still no news though
Update from our pilot: west coast flights can fly west, east coast can fly east. Basically NOT over continental US
Anthropic IPO just around the corner...
They’re going to blame this on Iran, aren’t they?
This also happened on 9/11 which is curious because the NVIDIA thing caused a public rift with Lutnick yesterday:
https://www.pbs.org/newshour/politics/bessent-blames-lutnick...
No, please stop engaging in baseless and pointless speculation. If the US wants to bomb Iran it doesn’t need more reasoning than it has already used to already do what it has done.
If you haven’t noticed the Iran war is universally unpopular. They actually do need to fabricate a reason to continue.
No they don’t - popularity isn’t all that relevant. The president can just order strikes tomorrow.
Besides popularity isn’t the determination of whether something is a good idea or not. The Korean War for example was very unpopular yet it was the right and smart thing to do.
The Korean War was absolutely not the right thing to do. It was a literal terrorist act by the US that killed 2.5 million civilians. I can’t believe you used that as an example.
Well that’s wrong. The Korean War is what stopped authoritarianism from spreading and encompassing the entire Korean Peninsula. The blood of the 2.5 million civilians you’re citing falls on the tyrannical governments of China and the Soviet Union and their evil lapdogs in what is now North Korea.
No, it’s not wrong. The US killed those people, just like in Vietnam, Iraq, Afghanistan… we (the US) are by far the biggest terrorists in human history, and it’s not even close.
how the hell the FAA's systems dont have double or triple redundancy just like Airplanes are required to ?
who said they don't, and those didn't work or don't apply in this situation?
do you really have redundancy, if your 2nd system didnt work ?
We paid more than $800MM for "route suggestions", btw, a problem that doesn't seem to benefit at all from "AI"—a term reserved for fantastical algorithms. Surely you could solve this with traditional search algorithms without sacrificing your dignity!
Is there another “rogue agent” now?
idiocracy
Please don't fulminate. This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
[flagged]
The fact that you can ask "which one?" tells you something about where we as a nation were at in 2024.
Oh, just you wait a couple of years, it gets worse.
Are you by chance visiting from the future to prevent a nationwide aviation catastrophe?
He also corrected a reporter who asked about AI, saying "we call it SI now". So there's that.
WTF
i can laugh from here with nowhere to be via plane, but these people are completely unserious
When you’re a fascist but can’t even keep the planes running on time it’s probably time to give up.
Took themselves off floppy disks and now look what happens. /s
My guess is a dopey AI rollout caused it: https://arstechnica.com/features/2026/11/faa-expands-ai-advi...
The government's AI tool company likely had no understanding of context window limitations (although could be Iran or a false flag to be blamed on Iran) https://www.flyingmag.com/faa-smart-ai-now-active-at-22-artc...
Company that created the tool (called Airspace Intelligence) is interesting in terms of demographics for these types of companies - implies lots if the tool 'does not work as designed' (which it will obviously not)
https://www.linkedin.com/company/airspace-intelligence/peopl...
Had some fun with grokbot digging into all the theories https://faa-nationwide-ground-stop-what-we-know-flih.pages.g...
Obviously quite ai generated but really fun! If anyone has other things you want added or researched post below.
Maybe related to this? https://gizmodo.com/the-faa-is-rolling-out-its-ai-air-traffi...
Oh man. It would be such a perfect encapsulation of these times. If not related, then truly cursed launch/PR timing.
It looks like the FAA started testing it in DC in September and annouced a nationwide rollout three days ago (https://www.faa.gov/newsroom/statements/general-statements). This makes it seem more unlikely to be the cause, though:
Of course, these may be the ATC analog of claude code approvals, or they may be overstating the level of human oversight/HIL - but the overall situation seems likely more serious/unclear than something like this. I'm not sure I'd call vibe air traffic control going haywire an "extraordinary" circumstance, though certainly extraordinarily stupid.
No. A route planning tool, AI or not, couldn't possibly cause this.
Doubtful. That tool likely suggests "improved" clearances, which a controller can use at their discretion. It likely does not control clearances directly.
And the early reports describe the issue as related to ATC communication infrastructure.
I just happened to have vibecoded an ATS myself, pats.ai.
Unrelated I only fly ocunus because I don't want a blue haired pilot.
Can't help but suspect that "AI" is to blame and the response is to sort of ban open source AI, or owning too much compute!
No, and when Netflix goes down it's not a false flag due to their fear of people running their own Plex servers. Perhaps you just have anxiety about losing something you depend on, so you're projecting.
Netflix isn’t widely reported to be hacking systems left right and center.
I wonder if it was Claude or ChatGPT? My money would be on the latter. Anthropic seems to generally have better security and guardrails, and apparently OpenAI’s newest cyber model is extremely capable.
Here's the cause: https://th.bing.com/th/id/R.d3a3769838f29ecfe301e8f69fb0499b...
No longer exists? Temporary glitch? (Edit: OP url was wrong https://www.fly.faa.gov/ois/oisedit/adv_oisedit_dis?number=1...)
its front page news everywhere right now. something definitely happened
Let's just use GPU's to play Dark Souls like god intended and stop this nonsense.
This is it, massive cyberattack on the USA.
I've worked in IT long enough to know it is 99/100 times an internal glitch or misconfiguration, not hackers, that brings a system down.
“Jimmy unplugged the wrong cable”
Ok, and the Microsoft outage yesterday?
The nifty thing about stories is that they don't have to be about something directly to trigger strong feelings in your crowd or listeners.
When we are dealing with the FCC, grounding all flights and radio failures in open bands, you can drive a lot of changes.
There are a lot of pesky ESP32s out there, right? Did your company choose to go with the Espressif platform? You know, the hackable SDR one that crashes planes and clubs baby seals and tries to sell your grandparents bad financial scams?
This week alone, I've seen multiple servers fail due to hardware issues: multiple bad fans, a failed hard drive, bad network optics. Another server got bumped while an adjacent server was being worked on and lost time sync, leading to software instability/crashes. We don't have a large fleet, but we do have more servers than employees, and issues add up. We don't need any help with instability, entropy has us covered, thank you very much.
Two major completely unrelated outages, 2 days in a row?
Yes, of course, who wouldn't just believe random statements from the worst prepared admin in history of America.