> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
It indeed is and I like this analogy a lot. A lot of critical systems can definitely learn from aviation. It's one of the finest balances of humans and automation.
Rent seeking has a specific definition and it fits here. When Autopilot does almost all the critical work needed (besides announcing frivolous stuff in carbon-based voice), and you have mandatory seats for humans to babysit the Autopilot mainly to satisfy some law, the term fits.
I am not convinced humans can do much better in "disasters" than Autopilot, despite the emotional stories to the contrary (survivorship bias is one hell of a drug).
This take is so ignorant and provably false. I can't tell if you're trying to ragebait or if you actually believe this.
The autopilot does not do "almost all critical work", not even close. It only does some of the routine work (definitely not all of it). But the real critical work is when stuff goes wrong, dealing with emergencies.
The inherent complexity of aviation means that there are so many different failure modes and combinations thereof that no traditional system could ever handle all possibilities and edge cases.
I'm not going to claim that an AI could never do it. But as of right now, with current technology, well-trained pilots are absolutely critical to safety in even the most advanced airliners.
Also, the survivorship bias argument runs against you. Most events where pilots save the day are completely invisible. Like catching an incorrect clearance, a misconfigured approach, dealing with a technical malfunction, etc.
And even when disasters happen, it is very rare that simply engaging the autopilot would have saved the day. In fact, there are plenty of examples where the opposite is the case, and dropping to a lower level of automation would, in hindsight, have been better.
Your first sentence would lead one to believe that you have some substantial evidence to demonstrate the ignorance and falsehood you identified. It's impressive that you are regurgitating the same formulaic-template set of cliches that boil down to "X is so complex bro, how can you even?".
Note that I never claimed aviation relies on AI right now, most of the Autopilot systems are deterministic automation systems, but the original point stands regardless.
Really? Do autopilots diagnose threatening situations in the main cabin? Understand something unexpected happening on the ground? Visually diagnose situations? Properly identify threat levels?
It's genuinely worrying to me to see statements like this. As engineers, we of all people should know that there's 1 billion variables that could go wrong and another 1 billion we aren't even aware of yet...
As an airline captain, your take is completely ridiculous. The autopilot takes care of the mechanical act of guiding the plane along the desired path, but it does nothing else. There are hundreds of other tasks we need to perform in order to effect a safe flight, and the autopilot does nothing to help us.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
I’m using Elixir for every (web) project now specifically because the cost is negligible. I can't think of a reason to use anything else and the agents are _really_ good at writing it, particularly given the excellent structural guardrails that are in place.
Being able to ensure all of the test run freakishly fast async is a huge win too.
Elixir is great from a syntax perspective. It gets compared to ruby a lot for some reason but they're nothing alike. Elixir syntax is very deliberate about everything.
The BEAM runtime is the real win. Your options are Erlang, Elixir and Gleam to get the BEAM.
Out of the 3, I prefer Elixir and Jose is a brilliant leader for it.
Those of us who care. Those optimizing for economic returns or out of ideology are not. More carnage will be required for incentives to swing back I think.
"Security is for losers"
> Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”
Vote with your feet or prepare for your eventual breach notification and data loss event. You control who you choose to do business with, and who custodians and processes your data, it is a choice. Make good choices.
The problem is, ultimately, most companies have this mindset. Certainly the ones you've heard of do.
Charlie Munger once said, "Show me the incentives, and I'll show you the outcomes." There are very, very, very few actual incentives to operate a real infosec effort at most companies. Think about it: how many companies have you heard of going out-of-business because of breaches? If you're like most Americans, you've probably received at least one letter from some company you did business with telling you that their systems were compromised and your data may or may not have been stolen as a result. They'll usually close out the letter with an offer of free credit monitoring for some finite length of time. Since that's useless and a lawsuit is a pain in the ass with no guarantee of benefit, the vast majority of people simply sigh and toss the letter in the trash.
There's no real way to pin identity theft or other consequences on a given data breach, so it's hard to make any sort of claim that way.
At the end of the day, the company has saved money by not instituting best InfoSec practices and red-teaming their software on a routine basis. There are no civil or criminal penalties for them not doing so.
Which is why the recent movements for cybersecurity liabilities are so relevant, they are the incentives.
Now in some countries those with data security office role have more to worry about than only losing their job, if they sign off on something that isn't qualified to be out there in first place.
I've been doing a bit of research, and mailbox.org looks like a winner. It's in Europe and it supports IMAP. Those are pretty much my main requirements, I think.
Moving faster is good. But only if it means sending more agents (and humans, that part is important) to do the same amount of testing in less time. Dropping critical work is not that.
I want my encrypted email company to move slow, deliberately, and very carefully.
I'd like most software in general to move slower. Some, I'd like to not move at all. I hate churn for churn's sake, or piling useless features just so you pretend you did something to justify your subscription pricing, both of which were already the norm before AI and are worse now.
To speak in Apple terms, we don't miss more and faster crap novelties. We miss the Snow Leopard releases.
No, not necessarily. Sometimes, yes. For anything important, the last thing I want is a company that's moving too fast.
An email company? The base protocol RFC was written in the 70s. It's as mature as computer technology gets. There is zero value in moving fast here, all the value is in building an absolutely rock solid and secure solution.
> That's the sort of thing you really don't want to hear from your encrypted mail company.
Tone deaf award of the decade surely goes to that guy.
If it were some social game app or something, I guess it wouldn't be as tragic, but for a company whose sole reason to exist is to provide security... wow. Goodbye any trust ever again.
The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
Either he's been like this the entire time, trying to crack the whip and failing, and thus a bad leader who has repeatedly failed to get results, or he's (as his employees have said) very deep in AI Psychosis and completely abandoned his principles.
It doesn't matter which it is, they're both extremely damning for the company and his leadership.
>The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
As if that's a problem? I want them to be my fucking encrypted email. Not a calendar. Not a backup. Not a VPN. And they've already stretched to all kinds of shit.
Weak man's idea of a strong man. Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas, but I digress.
"Any man who must say 'I am the king' is no true king." The measure is not who you can bully and denigrate successfully, but who will follow you to hell.
>Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas
And whose idea of a strong man is this? I don't think many people would follow Putin to hell willingly, or that he accepts any criticism, or accept being challenged on his ideas. But he could be called a "strong man", at least the kind of "strong man" some people aspire to be.
According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people that is heavily selected for a particular characteristic you will end up with people that generally are outliers in some other significant ways too. In this case, caring deeply about email security is weird... unfortunately... and you can expect further weirdness from such people.
You can also read about people talking about this in the super-early computer user communities. When you had to be a bit crazy to even be interested in computers you ended up with people who were outliers in a lot of other ways too. As more people came in to the space it would regress to the mean... this is one of the things that "regression to the mean" can be regressing from.
> According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people
Social contagion [0] (not to be confused for memetics [1]) involves behaviour, emotions, or conditions spreading spontaneously through a group or network. [...] Throughout history all social contagions have had a single thing in common: the people who follow them genuinely believe their group is only doing positive effects, and cannot rationalize negative effects from within their group.
Tech executives are a group (and a network), its not some "regression to the mean" its common human social behaviour within group dynamics. It's not just those two there has been a progressive slide of executives joining the 'tech-bro' subgroup.
I mean if a company doesn't care about protecting its users' data then any conversation about improving security is gonna be inapplicable to them.
I'd also suggest that companies being so blasé is a direct result of lacking regulation. If there was no laws about aviation safety I bet we'd have more airlines with lax safety standards and more accidents.
"His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”"
Mmmmm... I think I may have to score this particular point for the CEO. Lovingly handcrafted code has been many things over the past 30 years, but "secure" is not one of the adjectives I'd apply to it. Anyone who thinks that the lovingly handcrafted code has always just been wonderfully secure needs to check the tint on their glasses.
A proper AI setup probably can produce more secure code more quickly than conventional development processes... however one of the keys will be to understand that yes, that will still be slower than writing code that is not rather carefully analyzed for security and a company like Proton had better have a strong commitment to never, ever skipping that step because it's slowing them down or it got expensive. "Proper" in "proper AI setup" is the most important word.
But that's not to say that I'm endorsing everything else he said. Even if it is true that companies will be able to stop hiring engineers, which at the moment I seriously doubt [1], a company like Proton should expect to be one of the last companies who can do that, not be getting publicly excited about it first. Hey we dumped all our engineers because our managers can just prompt our AIs just as well here in October 2026 would definitely be a "instantly cancel any Proton services I had" signal for me.
[1]: One of the things I don't expect to change is that there are plenty of people I would describe my actual day-to-day work to and they would consider it a living hell if they had to do it. I still quite enjoy it overall. And there are plenty of jobs that other people love and I would consider a living hell. Even in 25 years that's going to be the case. There will be people willing to put up with the process of working with the system to create this stuff... but the people who are willing to be professional about this, do the QA, do the work beyond the first fun prompts, is going to be a minority, and an even smaller minority who take the time to get good at it. There will always be people who are better and people who are worse.
While I agree that handcrafting code has never been any kind of guarantee of secure code, I'd argue that it remains a condition: if all you have is a set of specs (or even worse, specs written by AI and not read carefully by anyone) and no mean to ensure that the implementation actually matches the spec, you essentially have no understanding of what the code is doing, which is security suicide.
Of course, AI reviewers and security tools will catch some security errors, but this should come as an addition to a proper security design, not instead of it.
Just learned about his position on security today. Been a Protonmail user for a while now. Initially I switched to detach myself from the US corp monopolies. Its just a matter of time now before I shutdown my Protonmail account. CEOs should understand that company ethics and security are not just black pixels on a white background.
But why stop there? Aim higher!! End of all world’s wars, no more hunger anywhere anymore, free abundant clean electric power for everyone everywhere!
Jokes aside, you can’t get anywhere near zero fault unless to impose strict operating regulations , requirements and procedures and regular checks… just like in aviation.
And even there, boeing manages to fuck it up somehow.
There’s a limit to how much society can spend on aircraft safety, but it’s quite a lot because it’s so important.
Cyber Security teams have to work with large constraints because spending an ever increasing amount of money directly, or losing money indirectly (like not allowing product teams to move quickly) is explicitly not a goal of most companies.
So while we try to make zero mistakes, most companies are willing to take a non zero amount of extra risk of these things happening in exchange for other benefits.
Aviation is working with the same constraints. Development of new systems is very slow, that's true. But day-to-day operation is extremely fast paced, and still impressively safe. They are able to change an engine of an aircraft within hours, with a very high probability of not making any relevant mistakes in the process.
Sure, but at what cost? How quickly improvements are delivered is one thing, but the materials and chosen process also come at a premium. And while it probably is worth it in aviation due to life and health being highly irreplaceable, the value of for instance my chat history or photos is certainly finite and quite low. Should they be corrupted, leaked, or worst case gone, it would somewhat suck. But at the same time I've effectively concluded that safeguards are worth about 0.4% of my monthly take home after tax, and when the yearly bill comes due I'm each time increasingly questioning whether I actually do value them even that highly.
A less costly new engine design or less costly engine swap procedure that fails is worse than no airplane at all. If it's cheaper and fails, we don't want it. It wasn't cheaper, it was even worse than expensive.
It's not hyperbolic all or nothing. Sure infinite cost is just as useless as infinite failure, but there is no reason to talk about infinite.
This doesn't address how the cost of safety related procedures versus the cost of risks mitigated is calculated (which is way harder than saying "some") so that the buyer can actually know what "some" cost is.
It's also not addressing the parent comment's point: the bad outcomes in aviation tend to be very impactful (both in human safety and financially), so the tolerance for mistakes is way lower.
Compared to cybersecurity for a brochureware website, where it takes a far larger stretch to go from "security incident" to "someone's dead". (Also, people in cybersecurity aren't going to accept "it has some cost, and you might buy it if it's not too expensive".)
For the commercial flight industry this is very easy to calculate: zero deaths are acceptable. This is not like car manufacturer recall calculations (famously, referenced in Fight Club: if the lawsuit costs the manufacturer less than the recall, they don't recall). The airline industry is built different thanks to heavy regulation.
You could ask: is it _worth_ making it that safe? But we don't ask that. We just make it that safe.
I think there is a good point to be made that this level of safety (it costs what it costs) is probably not worth it for cybersecurity, in general, even if it's largely considered worth it for commercial aviation.
As software intermediates more and more of the physical world, we're gonna end up at least as regulated as finance, and potentially as much as aviation.
The only real question is how much damage will be caused before that happens.
You can also reduce blast radius from breaches, and make systems more robust, rather than just trying to make zero mistakes. How bad breaches are and how well you can recover is also part of the outcome, just framing it as zero breaches or nothing is not that helpful, even though reducing breaches is important.
No, it's recognizing mistakes will be made and no one is perfect. You will also make mistakes regarding blast radius - sometimes the blast doors don't close or function like you thought they would during an actual explosion.
Defense in depth is the idea. Swiss cheese model of failure.
Yeah, but this is bottlenecked by both your ability to observe and understand the system, which is precisely the thing that's being reduced by careless use of AI tools.
This isn't a different thing - it is just one of techniques used, and it still requires humans to understand the systems they are building. Teams that outsource their systems understanding to AI will fail, no matter how they frame it.
People always find new ways to make mistakes. The approach in aviation is to investigate, identify root causes, and do what is possible to prevent the same mistake happening again.
There is no guarantee we'll ever reach perfection.
It's more about managing mistakes. Pilots and other people in aviation still make a lot of mistakes all the time. But aviation figured out how to deal with that fact. Design procedures in a way, that mistakes self-correct most of the time. Add sanity checks exactly where they matter, and many other strategies.
Maybe it's different in your neck of the woods, but all the government portals I use tend to have access to rather sensitive PII, and I would really prefer their ops teams be up to snuff as a result.
A better metaphor would perhaps be machining. You will not find a shop that has no CNC anywhere, but the machinists who can also use manual machines will always have an edge for certain things. Not that they're working manually much in their day to day, and not as all as a failsafe, but you can just tell who has that kind of background.
(And you can also tell whose mind is stuck in the old world, too.)
It's first-principles thinking. You have to have some sense of how you would do it in order to reasonably engage with a machine's idea of how to do it. Otherwise you're just nodding along, like yeah sure bro uhhhh lgtm.
The problem is that people go on Twitter and see the posts from "thought leaders" and "influencers" who say things like "reading the code is not needed". Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
You only need a handful of occasions of creating massive billing errors for your customer's customers before they will slag you to all their CTO friends and pull the plug. Imagine Stripe began double-charging every third transaction once a week, and couldn't fix it after two months of thrashing because no one there has any understanding of how anything works anymore. They are simply done at that point, all the value of their brand lost.
As if engineers, especially those writing software to handle financial transactions (which are usually in regulated businesses), don’t know how to write conformance tests with invariant cases.
soooo wouldn't that be exactly the "deskilling" being discussed here, and isn't that deskilling concerning for money-handling firms?
having seen both (a) mission-critical human-built systems become nightmares for maintainability and extensibility and (b) bleeding-edge ai coding agents confidently and quickly fix a wrong problem in the face of an issue (or fix it in a slightly wrong way), I remain skeptical that this great de-skilling will all be perfectly fine.
of course, if you aren't looking at the code you won't fucking know until someone yells...
I disagree with several premises of the article. There is not going to be human led investigations of outages. People do not need to train on easy outages to be able to handle hard outages. Even if you wanted to train people on easy outages AI makes it easier than ever to setup up real world like issues to replace the need for real world experience. I also disagree in labeling it deskilling when it is a reallocation of skills to what is actually needed.
>concerning for money-handling firms
The real world is robust. It's okay for money handling code to have bugs. It's not the end of the world and AI will be able to have a lower bug rate than humans.
> The real world is robust. It's okay for money handling code to have bugs.
This is true. The world will go on. But your business will lose more money than it gained by not paying attention to the code. It will lose money because it will have to compensate customers or lose customers. That's how the world maintains it's robustness.
> Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
Name one. Point me to one developer that actually works - themselves, not just in the same company - on a system with serious customers that demand correctness and is spreading that message on Twitter.
Oh my that is convincing. Yes, I've seen lots of this sort of thing. I believe you, a Twitter influencer is behind it. AI building AI systems for AI to build AI systems with.
You're promoting your own company here, which as far as I can tell from the vibecoded website is Yet Another Agentic Platform of some sort. And prominently features a leaderboard with high scores for some reason.
It hardly meets the bar of a mission critical product like an ERP or payment processor layer where introducing random billing errors in customer data could be fatal to a B2B company's viability.
If you scroll up to the message you originally replied to, OP's goal posts from the start were companies like Stripe (which was specifically named).
Most businesses aren't in that category, that doesn't mean they aren't able to provide useful services and make money. But your company does not seem like it meets that bar.
Expanding the group to "any money making SaaS" would be moving the goalposts for the point OP was clearly making from the beginning.
Sorry for being a bit dismissive in my previous comment but it was fairly jarring clicking your link expecting a more responsive counter example to their argument, and then left a further bad taste after realizing from your bio that it was self-promotion.
And just to pile on, payments is an incredibly competitive space, and operational excellence is non negotiable for someone like Stripe, as they are a higher priced provider.
I think the parent take is valid if you disqualify people managers/leaders rather than ICs. I think that's also valid to disqualify those takes because they aren't in the trenches every day. They may have points strategically for what's best for the company, but not what's best for the workforce or the industry.
I don't see what the giant social cost of AI hucksters is supposed to be... who cares if some random influencers claim you can vibe code your entire existence? the thing that bothers you is that the grifters are basically right, just for the wrong reasons (and a bit too early). Comments like this imply that there is some cavalcade of situations like this that have caused actual harm to actual people, but it is actually just an emotional response to people who are personally distasteful, and borne out of a desire to make sure that the "assholes aren't proven right". stuff like "Imagine Stripe began double-charging every third transaction once a week" is just so silly... what about AI makes this more likely in any way whatsoever? If anything, AI would have such a subtle scheme for robbing us blind that it would defy characterization into pithy quotation marks. this feels like a complaint about humanity dressed up as a complaint about AI. for all we know, AI could actually improve the situation-- it's not like humanity has set some unimpeachable precedent.
AI can absolutely improve the situation. It has helped find and fix bugs I have despaired of ever fixing. But it cannot learn, and it cannot understand things for you. And people outsourcing all their understanding to it absolutely could lead to existential disaster for a system at the heart of the platform. Coding agents write too much code and ignore existing patterns. This has been studied quite seriously and they generally write 2.5x the amount of code a human does when refactoring or enhancing code in a real project. When a very large system slips past a critical point of decoherence you really cannot put any certain boundaries on what kind of problems become possible or how long it may take to fix them.
Yes, there are responsible engineers using AI in innovate ways and making things both more productive and better. But there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place.
> it cannot learn, and it cannot understand things for you
the former is demonstrably untrue (and wildly so), and the latter is just an egocentric/anthropocentric "the genie in a bottle only answered my wishes, it didn't tell me what to wish for". buried in that statement is the presupposition that humans being able to understand that something is actually Important; short of religious/faith-based reasons I don't see the justification for the primacy of human understanding when it comes to the determination of Truth. if anything, being "understandable" to humans strikes me as an anti-goal. are we really to believe that human comprehension is the limiting factor keeping society from degrading into "decoherence"? how well is that going for us? the reason humanity places such a premium on identifying "The Truth" is precisely because of how illusory it is for our fallible minds. LLMs struggle with many things, but coherence is least among them. that is why "slop" works so well-- if it wasn't at least mildly coherent, it would be easy to discriminate.
> there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place
and there are even more lazy humans who are happy to ascribe this to a shortcoming of the technology and not of humanity itself.
insofar as we are descending further into post-truth, it isn't because of AI slop blurring the lines between fiction and reality; it is because we are all collectively realizing that Truth isn't all it's cracked up to be. LLMs are forcing us to take a long hard look into the mirror and we don't necessarily like what we see.
For practical purposes an agent is learning if you put information in its context. So you can create a comprehensive architectural analysis and put it into context and suddenly the agent is familiar with the project. You can also delete information from context, which is a rather hard and imprecise process with humans.
> Maybe not all of the projects need the same properties.
Definitely not all projects need the same level of quality.
But per the thread, if you allow the skills of all the engineers in your company to rot, what happens when there is a system that needs the higher level of quality?
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Oh yeah, and you will find very quickly that most people are a bunch of flaming hypocrites when it comes to flying. Lots of wannabe aerospace engineers in tech. Half the people I have met who put aviation on a pedestal hate flying but love to trumpet the statistics of how safe commercial aviation is. Try asking them to get into a Cessna and all of a sudden the excuses come out. It's oh noes I have a fear of heights, oh noes general aviation is less safe and only slightly safer than a motorcycle, oh noes the FAA is corrupt and allowed the helicopter crash in DC to happen, oh noes I have a family etc.
Aviation safety discussions on hacker news have barely more more depth than biotech and healthcare discussions here. It's extremely ironic because a tremendous number of pilots are engineers, and especially software engineers. It's not exactly a cheap hobby. But you don't see massive numbers of English majors come crawling out of the woodwork to join a discussion on the safety of C++ vs Rust, but tech bros flock to worshipping aviation as a platonic safety idol like moths to a flame.
Half of aviation safety (especially for commercial aviation) is done on the ground and has nothing to do with aerospace engineering. Stuff like shift times for pilots and ATC, mandated rest periods, proper ground equipment in zero visibility conditions etc. Many developing countries cut corners on maintenance and spare parts. Mandated use of safety technologies (the DC helicopter incident was entirely preventable). Airports that are designed with safety in mind. If you have a dual engine failure immediately after take-off on an airliner in the middle of a built-up city you are generally fucked. Midwestern airports are the best if you have to do a forced approach. Transport category aircraft (basically anything bigger than a 12 seater plane) don't really have many options, they can't easily land on the runway like smaller planes. That's why the Hudson was a miracle. Your airliners stay safe through constant maintenance and inspections. Dual engine failures are very rare (unless through deliberate pilot action like the recent case in India).
On the engineering side you can look up ETOPS and the various ICAO and FAA and EASA standards for airliner class aerospace engineering
https://en.wikipedia.org/wiki/ETOPS
Trust me, y'all think airplanes are "safe" because of some sort of magical protocol. No single protocol actually works. It's mostly lots of layers that are stacked on top of each. It's why in aviation, catastrophic failure modeling uses something called the Swiss Cheese model; you need every single hole (error or failure) to line up to generally trigger a complete breakdown in safety. It may sound counter intuitive, but by aviation safety philosophy, a canvas and wooden frame plane operated under commercial standards and kept to a strict inspection schedule is "safer" than e.g. a privately operated 737 that doesn't follow manufacturer maintenance standards.
NASA has plenty of guidelines for writing "safe" aviation software. I doubt any software engineer will be willing to entertain the level of effort and certification needed to comply with those for e.g. their Django side project.
Err...have you actually asked someone to get into a Cessna?
Also, "fear of heights" is a totally valid excuse. Fear is a gut reaction and may have nothing to do with one's justified true beliefs. Eg., I know spiders on probability won't harm me but there simply won't be any amicable co-existence between me and any arachnid.
> Err...have you actually asked someone to get into a Cessna?
Yes? In practice once you convince them to get into the plane the only real issue in the air for those new to flying is air sickness.
Also fun fact a good number of pilots claim to have a fear of heights (never met one irl except online in aviation forums).
If you are in aviation/aerospace, don't be afraid to share the joy of flying. There's nothing quite like cruising at 90-100 knots @ 3000-4000ft AGL on a nice clear day.
I don't understand the perceived hypocrisy. The chance of death in a 1hr Cessna ride is something like 100x a domestic commercial jet flight, so it's reasonable to have different views of them? Maybe I'm missing your point.
Okay, just because aviation is now also developing in the wrong direction (maybe), all the things they did right in the past are suddenly pointless? I suggest to learn from the things they did right, not the things they did wrong.
This was just the default where I worked. We called the exercise Wheel of Misfortune: facilitator sets up an outage / bug in a non-prod environment (inspired by real events or not), spins a wheel, picks a victim, and the problem is solved collectively, manually. That's the standard format, but it could be much more freeform than that: the point is to educate each other regarding "incident management."
I will say, I'd always been skeptical that AI could be better at incident management than an experienced engineer with deep domain / institutional knowledge, but I was recently proven wrong: Claude root-caused and helped resolve a Postgres lock queue pileup that I believe would have taken me much longer to fix.
Re: aviation being a perfect industry to learn from, I think https://m.youtube.com/watch?v=5ESJH1NLMLs might be a good watch. I was introduced to this at a Wheel of Misfortune around 5 years ago, so it's interesting watching it today when AI is encroaching upon everything.
Aviation is a perfect industry to learn from if you are willing to hold IT to the same standard of regulatory scrutiny. You can bet your entire net worth that anyone running anything in the industry would fight tooth and nail to prevent that sort of regime from coming into force.
A (very) key difference is that this behaviour is mandated by the FAA, and consistent across all providers. There isn't a (legal) option of just not-doing-it because it's "cheaper".
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
You're right, there is no mandate for learning and doing better in SW engineering. Many companies just don't care, also kind of fine. This doesn't mean that everyone doesn't care and nobody wants to improve.
The thing is, without the FAA mandating something like this for SRE’s, no one will do it. In the moment, getting the broken thing back up and running is seen as far more important than the learning opportunity.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
The plane lands in the same amount of time whether an autopilot lands it or a human, so there's no incidental cost.
If you let a human SRE handle a problem completely, not only will it likely take longer than an AI, but since that SRE is now deskilled, it may take FAR longer. And that's a cost. Who's going to pay it?
no offense, but I think aviation is a horrible example. if we treated every problem like we treated aviation, we would still be living in the 1950s. honestly, probably the 1850s. hell, we sent a guy to the moon LONG before it was actually safe to do so. (is it safe even now?). do we really think LLMs would be more blasé about human life than humanity itself up until... iono, since the Baby Boomer generation? the "issue", such as it is, with AI is not "we need to make so many guardrails in order for it to be useful", it's that unlike aviation, where the ground truth is "did the plane crash or not?", AI is an open-ended experiment in what would happen where we don't have any feedback from anything other than... how humans feel about it. humans are the antithesis of paperclip maximizers.
I don't see any reason whatsoever why "AI regulations" would become antifragile from their having been "written in blood" in the way that aviation regulations have been; if anything, our inevitable present and future AI crises will each be exponentially more devastating than the last. Modern aviation is a triumph because of how fallible humans are-- that's why we delegate control of aircraft mostly to machines. The shape of AI disruption isn't "the computers aren't following the rules", it's that even by our own rules of rationality our world is terribly underspecified-- we have computer security that isn't secure, subjective rules of ethics/fairness/judgment that defy straightforward explication, "knowledge work" that is actually neither "knowledge" nor "work"... the whole situation is much closer to the human domestication of the dog (letting a potential threat into one's home) than any traditional story of humans coming to terms with their own technological innovations. human behavior is FAR easier to predict than un-aligned AI behavior-- that is why it is such a singular moment in human history.
I work at a startup and often deal with every part of the stack--programming, deployment, db, cloud environment, networking blah blah. AI agents have just gotten superhuman at computer use. I will never be able to use AWS CLI as well as claude does. I can spend time poking around in AWS, manually reading logs, looking at routing tables, cross checking with data in db, whatever, or I can give claude readonly access to all of that stuff and it will solve the issue literally 10x faster than I could, write a detailed report I can pass on to whoever, etc. Is it sometimes BSing? yes, but it just keeps getting better.
All that said, skill atrophy is real. Thinking about getting back on the interview circuit is p scary because of it. I dunno wtf is gonna happen.
I used to scour through server logs, build logs, test reports etc to figure out why tests were failing in CI. That was exhausting work and could take days of investigation. Now I get the exact root cause in a couple of minutes. Just had to give LLMs a token to call Jenkins API and it knows how to do the rest. What an unbelievable experience. Never ever thought computers would be able to do this in my lifetime.
Had this eye-opening moment a few days ago where I enabled our company-wide MCP server and asked for a problem we noticed.
The agent looked through slack messages, logs from our devops portal, open telemetry traces and searched code snippets from other services through our RAG code search tool and created nice tables with timings, explanations and suggestion to fix it.
This is an amazing use of LLMs, contrary to offloading your thinking to it during implementation (which does not mean not using to write code but to actually understand the systems you are creating with it, and all their edge cases, inner workings etc.)
But there's a new skill blossoming there, right? That's what I'm noticing. The atrophying skill is one you don't need anymore. The new one - prompting, routing the output, spotting the hallucinations, managing singal/noise etc. - is now where you can add tremendous value.
To say nothing of being bold enough to run systems like that in the first place. I know some of engineers that are still folding their arms and refusing to engage with AI!
I don't know how to spot hallucinations, manage signal/noise, without having a pretty good understanding of the problem space. It's a tough balancing act.
I really don’t want to see code generated by Claude without some constraint like rules in a Claude.md file. Even Fable on high is lazy enough to forget a ton of things that a senior engineer would not. For example tracing and handling all unhappy paths, not repeating code, not writing terrible doc strings, etc.
We are using quite a big list of rules and the output is significantly more aligned with what we interpret as “good” code.
SRE here, spotting the hallucinations requires you to have the knowledge to know when it's wrong. Best way to gain knowledge on systems is actually working on them. If AI do all the work, then you don't get the knowledge.
I don't know that that analogy fits for this comment. The argument is that the old skill is genuinely no longer needed in favour of the new skill, but the new skill requires experience from the old skill. I don't think being a good driver needs you to have already been a good cyclist.
I think it basically stems from the idea of “conceptual learning”. Which has been a pedagogical ideology for quite some time now. And is also convenient if you’re a bit addicted to your phone, a bit lazy due to AI, and want to still feel good about “learning” things.
We’ve known that one important component for learning (not the only thing that matters, but an important one) is the memorization of facts.
There seems to be a big urge to pretend that’s not true. E.g. you can learn arithmetic without memorizing your multiplication tables, literature without engaging with the canon, and now apparently coding without even having coded.
The idea is if we start with a top-down conceptual approach we will eventually vibe our way into knowledge acquisition. Which does sort of work.
However, sorry to be a curmudgeon, but the people who just “do the work” and actually spend time on task to learn a bunch of stuff … are actually more knowledgeable on average.
It's interesting how machines now mirror our relationship to sanity.
Just under the surface of human consciousness, we are constantly hallucinating and predicting what we may occur next. We are good at discarding what doesn't align with our working model of the world before reaching awareness and taking action. It's our attempt to conceptualize reality. Considering how un-grounded from reality humans are becoming, the fact we have to balance our nervous systems, the mistakes we do make day-to-day, and how minor physiological changes (from poison to injury to drugs to random chance) can disrupt the process, it doesn't take much for humans to make a major mistake or enter a psychotic state.
It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
Not even that Proton CEO claimed that. This is another one of those "no u" knee-jerk reactions littered all over this site, and I wonder when we'll first see stuff like "but humans also sometimes fail to load because of network issues", because there is no real distance between "humans hallucinate even more" and something like that.
Good ones force themselves to do technical work hands on regularly, but will generally reach a level where they can't actually do that anymore unless they are really good at building management systems and delegating. For me it has happened in cycles over the last 25 years; after 4-6 years in management I refresh skills and go back to an IC role because I can't stand it anymore. I've done that cycle three times but I won't do it quite the same way again because its probably impossible to do without a pay cut now, and because IC don't write code anymore anyway.
This is both like forcing all IC to develop management skills but its also like the shift to cloud infrastructure or 4GL languages - we are never going back to the old ways and will be building everything at a different level of abstraction with a different span of control.
People cannot deny that cars have made our physical bodies atrophy. Not all of us, maybe, but many people are way worse off physically than they would have been without cars
There's an analogy in there but I can't think of it. Maybe I'll ask Claude if it can explain it
No, everything you listed is low effort and minimal friction. It's exactly the luddite's greatest fear: to be delegated to a button pusher, to ensure that blinking green light continue to blink green. Anyone that can read and write can now add "tremendous value," so what do you think makes the value "tremendous"?
This. It seems most developers have never even had the experience of iterating on a system they understand, making predictions, seeing the result, and having a clear sense of what they are trying to accomplish. There are so many tools and systems, wrangling them becomes an end in itself.
The way I think about this -- AI makes all of us managers. We need to figure out when we are convinced by what is brought in front of us, when to ask to do more checks/validation, when to fully reject, when to accept. To do this properly we need to have to have enough background knowledge to evaluate the what is AI brings, otherwise otherwise a person in the loop is simply not needed if they only type a prompt and always click yes.
I think people indeed need to constantly practice our low level skills, otherwise they will atrophy.
You may never be as fast, but you should absolutely feel at least hypothetically capable of using the CLI at least as well. Or just.. Reading docs and understanding patterns or interfaces was never a superhuman skill and it still isn't. You are a lot more capable than you probably think!
And I am not just trying to platitude you here. Without that basic confidence everything ends up just eventually being harder with the AI, even if maybe the feedback and loop of it all feels faster.
Atrophy happens when you trade understanding for productivity. Productivity benefits your employer and your KPIs, but not your understanding and your opportunities going forward.
Understanding used to be an unavoidable of the exploration process, and writing documentation used to be the thing that never got done. Now, writing documentation is automatic, but reading it still isn't.
I've been managing a team of interns at my last job, and as they were all equipped with coding agents, they'd deliver complete solutions to problems once or several times per day that would normally take days, or weeks if you factor in necessary learning.
As review of their understanding, and for the sake of my own cognitive debt, I would first ask them the question "How does it work?" and they usually couldn't answer. After a handful of tasks, they started preparing answers. This forth-and-back is still more shallow than getting the deep knowledge chiseled through hours of trial and error, debugging and wrong directions. Spending two hours to find a problem to realize you forgot to save or upload a file, or had an x instead of a y. I look back with nostalgia, and I wonder if there weren't always better ways, and we just get to pick them now. I can literally convert any task, solved or not, into a quiz that brings me towards understanding, in rainbow-colored pig latin.
Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
I’m on the camp of using agents as much as you can, but make sure to understand what they did. However, it will still atrophy your skills, no matter what. It’s just different times now, as on the job you’re expected to deliver faster. And obviously that means less time spending on your craft, honing your skills and etc.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
Let’s say you have an incident, and you need a resolution. Do you spend 5 minutes tinking, then solving the problem or send out the AI for the hunt and spend 30 minutes grappling with hallucinations and non-sensical output?
The 5 minutes I’ve seen is usually by someone that will send the complete output to someone else to decipher, while saying they have figured it out. Then they’ll turn into a meat proxy between the team and the agent.
In real life, for systems where incidents that require such rapid responses, you usually have fail-safes an killswitches. When you were designing your system, did you put in place alerts and monitoring to mitigate risks?
I’m not sure if you’re running the latest frontier models. Every single engineer (and myself included!) has mentioned numerous times how nothing beats AI in debugging. My other guess would be that you have non-standard set up that Opus 5.5 doesn’t know off the bat how to debug.
I wouldn’t say the same a year ago, but usually we get preliminary reports during incidents before someone can even auth to debug. Almost all the time, nowadays, the report is correct. We obviously verify it.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
It is far more often the case that you will spend more time getting the answer out of the LLM than you would've gotten yourself. They suck ass at troubleshooting systems, very frequently jumping to completely implausible scenarios that no human would think of as the first thing to try, and almost never asking questions to clarify the situation the way a human does.
This has not at all been the case for me. LLMs (even the more-capable local models) are exceedingly good at troubleshooting when given proper context (clues, hints, a decent system prompt/skill that provides even the barest hint of how things work)
What you're describing sounds more like what would happen if you gave a chatbot without access to the repo as little context as possible and asked it to come up with a solution, blind.
Given the right tools and instructions I find them to be consistently superhuman. It's unreal.
I have an automated system responding to issues occurring on oceanconnect.ca, and every single diagnosis has been correct for months. Of course I still verify, I tune solutions, I review everything. But it's right. It's not a trivial system. It has a fairly large frontend, a large-scale model processing pipeline on the backend, a fairly complex database, and a queue for getting new data into the system. Claude does a stellar job of navigating and understanding problems, even when they're not 'real' or as they initially appear to be; things like quirks in old versions of Safari, or the model pipeline fails due to data integrity issues from the providers.
I've rarely encountered cases where I need to steer a frontier LLM much when troubleshooting. They're exceptional.
Sorry, this might be a thing in your org, and I have no idea hoe I can convince you, but it’s just not true in our environments. Maybe you might have a very non-standard set up, but frontiers do amazing job.
I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain. You can even have AI explain the code to you, so it's easier to understand.
I think a lot of this skill atrophy talk is over-blown. It definitely happens, I experienced it some, but as long as you slow down a bit, and clean up and understand your code, your skills won't atrophy. Actually, AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
I think perhaps my use of the word skill atrophy here is wrong.
Skill atrophy is when you're good at something, and because you don't practice it regularly, it gradually wanes over years, like when my former CEO said "I used to be good at programming" because he made a tiny chess computer and sold millions of copies, and now he... just doesn't code, and hasn't for decades.
Skill atrophy would be if I opened my text editor and I wasn't able to code after two years. Being a polyglot, there's always some language that feels rusty. And now they all do. But it hasn't been long enough to feel the atrophy.
What I'm describing is laziness to understand a problem domain.
> I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain.
I could. I do it constantly for projects where that premise is acceptable. And I've worked on projects where many people merge to main once an established set of steps has completed. "How does it work?" and "Show me the thing working." are good gates. Understanding (or even reading) the code is not a mandated prerequisite.
As an Open Source contributor? No way. Exactly because I'm contributing to projects where people don't share basic premises. Some people code by hand, others use coding agents. If I were compiling stuff and everyone else was hand-writing their assembly, I'd read and tidy up my generated assembly, too.
> AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
Totally. Better solutions come through those explainer sessions.
I’m doing interviews as a senior sre and frankly it’s extremely weird.
I’m being asked very open ended scenario-based questions, I’m answering honestly, and it seems to be going well (i’m moving through the interview funnel).
I’m not getting asked deeply technical questions. No weird obscure trivia. No stack-specific things. No linux internals. No weird networking trap questions. Coding part is essentially gone.
My interviewers so far have mostly been other senior SREs.
It looks like a market where a senior sre can recognise another senior sre. Some kind of closed circle.
Tell me more... I'm also in the same boat, and part of me feels more prepared to do high-level open-ended exploration as opposed to code-based testing.
Ah yes, it's just everyone else that's stupid; not you. You're still more capable than Claude right?
And just in case you haven't used the latest AI models and are being genuine, no it really is superhuman. Even if you are an AWS CLI expert, nobody in the world is going to be as fast as AI. And of course nobody is an expert in everything. You're always going to find some random domain where AI is read all of the research papers on it and you haven't.
AWS CLI‘s API surface is huge, it’s all of AWS. You can abstract over that with your own tools. Agents are great at these little greenfield scripts, both building and using them.
Skills that you need come and go. I owned a commodore 64 once and knew how to use it. All gone. I fired up an emulator a few years ago and then had a big "now what moment" where I realized I had no clue what to do with it. I had a vague memory of inserting tapes and issuing commands to load stuff from it. Function keys were important. And there was a manual I once had that explained it all. But no muscle memory. Nothing. I'm sure I could have figured it out eventually. Same with DOS. It's decades ago since I did anything on a DOS command line. All gone.
This doesn't matter. That stuff belongs in a museum. I've used AWS pretty extensively a few years ago. I switched to gcloud and other providers later. At some point in the last year, I started delegating all of the pesky work of interfacing with that stuff to AI. It's amazing. I've always referred to AWS documentation as a maze of misdirection. Just endless amounts of documentation mixed with marketing pages that are completely useless. All links lead back to those. So you are constantly fighting their marketing people to get at the bit you actually want which is a mess of fragmented reference documentation with parameters, endpoints, etc. Staying on top of that and keeping it in your head is a skill that will atrophy that I never cared about.
AI is great for figuring out micro detail like that. It's also great for coming up with a bigger plan that involves stringing together lots of tools, commands, scripts, and what not to execute a high level goal of "I want to run this thing over there". And with a bit of work you can actually do this the right way as well. It's fine. I never really liked devops to begin with. I can do it and I appreciate it done properly and can tell the difference. But I've always hated how it sucked up my time when what I really wanted to do was move our product forward. Instead of which I ended up dealing with a lot of arcane AWS or whatever crap that just needed dealing with. And there's always more of it. Just endless amount of micro detail. AI is great for getting on top of that.
SRE was always about understanding; understanding data flow, understanding bottlenecks, understanding logs, understanding application architecture, understanding end user functionality, understanding break points, SLIs, SLOs. Understanding how to mitigate, gracefully deteriorate, restart, restore, recreate, scale, etc.
There is no understanding with AI-generated applications, full stop.
idk, I develop a lot of connected systems: Pis, Macs, lights, Google speakers, custom ESP32 boards. Understanding the capabilities and communication methods to pool them together is still very much part of what I do. If you're thinking one prompt -> production via AI, sure. But I don't think we are anywhere near that because there are so many moving pieces which still require human sign off: legal, design, QA, etc. To me, it just shift ups the level of understanding rather than replaces it.
Seems like an odd criticism given that programming has always been about abstraction. But sure, I don’t know tree-level details (functions) but I also wasn’t doing leaf-level (bits) understanding because that became obsolete. Now I’m on forest level and expanding to eco systems.
You can always know more but at a certain point, after you’ve built enough validation and safe guards, you find it’s not the best use of your time and the things you can make beyond your specialization is quite remarkable.
Also, I think the toothpaste isn’t going back in the tube with this
The internet is full of people doing the “AI = compiler” comparison and it’s just wrong.
Maybe on a really small scale they can resemble a compiler because the number of possible outcomes is so small that it’s nearly deterministic.
But at any larger scale AI will not be deterministic anymore and at that point it will be relevant what the resulting code looks like. Because it can and will be slightly or very much off from your instruction.
If all inputs match a spec then why does it matter that the LLM is non-deterministic? I agree we might not be there yet, but the timelines for progress is shrinking and we are more likely to get there in a matter of months than not.
I figure out how to do something, wrap it in a bunch of functions, and then kinda just use it. If it just keeps doing what it should, I may kinda forget how it works. But I don't forget that I made it, and that I understood it, that will never not have happened. So in a way, I am able to "forget" some things because I still know I knew them. And I trust that programmers who write programs that I dream about writing also know what they're doing.
With LLM, you accumulate stuff with from knows where, ultimately fished from a vast ocean named question mark, by whatever turns of phrase you and involved agents happened to use, and what the temperature was at that moment. Which is of course where all the harnessing and looping and running code on the output comes in, but still, ultimately you have no idea. Which is fine for prototypes, but long term? Good luck.
I guess I'm just lost on things not being solved to a degree. Maybe its that I'm an iOS engineer where things are so locked down as far as goals and efficiencies that AI doing the work is verifiable by running the program, setting up some pre and post verification of code, and writing tests that look for certain edge cases. To be frank, it just seems like a trust thing. I trust it to the degree I am willing to to verify it.
“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger Dijkstra
I've thought a lot about this quote, and the best abstractions I use are absolutely this. I love array indexing as an abstraction, but in the back of my head I have pointer + index * size. I like having the kernel do memory mapping, but I know there's a buddy allocator and TLB cache there.
For the last 10 years, I don't think I met a lot of senior architects or dev who could understand all that.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field.
It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
I've seen these people rapidly disappear in the last 5 years at a big tech co. Old UNIX heads that had experience back to the 80s being made redundant. Who is replacing them? Mostly offshore 20 something year olds on a fraction of the salary a tenured person had, being told to just follow runbooks, pasting the runbooks into Claude or ChatGPT. These people can't explain anything, didn't really cut their teeth on doing things out of curiosity, and if asked a question, will just be playing copy-paste-back-and-forth with a LLM.
Agree up to certain levels but your examples are too specifc.
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
There's nothing unique about junior SREs. They're replaceable with almost any "vanilla Dev". You don't really build a unique experience or expertise until Senior+.
The difference between a junior SRE and a dev is that during an outage the dev will take responsibility for their own code only[1]
While the SRE will follow the problem to the next layer or component and try understand what is going wrong there even though they never heard of the system 10 minutes ago.
[1] "I know I'm on call for this team but the guy who sits next to me wrote that code" is something I've heard way too many times.
Have the AI explain it to you. You probably won't understand it as well as if you created it yourself, but there is (the potential for) lots of understanding.
What would you think of easier-to-debug infrastructure building blocks? I'm building an easier-to-debug web server and reverse proxy, and I want to know how this would compare to trying to use AI SRE agent to reduce debugging friction. Or maybe both complement each other?
There's a lot of chatter about how AI is advancing mathematics at the expense of human understanding. This is going to be true for software industry also at some point. The main gripe of mathematicians is how AI is discouraging young people from persevering. Right now job market is rough for junior SWE and unlike mathematics professors, nobody gives a damn. A few years down the line we'll see a real shortage of software engineers who "understands" and it will come back to bite the industry.
Join me in my War On Acronyms! (WOA) They are opaque, a form of in-group signaling, and the number of collisions is ridiculous. They hinder the flow of information much more than they facilitate it.
Thank you. I have been a software dev a long time. More than 30 years. I did not know what this acronym stood for. I read the article and the article it linked to. Neither spent half a sentence explaining what the acronym means.
Terraform deskilled us too. No one is saying, "SREs should configure their infra by hand sometimes to not lose that skill".
If AI can handle it, great! We have a new tool -- let's make the tool better.
Let's figure out how to use AI to make us detect and resolve issues faster, and even better, prevent issues. I've always said we can't consistently hit 5 9's without automatic detection and remediation. We are there now.
> SREs should configure their infra by hand sometimes to not lose that skill
Actually... yeah, I'd totally say that. Some SREs in my company never wrote a single line of HCL that wasn't a call to a in-house, pre-approved module. They think they know Terraform - they don't, and they don't know what's under these modules either.
Terraform, as with any language, is mostly useless for humans now. It is for the agents full stop. You are correct though, we need to know the underlying architecture and most importantly why it is the way it is.
That is not true of Terraform, nor any other language. If you aren't at minimum reading and understanding the code, you are bad at your job, full stop. More realistically, you should still be writing a lot of code, because most of the time you will waste more time trying to work with LLMs than doing it yourself.
I don't even have experience with Terraform at all...
I'm using something different myself: a Docker Compose + reverse proxy config + some other support files, pushed to a private GitHub repo, which then triggers a GitHub Actions workflow that leads to the repository being pulled on the VPS and the infrastructure to be "rebuilt" (docker compose up, reverse proxy graceful restart).
Opus 5.5 is quite effective at generating Terraform for a "typical" app that needs a load balancer, compute, and a database or two. That's the exact sort of cookie-cutter, rote work that AI is perfect for and no one is really going to miss.
SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
DevOps was a reasonable idea, but it rarely actually manifested. If a company had an actual "DevOps group" separate from the developers, by definition they weren't doing DevOps.
Part of the problem, and what ultimately drove me out of coding altogether, is that Ops is a specialized skillset completely orthoganal to the business of building applications. And, I was really bad at it due to a complete lack of interest. As cloud environments get more and more complicated one ends up spending more and more time dealing with that at the expense of building things.
Conversely, there are lots of folks who love ops type problems and are not well suited to building apps. So, it makes sense to put them on separate teams where they can each do the thing that they're best at, rather than fumble through the parts of the jobs they hate. But yes, at this point it's not really DevOps anymore, it's just ops.
At one point I was very hopeful that the progression from Infrastructure as a service to Platform as a service was going to simplify things. Instead we seem to have ended up with mind boggling complex services.
Some places do have really well built out platforms where logging, instrumentation, databases, deployment methodologies, IaC, etc have well-built happy paths that exclude a bunch of accidental complexity.
I think a lot of other places are either too small or grow too quickly with the wrong set of values and admit too much into their "platform" which had very little design to begin with.
IMO it was a worthwhile experiment that probably went too far before being rejected.
Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.
The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.
It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.
Isn't that just anyone with an "architect" job title though? I don't think that kind of dev is particularly rare at all.
I've found that "ops" has become far too overabstracted and jargony. Their moat is endless "certifications". Their "experts" are the dogmatic personality types who try everything in their power to prevent their feet from ever touching the ground like it's lava. They lack the kind of dev or sysadmin experience where you have to stand under your own constructions like a Roman engineer.
Any formerly-known-as "full stack" dev who kept at it for the last decade and a half would run circles around anyone in ops. That's every senior dev working right now who doesn't like to manage people.
It used to be that the ones who went SRE just wanted a pay bump for a bullshit job of relatively light coding puzzles and parroting stuff in meetings, but I guess that well has dried up with "AI".
Right but the dream and promise of devops was that early-to-mid career line engineers could do these things, that these weren't rare skills and combining them wasn't particularly challenging. This turned out not to be the case, these skills are the requirement not the outcome.
But if you have the budget and connections to hire a department of people with these skills and experience, and a problem interesting enough to keep them engaged in the day to day building, and a leader capable and charismatic enough to align them all into the business's goals... you should do that.
Coming from the Ops side, “full stack” usually translates to “our entire infrastructure stack is managed AWS services, we don’t know how they work”.
It turns out about as well as if you set the Ops team to shipping features.
Eg I’ve seen more than one full stack dev team build a synchronous RPC-over-message-bus framework because they can’t figure out how to make HTTP or actual RPC calls reliable. It is hands down the most poorly and overengineered thing I have ever seen on a service that takes single digit qps.
The ops teams fight that shit tooth and nail because if it blows up, they’re the ones who get called. And it is 100% too late to fix it by then. Ripping it out will take months, and the dev team will say they’ll do it, but they never will because the VP is leaning on them about some feature.
You either say no early, or you are forever on the hook for whatever abomination the intern has cooked up.
> Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful.
You’re so right. This is the key, whether a team is doing devops or not.
One of my major contributions when I took on a CTO role was getting dev, ops, and support teams to talk about the pain points. There were so many easy wins to make things run better because the teams didn’t understand the difficulties the other teams were going through.
Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.
conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.
> Ops is a specialized skillset completely orthog[o]nal to the business of building applications.
Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.
No, what you descrive is SRE. DevOps was the idea to give people that work 100% as developers the ability to do some amount of ops work (let say 5%), as a way to break the silo between Dev and Ops that existed in older organizations.
I am skeptical that either label is sufficiently well-defined and more importantly well-understood within the industry to offer definitive statements on the differences.
Of course it's misunderstood and misused, but that doesn't mean that more discerning people can't understand where it came from and why, and then used the term properly.
yeah, I am a software developer, not a system administrator.
In the early days of devops it was primarily focused on supporting developer productivity (CI/CD) but quickly morphed into sysadmin once cloud computing took off.
I think a more apt analogy is a chef who is not waiting tables. Which, by the way, are typically completely different roles in a restaurant.
Your comparison might work better for a dev with no testing experience. Write the code and throw it over the wall. But I fail to see how an application engineer should need to worry about the intricacies of, say, setting up virtual private gateways.
I disagree: ops is related to dev, because good dev work makes ops easy. But they are different: being good at ops requires a mindset of curiosity, risk management, and being willing to chase things to to last details.
Yes, big companies have a tendency to over-structure things. Devops? Sounds great, let's have a dedicated team for it. Agile? Absolutely, and to make sure it's done right, we need a rigid process around it.
The rigid process around agile is absolutely mind-blowing to me, and it seems like it's everywhere in corporate environments, but my current client dials it up to 11. Maddening.
Real DevOps is actually just Dev - in that the Devs are doing the Ops. The people called DevOps are just there to backstop the devs oncall when shit hits fan and you need someone more experienced in ops / incident management to escalate to.
I am consistently surprised that I'm unique for enjoying, and if I may be so bold, being pretty good at, both Ops and appdev. Now I'm not the greatest in the world, but I really do firmly believe now that most generalists should strive to be able to do both at a journeyman level.
To my understanding, this _is_ DevOps in the original sense of the word, and the version where they are a separate team is an offshoot bastardization.
At my last job, it was somewhere in-between where we had DevOps specialists who were regular software engineers who allocated the bulk of their time to building / running infra via Real Code (Pulumi vs. TF, minimal YAML wrangling etc) and built paths for app devs to self-service as much as they could. That, I felt, was a really excellent setup.
Devops definitely still exists, I've seen it. It is when people are responsible, even in the middle of the night, for the systems they fucked up engineering in the day.
SRE at google works closely with devs to architect and build systems together. Most of them are on the SWE ladder, are able to bug fix and scale directly. They're not just ops.
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
It turns out it's easier to hire people who specialize in scalability, reliability, debugging of distributed complex systems, cost ops, efficiency gains, etc. than it is to instantly ensure every single engineer that works there is an expert in all those things, without anybody on staff who is an expert in them. Much like database skills, IT security skills, etc.
It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.
Oddly the industry mostly rejected googles application of SRE even as they adopted the term. I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
> I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
You didn't look around very hard then. I acknowledge that certain areas of the company can consistently use subsets of a role, so you may not have encountered it personally, but your view is definitely wrong about the role in the company as a whole.
It’s cause they don’t know what they are talking about.
Easier to shit on people and things with hot take then actually have a grounded and coherent argument or realise that different companies/teams have different constraints, problems and environments they operate as well as different solutions.
When I worked at Google SREs only handled legacy products that were maintained but not in active development. SRE was its own vertical separate from SWE and they would only take on maintenance of a new service or product if SWE could prove to them it was stable.
"Zero" Interest Rate Period, during the COVID pandemic, where most governments offered very cheap bonds or printed money to reduce interest rates. This caused a lot of investment as money was "cheap" for investors and needed to find a worthwhile investment in order to grow faster than inflation.
This caused a lot of money to be poured into all kinds of projects, not all of which had a positive return on investment.
We should do it again but the right way this time.
Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.
More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.
Continuing the never ending trend of Sys Admins/DevOps/SRE job title holders with skills in specific tools rather than foundational skills always wondering why they are out of a job every 5 years.
That's reinforced by how hiring is done though. I rarely if ever run into companies that hire for fundamentals. Only the Google tier of companies do that.
That’s what companies hire for. I’ve learned those skills and no one ever really wants them.
If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.
It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.
It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.
I never understand this sentiment. I do SRE/DevOps/Platform Engineering/Infrastructure Engineering, and I definitely do need to know how underlying technologies work. Even when everything is kubernetes and YAML.
If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.
With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.
I think you’d be surprised if you look at the younger (post-2000-ish) companies. Specifically about this part:
> If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that.
You would be surprised how many workloads either are simple, or can be ham-fistedly shoved into a “simple” architecture.
Eg I’ve seen a lot of Postgres-on-k8s solutions that are basically just Postgres on bare metal with a bunch of useless abstractions. It writes to local disk, moving the pod to a new host is basically restoring from backup. Wrap it in some tool that ingests YAML and call it a day.
k8s is sort of the king of this. I hear people say they understand k8s and I can’t help but furrow my eyebrows. I don’t even pretend to, there are so many insanely complicated edge cases.
Eg if I’m sending UDP packets to a service and scale the deployment from 2 pods to 4 and the host for one of the new pods crashes and comes back up with a bad iptables rule, what are the range of potential behaviors k8s would show? What’s the difference between blocking say all traffic vs only traffic to one or two of the etcd instances? What if it’s blocking half the hosts in the cluster?
Storage has its own weird vagaries around single vs multi mounts and race conditions therein.
k8s is the king of “I don’t have to know that, the platform will do it”. I’m not sure I’ve ever heard anyone outside networking talk about whether a pod will saturate a NIC. Just chuck it in the cloud and let it figure itself out.
Kubernetes is probably more responsible for deskilling, or at least constraining, the expertise, of SRE/devops/platform/infrastructure engineers than anything.
I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Yea cause you wouldn't escalate to 50 people on the weekend when one person can do it. Op is basically saying they never worked on a big enough system.
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
If you look at the history of SRE, it was distinctly NOT sysadmins, Google specifically wanted software engineers hired to automate scaling as well as sysadmin type ops work.
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
On-call problems are mostly just a result of Kubernetes' "cattle not pets" brainrot ideology where people throw local simulate-ability out the window because the Rube-Goldberg web of pods and operators and cluster state gets too complex to simulate locally or keep in your head (esp. if code is located in 50+ microservice repos).
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
It's not that crashes don't happen, it's just that they don't "randomly" happen. The error codes from bad hardware are predictable (against kernel bugs... well I use an "allnoconfig" kernel, which disables like 95% of the kernel). Bad hardware, bad network, bad RAM, all of these are simulateable in QEMU or Linux (tc netem / blkdebug).
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
i'm not questioning your setup, I'm saying that if you need a 10,000 machines to run a system you're going to have kernel bugs and hardware bugs and you have no choice to treat machines like cattle.
> Real ones are oncall for the systems they build.
Code of Hammurabi's Law 229: "If a carpenter has erected a poorly constructed house, so that the walls cave in and kill the homeowner, that carpenter shall be executed."
AI isn't the only cause of deskilling. I've seen a few people go from coder to architect roles. Architects, being the decision makers, talk to vendors, CxOs, consultants,... and mainly spend their lives in powerpoints and visions.
When an engineer stops coding, managing servers, ... it takes about 6 months to transform even a really smart person into a buzzword factory. It is scary. People like Oracle or Microsoft sales are really good in decoupling all of us from the relevant day to day concerns. It is reversible too, with former architects getting really angry if they have to deal with the software they themselves procured or envisioned.
My take is that you need non stop contact with reality, both the computers and the customers, or you start to drift. I fear AI is causing this in all of us.
> To put it briefly, automation has made it more and more unlikely that ordinary airline pilots will ever have to face a raw crisis in flight—but also more and more unlikely that they will be able to cope with such a crisis if one arises. Moreover, it is not clear that there is a way to resolve this paradox.
I bought Sully Sullenberger's book in early 2012. He's the pilot who perfectly ditched his Airbus A320 in the Hudson River after losing both engines on takeoff to bird strikes.
One of the things from the book was how Sully spent a lot of time investigating other pilots' accidents: 'was there anything that pilot could have done to have survived?' When he got to his own emergency situation over the Hudson, time slowed down and he decided over the course of a few seconds that ditching in the river was their only option.
"One way of looking at this might be that for 42 years, I've been making small, regular deposits in this bank of experience, education and training. And on January 15, the balance was sufficient so that I could make a very large withdrawal." https://en.wikipedia.org/wiki/Sully_Sullenberger
I like this analogy that I've come up with: Non-writers can produce impressive articles that are unbearable to read the same way non-programmers can produce impressive software that is unbearable to use.
I think people are getting funneled into a new career path of being a technologist or technology director. People who can ask the right questions at the right time and direct attention to the appropriate areas given the goals and feedback.
People don't appreciate how much of engineering is just as you've described.
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
Admittedly, a lot of my identity was wrapped up in those hard earned skills. I have empathy for those struggling to chart a new path for themselves. It does not, however, change the fact that tools evolve. Ultimately, using tools effectively is how we are measured and while you may have needed a shovel yesterday, today you have an excavator.
> "...imagine the vibe-coder who has the same curiosity and inquisitiveness that you had..."
We're required to imagine them because they don't seem to exist in the real world. The vibe-coders seem almost entirely composed of the "I commit 20 kLOC a week without reading a single line, lol" or the "LLMs are like calculators/basic arithmetic skills are obsolete" lunatics.
This has always been the most valuable part of the job if you ask me. Its just that the other parts of the job were quite hard to get right letting people build a career out of that specialization.
I dug into general AI deskilling a couple days back[1] - no good signs anywhere, but there's a few things we can do like make concerted efforts to do programming (like a coding hour every week), do a Leet code question, and getting the agent to quiz before merging something (stolen from Notion).
Let's not rush to such judgment - our desires change as we age, the field has changed (including what nose-to-the-grindstone coding looks like) and burnout affects many.
Plus, some people enjoy the rigors of solving in-the-weeds code problems and others enjoy building systems-level designs. This was always true.
You guys act like the React component nonsense from the past 10 years is the same as the code I was learning at age 16. It's by far and away completely different. And none of it has been audited by Daybreak.
I'm very happy to let the robot handle any incidental React or other extraneous tools that end up adjacent to the core work I'm doing. It avoids getting blocked without interrupting teammates or spending time and cognitive capacity on stuff that has no lasting value in my work.
I've been having this same gnawing feeling. I just started reading the book "The Collision" by Eric C So which promises to cover this subject. I haven't finished it but seems decent so far.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I don't know if it's true. I didn't program at all between 2000 and 2015 and I've been able to restart without too much effort. I forgot many things, but I found it relatively easy to recover...
I'm more worried about people that did never program and started directly with LLM support...
I’m personally using it to augment my coding more than replace it. I find joy in it.
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
Nice video, but I have said it before, the difference between just “vibe coding” and reliably using AI is the specs you need to write first and the tests after. If you start prompting the end goal/s the AI will basically guide you on what needs to be done and you just accept it as long as it runs, and this is following what most software developers were doing before LLM, the agile way, the quick ship way, the daily standup and ship first discus later way, the code pairing way, which worked mostly (even though I hated personally) in teams, so these developers are replacing what they were used to do with an agent and that’s the mistake. However, if you go with traditional engineering ways, waterfall, specs based, blueprints, and then tests, you control the agents and guide them not the other way around, in your specs you define the tiniest possible thing to what library to be used or excluded, the methods and approaches, among others, the agent here will produce the syntax and help you in the tests, just like in other engineering aspects how machines didn’t take over, they follow your specs and frameworks. The syntax is the past, AI can produce better syntax than any human alive or all combined, if you think otherwise you are delusional and still didn’t move on, it’s time to stop the “developer” mindset and do the engineering one.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
We use AI at work, Claude specifically, and it’s faster than avg SRE for trivial tasks like cross signals correlation. But it requires a lot of effort to build a harness from your company contexts for it to work like an SRE. And it’s just a tool so if you have to build your knowledge to make it useful, otherwise it’s just a blunt. Or in other word, incompetent people are incompetent no matter of tools
Is it really that complicated to set up? A basic version is make everything work from a CLI in terminal coding agent of choice (probably already mostly there), hook it up to alerts, and let it run in a corner?
Not OP, for me the issue is nuance of all of your infrastructure, application and outside layers. There's a lot of documentation and such, but often times things don't work as expected to a degree and it requires filling in the blanks correctly. The amount of times I've had an AI Agent misinterpret an alert and assume it was XYZ when it was some random layer it had no idea about is often.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
From what I've seen the more complicated the deployment and the more tedious the investigation the more useful the automation is. The agent shouldn't be interpreting the alert much beyond gathering symptoms and identifiers for drilling down into all the underlying data and code. The security side of this is very tricky because the speed that makes the tooling useful also makes it hard to supervise.
What contexts would the AI model need for analyzing, let's say, cluster-wide incident other than the observability data that such systems anyway collect already?
Very often the causes of an incident aren't visible in metrics because the relevant metrics aren't collected. Sometimes an incident is caused by external actors, etc...
Yes, "entire" company codebase is a stretch as is design docs, status pages etc Helpful yes but not a hard requirement per se. In many places you won't find such level of details available and yet the incidents are still being solved
I don't even know if this is an AI thing. I've worked at places that have varying levels of operational maturity, and some of the ones that had very old SRE orgs also had a lot of their SREs operating mostly in the tier 3 given by the article.
One of these orgs had basically all of the noisy pages under control through automated remediation and a culture that pushed towards aggressive automation of pages that had the same root causes. Almost everything I was paged for was some novel or uncommon fault condition. Developing an intuition about how the system worked (well, more specifically, how the system failed) took a lot longer because most things I had to look at were some edge case failure.
> Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.
> But here’s the catch.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
How did they do this before AI? I have done a bunch of complex production level debugging and these Tier 3 incidents are always "starting from scratch" never seen before issues. The main "intuition" I built is persistence and verifying/refuting random hypothesis. In other words, I think we can build an intuition for Tier 3 issues only by engaging with Tier 3 issues.
Although, psychologically, some success in T1 and T2 issues is what gives us the confidence to even approach the T3 issues.
When you're just starting out in a system, every incident is a novel, never-before-seen issue. That's putting aside existing sources of knowledge like coworkers and runbooks of course, but I've had to do "software archeology" on lots of stuff because coworkers are gone and docs are wrong. But that's where you build up that intuition, like how the company typically builds things, how deployments usually work, what's that terrible load bearing service installed 10 years ago and not updated since, etc. Yeah every T3 incident eventually is novel, but being able to cut out the non-novel parts helps a lot
Deskilling has been happening for centuries. One doesn't farm their own vegetables, hunt for their meat, slaughter their own livestock, maintain their own vehicles (ahem - remember the horses and carriages). We've even stopped shifting the gears on our cars. Why are we complaining about AI?
The biggest problem I see with AI isn't deskilling. It's the same dopamine hit one gets from social media but for their profession.
Hang in there, in a few years we'll be mining gold cleaning up the mess.
I work in an AI sceptical organization, it's not that we don't use AI agent or consult chat bots, but they are kept a good distance from production. Talking to other SREs in AI heavy and AI first companies I don't see that ending well. SREs struggled with operating applications from developers who cannot or will not understand how computers and the internet work. Now many of the junior developers are even worse and some of them are given the title of "DevOps" and they are using AI to create some of the most broken ass Terraform code and bastard Helm charts to run their broken code. Some SREs are spending more time fighting agents and agent written infrastructure code than they would have, had they been allowed to write it themselves.
If tokens becomes to expensive, we're left with a ton of broken code and infrastructure that can only be managed at an inflated cost. There will be good money to be made consulting and cleaning up the mess of AI boom.
The devops department at my company, which has around 500 services across a load of kubernetes clusters / regions are all amazed at how well gpt-6.1-luna is at their jobs. They've all been using it exclusively for 2 weeks and between them have spent like $400.
The take where in a couple years there will be a massive revert or that anyone cares about terraform indentation or a shitty yaml template is wildly optimistic.
I do hope you're right though, but I'm not seeing it from my side of the fence.
I disagree, it gives me the ability to apply first principle knowledge unhindered by whatever fuckass idiotic idiosyncrasies a particular cloud platform considers “good design”
AWS, Azure, GCE, Alibaba Cloud - all the same crap, different names and slightly different jenga towers.
It applies to surprisingly a lot of crap and it’s a good substitute for where we can’t put automation in yet.
Imagine SRE that very quickly triaged the incident, gave solution and it turns out that solution made system ever worse, and after that he/she said: "You're absolutely, right!"
Deskilling is not necessarily a bad thing, if what you refer to as skills are mundane things that should be automated. I do agree with the author in that people are becoming lazy when it comes to critical thinking and evaluating the decisions or work that AI presents to them. These are people who have little passion to do what they do, and for whom their role is primarily a chore.
I don't think this is as black and white. You skill up by spending time with the domain. Of course, you learn better when you are involved in situations or jobs that actually matter. But getting rid of all the "mundane" distances you from the nitty-gritty and you lose that feel for the problem.
Right, there's a reason students that never dealt with a real problem, even when they get As, aren't as good as professionals. You need to do real work to get really good.
Yes. I bet it was really fun debugging punch card computers.
Also, skills can be re-acquired, if needed. People are acting like it's the end of the world for them, like their knowledge will become impossible to understand anymore. I'm not as fast at solving equations as I was in high school, but I still understand the logic behind it.
As an SRE Engineer (RAS syndrome, I know), I was just having this talk with my supervisor.
My original career/learning path was for a traditional kind of SysAdmin role, because that's what I like. Doing Linux administration and such. Then our whole stack went AWS/cloud-native, so, ok. Seems like no one cares about someone who can actually SSH into a Linux server and figure out what the problem is--they want everything to be built into CloudFormation so that when a server is misbehaving you pull a switch to trapdoor it into oblivion and drop an identical one into its place, in the hope that this one works. I didn't relish the idea of moving away to the increasingly abstract, but alright, I can deal. Reformulating my learning path into being a Certified AWS Full-Stack Engineer or whatever.
As that was in progress, my company drank the AI kool-aid and went all in on AI. Now I honestly don't know what I should be training for. They've been pretty consistent on "No new staff, just AI", "You don't need X, you just need AI"--so I am honestly not even sure at this point if my company even wants me spending time diving more into the AWS stack, or if their attitude is "The AI knows everything--put all of your time into AI prompt engineering and just push whatever buttons the AI tells you to do."
The desired company skills continue to drift from my "personal" desired skills, but if this is an industry trend I'm not sure I have much of a choice, because I need my job.
> Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures
AI is already pretty good at this in "SRE" workflows.
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
If it deskills that guy who reaches for complex bash when they should reach for golang but they don't want to bother with writing the build layer in the image, then I'm all for it.
Plenty of people citing aviation. Aviation is deeply regulated at a global level. This is never going to happen with software engineering. Even where it already happens “locally” like in some military sectors, the results are self revealing. It gets crushed by equivalent unregulated competition in pretty much all metrics.
I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.
> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.
Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?
SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.
I'm glad you apparently live in an ideal world where every incident is fully root caused and fixed at the origin, and all your coders produce perfect, bug-free code.
Every incident having a root cause is required. If your organization doesn’t do this, it basically means they don’t value reliability at all. The best SREs quit. Well either that or the SREs have the ability to transfer oncall responsibility to SWEs. Whether or not that can be fixed at the origin is an entirely different matter. Bug-free code doesn’t really matter: the best incidents are the kind that reveals hitherto unknown bugs.
I've started to use the term "downskilling" for what a number of companies are going through in the age of agents: layoffs of key employees in the name of increased efficiency, developers being incentivized to outsource their thinking when writing code, tests, specs, docs or reviews, agents increasingly replacing communication between team members...
I'm wondering if it would be a good idea to have the AI occasionally throw a wrench in the works of the dev environment Chaos Monkey style to keep the SRE team accustomed to handling responses.
No it's forcing everyone to realize all the jargon was made up to sell books and land speaking engagements. Money from patents (2016 Alice ruling upending that whole scam).
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
Is HN upvoting just because they agree with the title?
Clearly the "author" didn't even put in the effort necessary to make a real title. Not "SRE persists but a skill crisis looms" not "SRE careers remain but competency lowers in spite of AI" not "AI lowers SRE skill but demand is steady". Moreover, hasn't this type of insufferability been dramatically reduced in recent models? It screams that in addition to being slop, it's outdated.
I wanted to use this post as a source to write on my web server project's blog on why debuggable web infrastructure matters even if AI SRE agents exists, but the idea got delayed and I ended up posting it on HN... :')
Here's a thought experiment: why is it that I can't do long division?
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI makes stuff that was hard - someone's bread and butter - possible to ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for tasks that used to take forever to become something to steamroll through on your way to a much greater outcome. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
This is a poor analogy because I don't think your inability to do long division will cause as much damage as an inability to bring the site back up in an emergency situation.
Except as every damn AI app says: AI is prone to making mistakes. If AI deskills you with a non-zero chance of getting the problem wrong, that seems deeply, profoundly bad.
You can type the same long division problem into a calculator 65,000 times and you will get the right answer 65,000 times.
Computing tapped out on sources of leverage for solving deterministic problems a while ago. Computers got fast and our algorithms got really good.
We're on to the next frontier, and it's a bit weird. Jagged frontier and all. But it's opening up new worlds.
Luckily loads of problems are amenable to a non-deterministic approach - even better, many have lots of solutions. SRE is in that bucket. There are as many ways to SRE as there are SREs
(source: one of the first SREs at Uber, co-invented Chaos Engineering as a sub-branch of SRE, hosted first chaos eng conf at Uber, played a key role in Uber's global SRE and incident response "Ring0" initiative, managed many SRE teams over a period of roughly 7 years before a career pivot. You could say I know a thing or two about SRE.)
To SRE well you look at the situation and pull from a huge range of tools. With such a broad range of solves, you're better to focus on the outcome: did we achieve our SLOs? Are they aligned with customer expectations? Focus on defining that and providing the right support for it to be achieved and you'll be fine - you might even get a surprise.
Of all the comments on this post, this landed the most.
I feel there is a potential to "distill" the frontier models into useful tools that can run on smaller hardware. Or distill that knowledge to produce hardware, compute or otherwise. Iff one day the frontier model become inaccessible, the tools that were carved with them may remain valuable.
They will remain most valuable if they are designed in a way that is maintainable even without LLMs. If not, they can be studied as artifacts, or reference designs and then recreated using human intelligence.
The analog I'm playing with:
You can rent time at a high end machine shop and use the best CNC machines to produce extremely valuable tools. Once you leave, you don't have access to those tools, so it is advantageous to produce durable artifacts that are valuable on their own — on a different frontier.
Imagine you're going beyond the frontier physically, exploring new parts of the earth or space. You can't bring the giant machine shop, let alone the machinery of society, the entire economy etc. What do you bring? What do you build now while you have access to the best machines to take with you?
I recently won the red ribbon, second place, at the Lambtown fiber festival, for my lovely skein of undyed handspun yarn. Weaving it is next. However, I am probably an outlier, and should not be counted.
(It was a full 250g of this fantastical kempy Herdwick, absolutely unlike anything you can get commercially, and I'm honestly more proud of it than of anything I built in six years working at NVIDIA.)
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
Edit: "HackerNews" and people can't do long division??
> in the macro, humanity becomes far more capable as a species
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
I think a closer analogy would be forgetting basic arithmetic as a concept. Imagine how hard it would be to work with more complex math without understanding what division is. How do you prompt the genie, or understand its outputs, without the foundational knowledge?
Is basic arithmetic something that can be forgotten? Studies have suggested that even infants have an understanding of basic arithmetic. There is a reasonable likelihood that it is something innate.
As we're only imagining anyway, let's ignore the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
You're taking the analogy too literally. The point that I'm trying to make is that in math, and a lot of areas, is that concepts build upon one another. To understand something at a 7/10 of complexity it first needs you to understand the 6/10 concepts that it builds upon.
I believe it is more than you've missed the forest for the trees. You haven't accurately explained what understanding means. One might not rigorously understand x÷x=1 but still 'naturally' see that having two apples and taking one leaves one. And that's the thing: The genie, as in LLMs, are analogy machines so you can approach problems without a formal understanding as long as you can grasp the analogies. We have real world proof with 'vibe coders' producing incredibly complex math without much formal understanding of it. They know how to describe the input and interpret the output using words that relate to everyday things that they are familiar with, much like the apples on the table, and it accomplishes their goal.
Perhaps there is a limit, sure, but I suppose that's where you lead with a concrete example to explain where that line is.
Honestly, as the expectations change I don't see you really have a choice unless you're in the top 1% of engineers and can essentially dictate your terms. Even then I'd say the clock is running out.
At some point (we might already be there) someone insisting on doing everything manually will be seen like a carpenter who does traditional joinery. They might exist, but they're never going to be a significant percentage of the market compared to 4 guys with nail guns knocking walls together in hours.
Because for the vast majority of cases, "good enough" is good enough.
I had never heard about SRE before and suddenly it's all I see. Modern problems (slop fest) require modern solutions (someone to babysit broken systems)?
Site reliability engineering (SRE) is a discipline in the field of software engineering and IT infrastructure support that monitors and improves the availability and performance of deployed software systems and large software services (which are expected to deliver reliable response times across events such as new software deployments, hardware failures, and cybersecurity attacks).[1] There is typically a focus on automation and an infrastructure as code methodology. SRE uses elements of software engineering, IT infrastructure, web development, and operations[2] to assist with reliability. It is similar to DevOps as they both aim to improve the reliability and availability of deployed software systems.
But I'm not a factory worker sitting around waiting for something to break. I'm so busy. I'm solving more and ever more complex problems. If anything, I'm becoming more skilled...
At least part of it is that I waste less time debugging and testing, and now I have more time to learn new skills and technologies (faster).
> And to answer the burning question in your head, our goal isn’t AI that replaces SREs. It’s AI that supercharges SREs.
Ah, that bullshit reasoning again. No, if you build something that "supercharges SRE", someone will be made obsolete.
I'm geniuinely sick of this framing, seeing people at my company building company-wide skill repositories and shoving contents of their brains into slick Markdown documents. Well thanks, now our AI knows everything, good luck with your future career (if there is any left).
Site reliability engineers respond to emergencies and keeps systems online in real time. They are perpetually on call. As a swe, it's much better than having on call rotations.
Previous discussion on an extremely similar article: https://news.ycombinator.com/item?id=49574167. This one even name-checks The Ironies of Automation in pretty much the same way.
I feel like I did re-read the same article, or at least a version of it. Is this how content marketing operates these days? Find a popular blog post, rewrite it and attach your company's name to the end?
It's fairly hard to understand to me, but also entirely unsurprising that swapping one occupation for another would be interpreted as a net loss at individual skill level.
You guys are still doing things, correct? Is my experience so outside the norm, or is stuff still challenging?
This is happening to SWEs too. If you have a certain amount of experience, you are capable of critically analyzing AI output. In Big Tech parlance, these are the Senior SWEs and higher. You require little to no supervision and can work independently.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
The point I got from the article was that AI is a cause for skill atrophy for any level of seniority, and tends to worsen over time. I can see this happening to myself in some areas, while in others I’m able to learn new skills I’m interested in.
idk.. it’s upskilling many people. AI seems to be very kind to people with ideas. It takes away the boring work and leaves the interesting part. So yes, it replaces and deskills those who only do boring work
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
I had an incident in prod years ago, that no one was able to solve from my team, and it took days for me to figure out what is happening. It was a tricky one.
Deskilling is good, actually, as long as it's the right deskilling. "Skill" in software development is partly raw IQ and partly knowledge of complexity. Programming author Fred Brooks (author of The Mythical Man-Month) categorized complexity into "essential" and "incidental". Examples of incidental complexity are: React/Rails/Django/Laravel makes a backward-incompatible change to their interface and you need to scurry around the codebase fixing invocations. Or a senior developers on your team have a weird in-house coding style and all the junior devs need to learn it to get their code approved. Or management wants to move everything from one cloud provider to another so you have to change a bunch of resource URLs. Examples of essential complexity are: Self-driving car algorithms. Making your code run faster on the CPU. Adding support for a new web standard like SPDY.
I'm not lamenting the days when you needed to pull a seasoned (read: exquisitely expensive) COVOL developer out of retirement to make a change. Now AI can figure it out well enough, I just need to know how the basic concepts work.
I'm (still) a software engineer, and I've found it crucial to solve some LeetCode problems from time to time. It's honestly refreshing to write code yourself, make your own mistakes, and not end up low-key mad at a robot for failing to understand your lousy description because, apparently, it still can't read your mind.
Seeing this posted a week after I had to emergency push cve fixes to 1200 servers in production is pretty funny.
Every person on my team is capable of networking, product engineering, database, job processing and we have command of the entire stack. We will push updates for CVEs with stuff we manually compiled. We own the build system and have the ability to change product as needed. We all have experience in product and we're all capable of pushing fixes as needed. Frankly the SRE team is by far the smartest team I have ever worked with at any company I've ever worked at.
Is your org at the point where it would make sense just to upstream patches for FOSS projects and pull rather than patch and push to just your infra?
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
I generally don't reveal much about my current job on a blog post for security reasons but I can explain some of the reasoning for having your own build system.
We have many packages internally so inevitably we need to build them which necessitates an entire build matrix on it's own. But once you have your own build matrix adding a new package is trivial and most build in under 10 minutes in github actions. One issue is we don't want to use Canonical's default gcc flags so like Gentoo I setup a make.conf that is specific to the suite (22.04, 24.04, etc). This has allowed me to target a higher tier instruction set which is it's own micro optimization. Folks that upgrade blindly from 22.04 to 24.04 will see their performance drop from gcc compile flags alone. Also we have people who have access to CVE fixes before they are generally available so that's also another important reason. Anyway we DO use upstream most of the time for setting up a project folder or normalizing it against a new suite but then building it is using different gcc flags and might be pulling in our version of curl or something.
The ubiquity of this rhetoric is not an indication of its correctness. Repetitive tropes like "deskilling", "losing the ability to think" reflect a deep seated distrust of AI alone. No matter how many people parrot these fears the fact remains that you choose your attention. My manager has started coding, heavily, aided by frontier LLMs. He instinctively knows his boundaries and defers much of the more complex work to his engineers. We use the clankers too. Have we been deskilled? Hardly appropriate. Trans-skilled, or learning a new emphasis built upon our long experience? Much closer to the truth.
I work a "dev"ops job, and like yeah, I'm pretty sure this field is even more fucked than software engineering is.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
Eh, I think they're still terrible at architecture. They'll find bugs but they'll try to make them go away instead of building a sustainable system.
They also have a propensity to try to rewrite gigantic amounts of code in one go making changes riskier and harder to scrutinize... without using more LLMs.
> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Aviation is different because their rent-seeking is enforced by the required legal hedging. I'm not sure the same incentive structure exists in the software side of things.
In other words, we might need to find some hobbies.
It is a good industry to learn from. I'm not suggesting SW should become exactly like aviation.
It indeed is and I like this analogy a lot. A lot of critical systems can definitely learn from aviation. It's one of the finest balances of humans and automation.
But there is a lot of software in aviation. And it is slow to develop, and expensive. And it has defects.
In general, it is of high quality.
In what way is requiring someone holding the title pilot to know how to manually fly a plane in case of disaster “rent-seeking”?
Rent seeking has a specific definition and it fits here. When Autopilot does almost all the critical work needed (besides announcing frivolous stuff in carbon-based voice), and you have mandatory seats for humans to babysit the Autopilot mainly to satisfy some law, the term fits.
I am not convinced humans can do much better in "disasters" than Autopilot, despite the emotional stories to the contrary (survivorship bias is one hell of a drug).
This take is so ignorant and provably false. I can't tell if you're trying to ragebait or if you actually believe this.
The autopilot does not do "almost all critical work", not even close. It only does some of the routine work (definitely not all of it). But the real critical work is when stuff goes wrong, dealing with emergencies.
The inherent complexity of aviation means that there are so many different failure modes and combinations thereof that no traditional system could ever handle all possibilities and edge cases.
I'm not going to claim that an AI could never do it. But as of right now, with current technology, well-trained pilots are absolutely critical to safety in even the most advanced airliners.
Also, the survivorship bias argument runs against you. Most events where pilots save the day are completely invisible. Like catching an incorrect clearance, a misconfigured approach, dealing with a technical malfunction, etc.
And even when disasters happen, it is very rare that simply engaging the autopilot would have saved the day. In fact, there are plenty of examples where the opposite is the case, and dropping to a lower level of automation would, in hindsight, have been better.
Your first sentence would lead one to believe that you have some substantial evidence to demonstrate the ignorance and falsehood you identified. It's impressive that you are regurgitating the same formulaic-template set of cliches that boil down to "X is so complex bro, how can you even?".
Note that I never claimed aviation relies on AI right now, most of the Autopilot systems are deterministic automation systems, but the original point stands regardless.
As someone studying for a pilot license, your take is literally ridiculous.
You should perhaps look at all the crashes where the autopilot stalled the aircraft on approach due to incorrect settings.
Really? Do autopilots diagnose threatening situations in the main cabin? Understand something unexpected happening on the ground? Visually diagnose situations? Properly identify threat levels?
It's genuinely worrying to me to see statements like this. As engineers, we of all people should know that there's 1 billion variables that could go wrong and another 1 billion we aren't even aware of yet...
As an airline captain, your take is completely ridiculous. The autopilot takes care of the mechanical act of guiding the plane along the desired path, but it does nothing else. There are hundreds of other tasks we need to perform in order to effect a safe flight, and the autopilot does nothing to help us.
You might want to fire up a flight sim or at least read up on what “autopilot” actually means. It’s pretty much 3D cruise control and that’s it.
Pilots are nothing near powerful enough to seek any rent on that situation.
Instead, they are the weakest part on that market.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
I’m using Elixir for every (web) project now specifically because the cost is negligible. I can't think of a reason to use anything else and the agents are _really_ good at writing it, particularly given the excellent structural guardrails that are in place.
Being able to ensure all of the test run freakishly fast async is a huge win too.
I don't get elixir, maybe I'm not ruby brained enough. Too much syntax sugar for me
Elixir is great from a syntax perspective. It gets compared to ruby a lot for some reason but they're nothing alike. Elixir syntax is very deliberate about everything.
The BEAM runtime is the real win. Your options are Erlang, Elixir and Gleam to get the BEAM.
Out of the 3, I prefer Elixir and Jose is a brilliant leader for it.
How often do you need the HA features BEAM can give you in a webapp though?
Not losing production data and not getting breached is also something we try to make zero mistakes at.
Those of us who care. Those optimizing for economic returns or out of ideology are not. More carnage will be required for incentives to swing back I think.
"Security is for losers"
> Andy Yen, the CEO of Proton, had a message for an employee in the company’s security Slack channel who urged the company to at least take a few days to pentest newly written code before pushing it live. “That’s bullshit and loser thinking. Of course it’s possible to be fast and secure,” Yen wrote in the channel at the end of August. “Stop thinking always of how to be slower, and focus more on how to be faster.” In a follow-up message, Yen reiterated his point: “only losers think like that.” His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”
Proton CEO Andy Yen: "This will be the last year that we hire engineers." - https://news.ycombinator.com/item?id=50295112 - October 2026
Manifestos, Fear, and 'Loser Thinking': Inside Proton's AI Reckoning - https://news.ycombinator.com/item?id=50289692 - October 2026
Good luck.
Edit: There appears to be some consequences for acting like this and those actions being made public.
Proton's CEO Cancels AI 'Re-Interviews' - https://news.ycombinator.com/item?id=50307183 - October 2026
That's the sort of thing you really don't want to hear from your encrypted mail company.
Vote with your feet or prepare for your eventual breach notification and data loss event. You control who you choose to do business with, and who custodians and processes your data, it is a choice. Make good choices.
The problem is, ultimately, most companies have this mindset. Certainly the ones you've heard of do.
Charlie Munger once said, "Show me the incentives, and I'll show you the outcomes." There are very, very, very few actual incentives to operate a real infosec effort at most companies. Think about it: how many companies have you heard of going out-of-business because of breaches? If you're like most Americans, you've probably received at least one letter from some company you did business with telling you that their systems were compromised and your data may or may not have been stolen as a result. They'll usually close out the letter with an offer of free credit monitoring for some finite length of time. Since that's useless and a lawsuit is a pain in the ass with no guarantee of benefit, the vast majority of people simply sigh and toss the letter in the trash.
There's no real way to pin identity theft or other consequences on a given data breach, so it's hard to make any sort of claim that way.
At the end of the day, the company has saved money by not instituting best InfoSec practices and red-teaming their software on a routine basis. There are no civil or criminal penalties for them not doing so.
So they don't.
Which is why the recent movements for cybersecurity liabilities are so relevant, they are the incentives.
Now in some countries those with data security office role have more to worry about than only losing their job, if they sign off on something that isn't qualified to be out there in first place.
https://germancomplianceinstitute.com/blogs/news/personal-li...
I can imagine people are quickly looking for an alternative now. I wouldn't want to hear this from any company I'm a customer of.
FWIW, Fastmail is awesome.
I've been doing a bit of research, and mailbox.org looks like a winner. It's in Europe and it supports IMAP. Those are pretty much my main requirements, I think.
Do they have a free tier?
Moving faster is good. But only if it means sending more agents (and humans, that part is important) to do the same amount of testing in less time. Dropping critical work is not that.
Oblig. Simpsons:
Homer: “There’s three ways to do things. The right way, the wrong way, and the Max Power way!
Lisa: "Isn’t that also the wrong way?"
Homer: "Yeah, but... faster.”
> Moving faster is good.
Says who? For what purpose? To whose benefit?
I want my encrypted email company to move slow, deliberately, and very carefully.
I'd like most software in general to move slower. Some, I'd like to not move at all. I hate churn for churn's sake, or piling useless features just so you pretend you did something to justify your subscription pricing, both of which were already the norm before AI and are worse now.
To speak in Apple terms, we don't miss more and faster crap novelties. We miss the Snow Leopard releases.
Why is moving faster good? I don’t see much benefit to it, unless I am a shareholder.
Even if you're a shareholder, the long term gains of an investment are much higher than the short term ones.
The trouble is that executives are bonused on much shorter time scales, and they run the company.
> Moving faster is good.
No, not necessarily. Sometimes, yes. For anything important, the last thing I want is a company that's moving too fast.
An email company? The base protocol RFC was written in the 70s. It's as mature as computer technology gets. There is zero value in moving fast here, all the value is in building an absolutely rock solid and secure solution.
The very opposite of move fast and break things.
> That's the sort of thing you really don't want to hear from your encrypted mail company.
Tone deaf award of the decade surely goes to that guy.
If it were some social game app or something, I guess it wouldn't be as tragic, but for a company whose sole reason to exist is to provide security... wow. Goodbye any trust ever again.
Wow this makes me hesitate to renew with them
The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
Either he's been like this the entire time, trying to crack the whip and failing, and thus a bad leader who has repeatedly failed to get results, or he's (as his employees have said) very deep in AI Psychosis and completely abandoned his principles.
It doesn't matter which it is, they're both extremely damning for the company and his leadership.
>The irony here with Yen (and to be clear, fuck him) is that Proton has been _slow as molasses_ since the beginning. It took them literal years to launch a calendar. A calendar!
As if that's a problem? I want them to be my fucking encrypted email. Not a calendar. Not a backup. Not a VPN. And they've already stretched to all kinds of shit.
They are that. Is something fundamental missing from Protonmail?
My point is they announced they’d ship a calendar in 2019 and it took 2 years to ship a full release and 3 to ship an iOS app.
It also sort of sucks, and in 2024 (year 5) they announced they’re rebuilding it. The rebuild partially shipped in 2026.
For a guy suddenly obsessed with speed these timelines are even more absurd.
I would rather have protonmail without all the stuff they are adding there
What is it with weirdo tech executives and calling people losers? First DHH, now this guy.
Weak man's idea of a strong man. Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas, but I digress.
"Any man who must say 'I am the king' is no true king." The measure is not who you can bully and denigrate successfully, but who will follow you to hell.
>Strong leaders protect and support those who work for them, and accept legitimate criticism and being challenged on their ideas
And whose idea of a strong man is this? I don't think many people would follow Putin to hell willingly, or that he accepts any criticism, or accept being challenged on his ideas. But he could be called a "strong man", at least the kind of "strong man" some people aspire to be.
Anybody who is as responsible for popularizing Ruby on Rails & YAML as DHH has, has no standing to call anyone else a loser.
According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people that is heavily selected for a particular characteristic you will end up with people that generally are outliers in some other significant ways too. In this case, caring deeply about email security is weird... unfortunately... and you can expect further weirdness from such people.
You can also read about people talking about this in the super-early computer user communities. When you had to be a bit crazy to even be interested in computers you ended up with people who were outliers in a lot of other ways too. As more people came in to the space it would regress to the mean... this is one of the things that "regression to the mean" can be regressing from.
It's not weirdness, it's just the combination of being an asshole and nobody in his life being able to tell him 'No.'
> According to my friendly neighborhood AI there is no commonly accepted term for this effect, but when you have a community of people
Social contagion [0] (not to be confused for memetics [1]) involves behaviour, emotions, or conditions spreading spontaneously through a group or network. [...] Throughout history all social contagions have had a single thing in common: the people who follow them genuinely believe their group is only doing positive effects, and cannot rationalize negative effects from within their group.
Tech executives are a group (and a network), its not some "regression to the mean" its common human social behaviour within group dynamics. It's not just those two there has been a progressive slide of executives joining the 'tech-bro' subgroup.
[0] https://en.wikipedia.org/wiki/Social_contagion
[1] https://en.wikipedia.org/wiki/Memetics
God am I happy to not have switched to Proton right now!
Haha same! Was about to pull the trigger in the next few weeks
I kick myself for having adopted part of its paid offerings.
I mean if a company doesn't care about protecting its users' data then any conversation about improving security is gonna be inapplicable to them.
I'd also suggest that companies being so blasé is a direct result of lacking regulation. If there was no laws about aviation safety I bet we'd have more airlines with lax safety standards and more accidents.
"His ultimate message for the team was to “stop lovingly handcrafting all our code and use AI more.”"
Mmmmm... I think I may have to score this particular point for the CEO. Lovingly handcrafted code has been many things over the past 30 years, but "secure" is not one of the adjectives I'd apply to it. Anyone who thinks that the lovingly handcrafted code has always just been wonderfully secure needs to check the tint on their glasses.
A proper AI setup probably can produce more secure code more quickly than conventional development processes... however one of the keys will be to understand that yes, that will still be slower than writing code that is not rather carefully analyzed for security and a company like Proton had better have a strong commitment to never, ever skipping that step because it's slowing them down or it got expensive. "Proper" in "proper AI setup" is the most important word.
But that's not to say that I'm endorsing everything else he said. Even if it is true that companies will be able to stop hiring engineers, which at the moment I seriously doubt [1], a company like Proton should expect to be one of the last companies who can do that, not be getting publicly excited about it first. Hey we dumped all our engineers because our managers can just prompt our AIs just as well here in October 2026 would definitely be a "instantly cancel any Proton services I had" signal for me.
[1]: One of the things I don't expect to change is that there are plenty of people I would describe my actual day-to-day work to and they would consider it a living hell if they had to do it. I still quite enjoy it overall. And there are plenty of jobs that other people love and I would consider a living hell. Even in 25 years that's going to be the case. There will be people willing to put up with the process of working with the system to create this stuff... but the people who are willing to be professional about this, do the QA, do the work beyond the first fun prompts, is going to be a minority, and an even smaller minority who take the time to get good at it. There will always be people who are better and people who are worse.
While I agree that handcrafting code has never been any kind of guarantee of secure code, I'd argue that it remains a condition: if all you have is a set of specs (or even worse, specs written by AI and not read carefully by anyone) and no mean to ensure that the implementation actually matches the spec, you essentially have no understanding of what the code is doing, which is security suicide.
Of course, AI reviewers and security tools will catch some security errors, but this should come as an addition to a proper security design, not instead of it.
> "Security is for losers"
Lol, brave words in an age of $6B heists and major CVEs every week.
Eww. I have now lost all interest in their products.
Just learned about his position on security today. Been a Protonmail user for a while now. Initially I switched to detach myself from the US corp monopolies. Its just a matter of time now before I shutdown my Protonmail account. CEOs should understand that company ethics and security are not just black pixels on a white background.
Wow. I'm literally reconsidering having any mails at proton anymore. This is beyond abysmal attitude.
But why stop there? Aim higher!! End of all world’s wars, no more hunger anywhere anymore, free abundant clean electric power for everyone everywhere!
Jokes aside, you can’t get anywhere near zero fault unless to impose strict operating regulations , requirements and procedures and regular checks… just like in aviation.
And even there, boeing manages to fuck it up somehow.
There’s a limit to how much society can spend on aircraft safety, but it’s quite a lot because it’s so important. Cyber Security teams have to work with large constraints because spending an ever increasing amount of money directly, or losing money indirectly (like not allowing product teams to move quickly) is explicitly not a goal of most companies. So while we try to make zero mistakes, most companies are willing to take a non zero amount of extra risk of these things happening in exchange for other benefits.
Aviation is working with the same constraints. Development of new systems is very slow, that's true. But day-to-day operation is extremely fast paced, and still impressively safe. They are able to change an engine of an aircraft within hours, with a very high probability of not making any relevant mistakes in the process.
Sure, but at what cost? How quickly improvements are delivered is one thing, but the materials and chosen process also come at a premium. And while it probably is worth it in aviation due to life and health being highly irreplaceable, the value of for instance my chat history or photos is certainly finite and quite low. Should they be corrupted, leaked, or worst case gone, it would somewhat suck. But at the same time I've effectively concluded that safeguards are worth about 0.4% of my monthly take home after tax, and when the yearly bill comes due I'm each time increasingly questioning whether I actually do value them even that highly.
"Sure, but at what cost?"
At the appropriate and necessary cost.
A less costly new engine design or less costly engine swap procedure that fails is worse than no airplane at all. If it's cheaper and fails, we don't want it. It wasn't cheaper, it was even worse than expensive.
It's not hyperbolic all or nothing. Sure infinite cost is just as useless as infinite failure, but there is no reason to talk about infinite.
At what cost? Some. At some cost.
> At what cost? Some. At some cost.
This doesn't address how the cost of safety related procedures versus the cost of risks mitigated is calculated (which is way harder than saying "some") so that the buyer can actually know what "some" cost is.
It's also not addressing the parent comment's point: the bad outcomes in aviation tend to be very impactful (both in human safety and financially), so the tolerance for mistakes is way lower.
Compared to cybersecurity for a brochureware website, where it takes a far larger stretch to go from "security incident" to "someone's dead". (Also, people in cybersecurity aren't going to accept "it has some cost, and you might buy it if it's not too expensive".)
For the commercial flight industry this is very easy to calculate: zero deaths are acceptable. This is not like car manufacturer recall calculations (famously, referenced in Fight Club: if the lawsuit costs the manufacturer less than the recall, they don't recall). The airline industry is built different thanks to heavy regulation.
You could ask: is it _worth_ making it that safe? But we don't ask that. We just make it that safe.
I think there is a good point to be made that this level of safety (it costs what it costs) is probably not worth it for cybersecurity, in general, even if it's largely considered worth it for commercial aviation.
As software intermediates more and more of the physical world, we're gonna end up at least as regulated as finance, and potentially as much as aviation.
The only real question is how much damage will be caused before that happens.
You can also reduce blast radius from breaches, and make systems more robust, rather than just trying to make zero mistakes. How bad breaches are and how well you can recover is also part of the outcome, just framing it as zero breaches or nothing is not that helpful, even though reducing breaches is important.
Isn't that the same thing as "not making mistakes"? If you can reduce the blast radius and didn't, wasn't that a major mistake in the first place?
No, it's recognizing mistakes will be made and no one is perfect. You will also make mistakes regarding blast radius - sometimes the blast doors don't close or function like you thought they would during an actual explosion.
Defense in depth is the idea. Swiss cheese model of failure.
Yeah, but this is bottlenecked by both your ability to observe and understand the system, which is precisely the thing that's being reduced by careless use of AI tools.
A lot of code is certainly not written careful enough. Plenty examples of plaintext passwords storage in the wild.
This isn't a different thing - it is just one of techniques used, and it still requires humans to understand the systems they are building. Teams that outsource their systems understanding to AI will fail, no matter how they frame it.
Aviation is an example of how there's no way to have zero mistakes, or even zero fatal mistakes.
People always find new ways to make mistakes. The approach in aviation is to investigate, identify root causes, and do what is possible to prevent the same mistake happening again.
There is no guarantee we'll ever reach perfection.
The lesson from Aviation is how to constantly have mistakes without failure.
Sometimes they close down airports, that’s risk management not complete risk avoidance.
It's more about managing mistakes. Pilots and other people in aviation still make a lot of mistakes all the time. But aviation figured out how to deal with that fact. Design procedures in a way, that mistakes self-correct most of the time. Add sanity checks exactly where they matter, and many other strategies.
No coincidence it is the textbook example of a complex system.
https://how.complexsystems.fail/
Maybe it's different in your neck of the woods, but all the government portals I use tend to have access to rather sensitive PII, and I would really prefer their ops teams be up to snuff as a result.
> I'm fine if some of the stuff is having scheduled or unscheduled maintenance.
You might be, and your customers may not! And your competitor could be more than happy to offer a more reliable service.
Then we all get shit service based on interrupted operation without due maintenance and more stress.
A better metaphor would perhaps be machining. You will not find a shop that has no CNC anywhere, but the machinists who can also use manual machines will always have an edge for certain things. Not that they're working manually much in their day to day, and not as all as a failsafe, but you can just tell who has that kind of background.
(And you can also tell whose mind is stuck in the old world, too.)
It's first-principles thinking. You have to have some sense of how you would do it in order to reasonably engage with a machine's idea of how to do it. Otherwise you're just nodding along, like yeah sure bro uhhhh lgtm.
The OP didn't present an absolute. They said "Aviation is a perfect industry to learn from" and I 100% agree with them.
The problem is that people go on Twitter and see the posts from "thought leaders" and "influencers" who say things like "reading the code is not needed". Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
You only need a handful of occasions of creating massive billing errors for your customer's customers before they will slag you to all their CTO friends and pull the plug. Imagine Stripe began double-charging every third transaction once a week, and couldn't fix it after two months of thrashing because no one there has any understanding of how anything works anymore. They are simply done at that point, all the value of their brand lost.
As if engineers, especially those writing software to handle financial transactions (which are usually in regulated businesses), don’t know how to write conformance tests with invariant cases.
Who are the people who are responsibly writing all this test code but declare loudly on Twitter that no one needs to read the production code?
The test code is written and checked by AI without humans needing to be the ones looking at it.
I've seen some terrible AI-written tests. You have to check that code too. You have to check all the code.
soooo wouldn't that be exactly the "deskilling" being discussed here, and isn't that deskilling concerning for money-handling firms?
having seen both (a) mission-critical human-built systems become nightmares for maintainability and extensibility and (b) bleeding-edge ai coding agents confidently and quickly fix a wrong problem in the face of an issue (or fix it in a slightly wrong way), I remain skeptical that this great de-skilling will all be perfectly fine.
of course, if you aren't looking at the code you won't fucking know until someone yells...
I disagree with several premises of the article. There is not going to be human led investigations of outages. People do not need to train on easy outages to be able to handle hard outages. Even if you wanted to train people on easy outages AI makes it easier than ever to setup up real world like issues to replace the need for real world experience. I also disagree in labeling it deskilling when it is a reallocation of skills to what is actually needed.
>concerning for money-handling firms
The real world is robust. It's okay for money handling code to have bugs. It's not the end of the world and AI will be able to have a lower bug rate than humans.
> The real world is robust. It's okay for money handling code to have bugs.
This is true. The world will go on. But your business will lose more money than it gained by not paying attention to the code. It will lose money because it will have to compensate customers or lose customers. That's how the world maintains it's robustness.
> Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
That's just not true.
Name one. Point me to one developer that actually works - themselves, not just in the same company - on a system with serious customers that demand correctness and is spreading that message on Twitter.
https://swamp-club.com
Real customers. Real revenue. Many industries. All written with AI.
Oh my that is convincing. Yes, I've seen lots of this sort of thing. I believe you, a Twitter influencer is behind it. AI building AI systems for AI to build AI systems with.
Thanks!
Unable to understand sarcasm, AI truly allows a certain type of mouth breather to shine.
You're promoting your own company here, which as far as I can tell from the vibecoded website is Yet Another Agentic Platform of some sort. And prominently features a leaderboard with high scores for some reason.
It hardly meets the bar of a mission critical product like an ERP or payment processor layer where introducing random billing errors in customer data could be fatal to a B2B company's viability.
Ok. Keep moving the goal posts.
I guess we are not serious engineers making a serious product for serious business. I'll make sure to let our customers know.
If you scroll up to the message you originally replied to, OP's goal posts from the start were companies like Stripe (which was specifically named).
Most businesses aren't in that category, that doesn't mean they aren't able to provide useful services and make money. But your company does not seem like it meets that bar.
Expanding the group to "any money making SaaS" would be moving the goalposts for the point OP was clearly making from the beginning.
Sorry for being a bit dismissive in my previous comment but it was fairly jarring clicking your link expecting a more responsive counter example to their argument, and then left a further bad taste after realizing from your bio that it was self-promotion.
And just to pile on, payments is an incredibly competitive space, and operational excellence is non negotiable for someone like Stripe, as they are a higher priced provider.
Your site does not give me “serious” vibes. Swamp club? More like slop club.
His top customer is skunk-ape.
I think the parent take is valid if you disqualify people managers/leaders rather than ICs. I think that's also valid to disqualify those takes because they aren't in the trenches every day. They may have points strategically for what's best for the company, but not what's best for the workforce or the industry.
I don't see what the giant social cost of AI hucksters is supposed to be... who cares if some random influencers claim you can vibe code your entire existence? the thing that bothers you is that the grifters are basically right, just for the wrong reasons (and a bit too early). Comments like this imply that there is some cavalcade of situations like this that have caused actual harm to actual people, but it is actually just an emotional response to people who are personally distasteful, and borne out of a desire to make sure that the "assholes aren't proven right". stuff like "Imagine Stripe began double-charging every third transaction once a week" is just so silly... what about AI makes this more likely in any way whatsoever? If anything, AI would have such a subtle scheme for robbing us blind that it would defy characterization into pithy quotation marks. this feels like a complaint about humanity dressed up as a complaint about AI. for all we know, AI could actually improve the situation-- it's not like humanity has set some unimpeachable precedent.
AI can absolutely improve the situation. It has helped find and fix bugs I have despaired of ever fixing. But it cannot learn, and it cannot understand things for you. And people outsourcing all their understanding to it absolutely could lead to existential disaster for a system at the heart of the platform. Coding agents write too much code and ignore existing patterns. This has been studied quite seriously and they generally write 2.5x the amount of code a human does when refactoring or enhancing code in a real project. When a very large system slips past a critical point of decoherence you really cannot put any certain boundaries on what kind of problems become possible or how long it may take to fix them.
Yes, there are responsible engineers using AI in innovate ways and making things both more productive and better. But there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place.
> it cannot learn, and it cannot understand things for you
the former is demonstrably untrue (and wildly so), and the latter is just an egocentric/anthropocentric "the genie in a bottle only answered my wishes, it didn't tell me what to wish for". buried in that statement is the presupposition that humans being able to understand that something is actually Important; short of religious/faith-based reasons I don't see the justification for the primacy of human understanding when it comes to the determination of Truth. if anything, being "understandable" to humans strikes me as an anti-goal. are we really to believe that human comprehension is the limiting factor keeping society from degrading into "decoherence"? how well is that going for us? the reason humanity places such a premium on identifying "The Truth" is precisely because of how illusory it is for our fallible minds. LLMs struggle with many things, but coherence is least among them. that is why "slop" works so well-- if it wasn't at least mildly coherent, it would be easy to discriminate.
> there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place and there are even more lazy humans who are happy to ascribe this to a shortcoming of the technology and not of humanity itself.
insofar as we are descending further into post-truth, it isn't because of AI slop blurring the lines between fiction and reality; it is because we are all collectively realizing that Truth isn't all it's cracked up to be. LLMs are forcing us to take a long hard look into the mirror and we don't necessarily like what we see.
Please point me to the paper about the coding model that can continuously learn.
For practical purposes an agent is learning if you put information in its context. So you can create a comprehensive architectural analysis and put it into context and suddenly the agent is familiar with the project. You can also delete information from context, which is a rather hard and imprecise process with humans.
The amount of trust AI has set on fire is incredible.
Business and people that I once though competent I trust no longer.
I'm watching leaders I thought had vision throw the future away for the shortest-term gains.
What have we become?
> Maybe not all of the projects need the same properties.
Definitely not all projects need the same level of quality.
But per the thread, if you allow the skills of all the engineers in your company to rot, what happens when there is a system that needs the higher level of quality?
Just like pilots, SREs will have to spend a minimum number of annual hours in the simulator to keep their skills sharp.
SREs will be de-materialized and converted to a Docker image that will be run on Kubernetes, in the prod env.
Exactly as foretold: https://www.youtube.com/watch?v=0otdTJa9IUc.
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Oh yeah, and you will find very quickly that most people are a bunch of flaming hypocrites when it comes to flying. Lots of wannabe aerospace engineers in tech. Half the people I have met who put aviation on a pedestal hate flying but love to trumpet the statistics of how safe commercial aviation is. Try asking them to get into a Cessna and all of a sudden the excuses come out. It's oh noes I have a fear of heights, oh noes general aviation is less safe and only slightly safer than a motorcycle, oh noes the FAA is corrupt and allowed the helicopter crash in DC to happen, oh noes I have a family etc.
Aviation safety discussions on hacker news have barely more more depth than biotech and healthcare discussions here. It's extremely ironic because a tremendous number of pilots are engineers, and especially software engineers. It's not exactly a cheap hobby. But you don't see massive numbers of English majors come crawling out of the woodwork to join a discussion on the safety of C++ vs Rust, but tech bros flock to worshipping aviation as a platonic safety idol like moths to a flame.
Not aware of the concerns you have, but they seem just as valid about other similar concerns across industries.
Any books you'd recommend or journalists to follow that cover this beat?
Half of aviation safety (especially for commercial aviation) is done on the ground and has nothing to do with aerospace engineering. Stuff like shift times for pilots and ATC, mandated rest periods, proper ground equipment in zero visibility conditions etc. Many developing countries cut corners on maintenance and spare parts. Mandated use of safety technologies (the DC helicopter incident was entirely preventable). Airports that are designed with safety in mind. If you have a dual engine failure immediately after take-off on an airliner in the middle of a built-up city you are generally fucked. Midwestern airports are the best if you have to do a forced approach. Transport category aircraft (basically anything bigger than a 12 seater plane) don't really have many options, they can't easily land on the runway like smaller planes. That's why the Hudson was a miracle. Your airliners stay safe through constant maintenance and inspections. Dual engine failures are very rare (unless through deliberate pilot action like the recent case in India).
On the engineering side you can look up ETOPS and the various ICAO and FAA and EASA standards for airliner class aerospace engineering https://en.wikipedia.org/wiki/ETOPS
Trust me, y'all think airplanes are "safe" because of some sort of magical protocol. No single protocol actually works. It's mostly lots of layers that are stacked on top of each. It's why in aviation, catastrophic failure modeling uses something called the Swiss Cheese model; you need every single hole (error or failure) to line up to generally trigger a complete breakdown in safety. It may sound counter intuitive, but by aviation safety philosophy, a canvas and wooden frame plane operated under commercial standards and kept to a strict inspection schedule is "safer" than e.g. a privately operated 737 that doesn't follow manufacturer maintenance standards.
NASA has plenty of guidelines for writing "safe" aviation software. I doubt any software engineer will be willing to entertain the level of effort and certification needed to comply with those for e.g. their Django side project.
Err...have you actually asked someone to get into a Cessna?
Also, "fear of heights" is a totally valid excuse. Fear is a gut reaction and may have nothing to do with one's justified true beliefs. Eg., I know spiders on probability won't harm me but there simply won't be any amicable co-existence between me and any arachnid.
> Err...have you actually asked someone to get into a Cessna?
Yes? In practice once you convince them to get into the plane the only real issue in the air for those new to flying is air sickness.
Also fun fact a good number of pilots claim to have a fear of heights (never met one irl except online in aviation forums).
If you are in aviation/aerospace, don't be afraid to share the joy of flying. There's nothing quite like cruising at 90-100 knots @ 3000-4000ft AGL on a nice clear day.
I don't understand the perceived hypocrisy. The chance of death in a 1hr Cessna ride is something like 100x a domestic commercial jet flight, so it's reasonable to have different views of them? Maybe I'm missing your point.
Okay, just because aviation is now also developing in the wrong direction (maybe), all the things they did right in the past are suddenly pointless? I suggest to learn from the things they did right, not the things they did wrong.
Try again. Commercial aviation, with people on board going between point A and point B.
That 2nd pilot makes up for lot's of impossible branches in the fault tree.
I believe this is when the term "okay boomer" is warranted.
This was just the default where I worked. We called the exercise Wheel of Misfortune: facilitator sets up an outage / bug in a non-prod environment (inspired by real events or not), spins a wheel, picks a victim, and the problem is solved collectively, manually. That's the standard format, but it could be much more freeform than that: the point is to educate each other regarding "incident management."
I will say, I'd always been skeptical that AI could be better at incident management than an experienced engineer with deep domain / institutional knowledge, but I was recently proven wrong: Claude root-caused and helped resolve a Postgres lock queue pileup that I believe would have taken me much longer to fix.
Re: aviation being a perfect industry to learn from, I think https://m.youtube.com/watch?v=5ESJH1NLMLs might be a good watch. I was introduced to this at a Wheel of Misfortune around 5 years ago, so it's interesting watching it today when AI is encroaching upon everything.
Aviation is a perfect industry to learn from if you are willing to hold IT to the same standard of regulatory scrutiny. You can bet your entire net worth that anyone running anything in the industry would fight tooth and nail to prevent that sort of regime from coming into force.
A (very) key difference is that this behaviour is mandated by the FAA, and consistent across all providers. There isn't a (legal) option of just not-doing-it because it's "cheaper".
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
You're right, there is no mandate for learning and doing better in SW engineering. Many companies just don't care, also kind of fine. This doesn't mean that everyone doesn't care and nobody wants to improve.
> Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes.
Aviation is used to paying through the nose for stuff. For developers, employers want to pay as little money as possible.
But, at this point in history, they are paying through the nose. As they say.
Cut dev salaries by a factor of 3, and then it would be normal.
The thing is, without the FAA mandating something like this for SRE’s, no one will do it. In the moment, getting the broken thing back up and running is seen as far more important than the learning opportunity.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
Isn't this the whole point of something like Chaos Monkey? Has the industry at large just lost the plot here?
lol wait until self driving becomes popular.
Those will be fun times.
I wish it was already the case. Tesla FSD is already better than almost every human driver out there.
It's not a great example.
The plane lands in the same amount of time whether an autopilot lands it or a human, so there's no incidental cost.
If you let a human SRE handle a problem completely, not only will it likely take longer than an AI, but since that SRE is now deskilled, it may take FAR longer. And that's a cost. Who's going to pay it?
Is that true? Wouldn't a pilot mistake in theory cause a plane to take longer?
no offense, but I think aviation is a horrible example. if we treated every problem like we treated aviation, we would still be living in the 1950s. honestly, probably the 1850s. hell, we sent a guy to the moon LONG before it was actually safe to do so. (is it safe even now?). do we really think LLMs would be more blasé about human life than humanity itself up until... iono, since the Baby Boomer generation? the "issue", such as it is, with AI is not "we need to make so many guardrails in order for it to be useful", it's that unlike aviation, where the ground truth is "did the plane crash or not?", AI is an open-ended experiment in what would happen where we don't have any feedback from anything other than... how humans feel about it. humans are the antithesis of paperclip maximizers.
I don't see any reason whatsoever why "AI regulations" would become antifragile from their having been "written in blood" in the way that aviation regulations have been; if anything, our inevitable present and future AI crises will each be exponentially more devastating than the last. Modern aviation is a triumph because of how fallible humans are-- that's why we delegate control of aircraft mostly to machines. The shape of AI disruption isn't "the computers aren't following the rules", it's that even by our own rules of rationality our world is terribly underspecified-- we have computer security that isn't secure, subjective rules of ethics/fairness/judgment that defy straightforward explication, "knowledge work" that is actually neither "knowledge" nor "work"... the whole situation is much closer to the human domestication of the dog (letting a potential threat into one's home) than any traditional story of humans coming to terms with their own technological innovations. human behavior is FAR easier to predict than un-aligned AI behavior-- that is why it is such a singular moment in human history.
I work at a startup and often deal with every part of the stack--programming, deployment, db, cloud environment, networking blah blah. AI agents have just gotten superhuman at computer use. I will never be able to use AWS CLI as well as claude does. I can spend time poking around in AWS, manually reading logs, looking at routing tables, cross checking with data in db, whatever, or I can give claude readonly access to all of that stuff and it will solve the issue literally 10x faster than I could, write a detailed report I can pass on to whoever, etc. Is it sometimes BSing? yes, but it just keeps getting better.
All that said, skill atrophy is real. Thinking about getting back on the interview circuit is p scary because of it. I dunno wtf is gonna happen.
I used to scour through server logs, build logs, test reports etc to figure out why tests were failing in CI. That was exhausting work and could take days of investigation. Now I get the exact root cause in a couple of minutes. Just had to give LLMs a token to call Jenkins API and it knows how to do the rest. What an unbelievable experience. Never ever thought computers would be able to do this in my lifetime.
Had this eye-opening moment a few days ago where I enabled our company-wide MCP server and asked for a problem we noticed. The agent looked through slack messages, logs from our devops portal, open telemetry traces and searched code snippets from other services through our RAG code search tool and created nice tables with timings, explanations and suggestion to fix it.
This is an amazing use of LLMs, contrary to offloading your thinking to it during implementation (which does not mean not using to write code but to actually understand the systems you are creating with it, and all their edge cases, inner workings etc.)
But there's a new skill blossoming there, right? That's what I'm noticing. The atrophying skill is one you don't need anymore. The new one - prompting, routing the output, spotting the hallucinations, managing singal/noise etc. - is now where you can add tremendous value.
To say nothing of being bold enough to run systems like that in the first place. I know some of engineers that are still folding their arms and refusing to engage with AI!
I don't know how to spot hallucinations, manage signal/noise, without having a pretty good understanding of the problem space. It's a tough balancing act.
Every new model and you might as well chuck 90% of that knowledge in the bin.
Last Claude major version they released a blog post basically telling you to chuck away your Claude.md file and start again.
It was only 18 months ago when people were telling you to use ultrathink:
https://news.ycombinator.com/item?id=43739997
And then that went away about a year ago, and now it’s sort of back again but not quite as it was.
I really don’t want to see code generated by Claude without some constraint like rules in a Claude.md file. Even Fable on high is lazy enough to forget a ton of things that a senior engineer would not. For example tracing and handling all unhappy paths, not repeating code, not writing terrible doc strings, etc.
We are using quite a big list of rules and the output is significantly more aligned with what we interpret as “good” code.
>spotting the hallucinations
SRE here, spotting the hallucinations requires you to have the knowledge to know when it's wrong. Best way to gain knowledge on systems is actually working on them. If AI do all the work, then you don't get the knowledge.
Exactly. How hard is it to understand for people?
If you don't ride your bicycle for years and years because you can just write "drive yourself"
Then your skills are atrophying. Typing isn't a skill
I don't know that that analogy fits for this comment. The argument is that the old skill is genuinely no longer needed in favour of the new skill, but the new skill requires experience from the old skill. I don't think being a good driver needs you to have already been a good cyclist.
> I don't think being a good driver needs you to have already been a good cyclist.
It's an established fact by now that if you know coding then you can get better output from Sonnet than vibe coders can from Opus
I think it basically stems from the idea of “conceptual learning”. Which has been a pedagogical ideology for quite some time now. And is also convenient if you’re a bit addicted to your phone, a bit lazy due to AI, and want to still feel good about “learning” things.
We’ve known that one important component for learning (not the only thing that matters, but an important one) is the memorization of facts.
There seems to be a big urge to pretend that’s not true. E.g. you can learn arithmetic without memorizing your multiplication tables, literature without engaging with the canon, and now apparently coding without even having coded.
The idea is if we start with a top-down conceptual approach we will eventually vibe our way into knowledge acquisition. Which does sort of work.
However, sorry to be a curmudgeon, but the people who just “do the work” and actually spend time on task to learn a bunch of stuff … are actually more knowledgeable on average.
> you can learn arithmetic without memorizing your multiplication tables
How? If you don't learn multiplication tables you're multiplying on the go. And that's inefficient
Also, this isn't the right analogy. I get where you're coming from.
We move on to the higher form. Like we don't need to code in binary, or assembly language We got C, Python. We then got compilers.
But LLMs are just english. There is no skill here in writing the prompt. "Give me a crud app that does xyz" is hardly a skill
Tomorrow if AI data centers fail then what do we do of the typing skills?
When we shifted from Assembly to Compiled languages or to Python. We still had to learn something else
Here, using AI, its just English.
How do technical managers solve this problem? I've never been a real manager.
I guess humans don't hallucinate that much
Humans hallucinate much more, but they're not that good at making the hallucinations convincing.
No they don't. Any human who hallucinated as much as an LLM does would be committed.
It's interesting how machines now mirror our relationship to sanity.
Just under the surface of human consciousness, we are constantly hallucinating and predicting what we may occur next. We are good at discarding what doesn't align with our working model of the world before reaching awareness and taking action. It's our attempt to conceptualize reality. Considering how un-grounded from reality humans are becoming, the fact we have to balance our nervous systems, the mistakes we do make day-to-day, and how minor physiological changes (from poison to injury to drugs to random chance) can disrupt the process, it doesn't take much for humans to make a major mistake or enter a psychotic state.
It wouldn't surprise me if a systematic approach to solve these problems with LLMs could be applied to human/hybrid teams.
Not even that Proton CEO claimed that. This is another one of those "no u" knee-jerk reactions littered all over this site, and I wonder when we'll first see stuff like "but humans also sometimes fail to load because of network issues", because there is no real distance between "humans hallucinate even more" and something like that.
> "but humans also sometimes fail to load because of network issues"
They do. It's called "stuck in traffic."
Humans don't literally hallucinate much, but we do lie, or are wrong.
Humans are deterministic and have personality traits. The first couple of times getting bs’d and that person is now a known quality.
Agents fail randomly. One day a stellar dev and next a bs artist.
So this idea that LLM hallucination is not unlike human failings is without merit.
> Humans are deterministic
what
It's 1930 again
Good ones force themselves to do technical work hands on regularly, but will generally reach a level where they can't actually do that anymore unless they are really good at building management systems and delegating. For me it has happened in cycles over the last 25 years; after 4-6 years in management I refresh skills and go back to an IC role because I can't stand it anymore. I've done that cycle three times but I won't do it quite the same way again because its probably impossible to do without a pay cut now, and because IC don't write code anymore anyway.
This is both like forcing all IC to develop management skills but its also like the shift to cloud infrastructure or 4GL languages - we are never going back to the old ways and will be building everything at a different level of abstraction with a different span of control.
The good ones still make time to knock out some lifelong backlog resident tickets to sharpen their axe on.
The bad ones get promoted for spending the time instead playing politics.
> The atrophying skill is one you don't need anymore.
of course. who needs to walk when we have cars
This is always a good example
People cannot deny that cars have made our physical bodies atrophy. Not all of us, maybe, but many people are way worse off physically than they would have been without cars
There's an analogy in there but I can't think of it. Maybe I'll ask Claude if it can explain it
A company comprised entirely of management. I'm sure only great things will come of this.
No, everything you listed is low effort and minimal friction. It's exactly the luddite's greatest fear: to be delegated to a button pusher, to ensure that blinking green light continue to blink green. Anyone that can read and write can now add "tremendous value," so what do you think makes the value "tremendous"?
> I dunno wtf is gonna happen.
me either but i do know it will happen sooner than i expect
Velocity is sought because we lost traction. If you can effect your designs in few actions, speed has nothing on intention.
This. It seems most developers have never even had the experience of iterating on a system they understand, making predictions, seeing the result, and having a clear sense of what they are trying to accomplish. There are so many tools and systems, wrangling them becomes an end in itself.
The way I think about this -- AI makes all of us managers. We need to figure out when we are convinced by what is brought in front of us, when to ask to do more checks/validation, when to fully reject, when to accept. To do this properly we need to have to have enough background knowledge to evaluate the what is AI brings, otherwise otherwise a person in the loop is simply not needed if they only type a prompt and always click yes.
I think people indeed need to constantly practice our low level skills, otherwise they will atrophy.
Managers or rather inventors and solution providers? We can do a lot more than just manage now.
You may never be as fast, but you should absolutely feel at least hypothetically capable of using the CLI at least as well. Or just.. Reading docs and understanding patterns or interfaces was never a superhuman skill and it still isn't. You are a lot more capable than you probably think!
And I am not just trying to platitude you here. Without that basic confidence everything ends up just eventually being harder with the AI, even if maybe the feedback and loop of it all feels faster.
> skill atrophy is real
It is also both a choice and avoidable.
Atrophy happens when you trade understanding for productivity. Productivity benefits your employer and your KPIs, but not your understanding and your opportunities going forward.
Understanding used to be an unavoidable of the exploration process, and writing documentation used to be the thing that never got done. Now, writing documentation is automatic, but reading it still isn't.
I've been managing a team of interns at my last job, and as they were all equipped with coding agents, they'd deliver complete solutions to problems once or several times per day that would normally take days, or weeks if you factor in necessary learning.
As review of their understanding, and for the sake of my own cognitive debt, I would first ask them the question "How does it work?" and they usually couldn't answer. After a handful of tasks, they started preparing answers. This forth-and-back is still more shallow than getting the deep knowledge chiseled through hours of trial and error, debugging and wrong directions. Spending two hours to find a problem to realize you forgot to save or upload a file, or had an x instead of a y. I look back with nostalgia, and I wonder if there weren't always better ways, and we just get to pick them now. I can literally convert any task, solved or not, into a quiz that brings me towards understanding, in rainbow-colored pig latin.
Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
I’m on the camp of using agents as much as you can, but make sure to understand what they did. However, it will still atrophy your skills, no matter what. It’s just different times now, as on the job you’re expected to deliver faster. And obviously that means less time spending on your craft, honing your skills and etc.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
Let’s say you have an incident, and you need a resolution. Do you spend 5 minutes tinking, then solving the problem or send out the AI for the hunt and spend 30 minutes grappling with hallucinations and non-sensical output?
The 5 minutes I’ve seen is usually by someone that will send the complete output to someone else to decipher, while saying they have figured it out. Then they’ll turn into a meat proxy between the team and the agent.
In real life, for systems where incidents that require such rapid responses, you usually have fail-safes an killswitches. When you were designing your system, did you put in place alerts and monitoring to mitigate risks?
I’m not sure if you’re running the latest frontier models. Every single engineer (and myself included!) has mentioned numerous times how nothing beats AI in debugging. My other guess would be that you have non-standard set up that Opus 5.5 doesn’t know off the bat how to debug.
I wouldn’t say the same a year ago, but usually we get preliminary reports during incidents before someone can even auth to debug. Almost all the time, nowadays, the report is correct. We obviously verify it.
> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
It is far more often the case that you will spend more time getting the answer out of the LLM than you would've gotten yourself. They suck ass at troubleshooting systems, very frequently jumping to completely implausible scenarios that no human would think of as the first thing to try, and almost never asking questions to clarify the situation the way a human does.
This has not at all been the case for me. LLMs (even the more-capable local models) are exceedingly good at troubleshooting when given proper context (clues, hints, a decent system prompt/skill that provides even the barest hint of how things work)
What you're describing sounds more like what would happen if you gave a chatbot without access to the repo as little context as possible and asked it to come up with a solution, blind.
> They suck ass at troubleshooting systems
Given the right tools and instructions I find them to be consistently superhuman. It's unreal.
I have an automated system responding to issues occurring on oceanconnect.ca, and every single diagnosis has been correct for months. Of course I still verify, I tune solutions, I review everything. But it's right. It's not a trivial system. It has a fairly large frontend, a large-scale model processing pipeline on the backend, a fairly complex database, and a queue for getting new data into the system. Claude does a stellar job of navigating and understanding problems, even when they're not 'real' or as they initially appear to be; things like quirks in old versions of Safari, or the model pipeline fails due to data integrity issues from the providers.
I've rarely encountered cases where I need to steer a frontier LLM much when troubleshooting. They're exceptional.
Sorry, this might be a thing in your org, and I have no idea hoe I can convince you, but it’s just not true in our environments. Maybe you might have a very non-standard set up, but frontiers do amazing job.
I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain. You can even have AI explain the code to you, so it's easier to understand.
I think a lot of this skill atrophy talk is over-blown. It definitely happens, I experienced it some, but as long as you slow down a bit, and clean up and understand your code, your skills won't atrophy. Actually, AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
I think perhaps my use of the word skill atrophy here is wrong.
Skill atrophy is when you're good at something, and because you don't practice it regularly, it gradually wanes over years, like when my former CEO said "I used to be good at programming" because he made a tiny chess computer and sold millions of copies, and now he... just doesn't code, and hasn't for decades.
Skill atrophy would be if I opened my text editor and I wasn't able to code after two years. Being a polyglot, there's always some language that feels rusty. And now they all do. But it hasn't been long enough to feel the atrophy.
What I'm describing is laziness to understand a problem domain.
> I couldn't imagine submitting a PR for code that I didn't understand and couldn't explain.
I could. I do it constantly for projects where that premise is acceptable. And I've worked on projects where many people merge to main once an established set of steps has completed. "How does it work?" and "Show me the thing working." are good gates. Understanding (or even reading) the code is not a mandated prerequisite.
As an Open Source contributor? No way. Exactly because I'm contributing to projects where people don't share basic premises. Some people code by hand, others use coding agents. If I were compiling stuff and everyone else was hand-writing their assembly, I'd read and tidy up my generated assembly, too.
> AI provides a great opportunity to learn at an even faster rate by using AI to help explain complex topics.
Totally. Better solutions come through those explainer sessions.
I’m doing interviews as a senior sre and frankly it’s extremely weird.
I’m being asked very open ended scenario-based questions, I’m answering honestly, and it seems to be going well (i’m moving through the interview funnel).
I’m not getting asked deeply technical questions. No weird obscure trivia. No stack-specific things. No linux internals. No weird networking trap questions. Coding part is essentially gone.
My interviewers so far have mostly been other senior SREs.
It looks like a market where a senior sre can recognise another senior sre. Some kind of closed circle.
So weird, so so weird.
It's a vibe interview :)
Tell me more... I'm also in the same boat, and part of me feels more prepared to do high-level open-ended exploration as opposed to code-based testing.
What kind of questions get asked?
>> I will never be able to use AWS CLI as well as claude does.
Are you sure that it's superhuman; and not super -you- man?
Ah yes, it's just everyone else that's stupid; not you. You're still more capable than Claude right?
And just in case you haven't used the latest AI models and are being genuine, no it really is superhuman. Even if you are an AWS CLI expert, nobody in the world is going to be as fast as AI. And of course nobody is an expert in everything. You're always going to find some random domain where AI is read all of the research papers on it and you haven't.
AWS CLI‘s API surface is huge, it’s all of AWS. You can abstract over that with your own tools. Agents are great at these little greenfield scripts, both building and using them.
Skills that you need come and go. I owned a commodore 64 once and knew how to use it. All gone. I fired up an emulator a few years ago and then had a big "now what moment" where I realized I had no clue what to do with it. I had a vague memory of inserting tapes and issuing commands to load stuff from it. Function keys were important. And there was a manual I once had that explained it all. But no muscle memory. Nothing. I'm sure I could have figured it out eventually. Same with DOS. It's decades ago since I did anything on a DOS command line. All gone.
This doesn't matter. That stuff belongs in a museum. I've used AWS pretty extensively a few years ago. I switched to gcloud and other providers later. At some point in the last year, I started delegating all of the pesky work of interfacing with that stuff to AI. It's amazing. I've always referred to AWS documentation as a maze of misdirection. Just endless amounts of documentation mixed with marketing pages that are completely useless. All links lead back to those. So you are constantly fighting their marketing people to get at the bit you actually want which is a mess of fragmented reference documentation with parameters, endpoints, etc. Staying on top of that and keeping it in your head is a skill that will atrophy that I never cared about.
AI is great for figuring out micro detail like that. It's also great for coming up with a bigger plan that involves stringing together lots of tools, commands, scripts, and what not to execute a high level goal of "I want to run this thing over there". And with a bit of work you can actually do this the right way as well. It's fine. I never really liked devops to begin with. I can do it and I appreciate it done properly and can tell the difference. But I've always hated how it sucked up my time when what I really wanted to do was move our product forward. Instead of which I ended up dealing with a lot of arcane AWS or whatever crap that just needed dealing with. And there's always more of it. Just endless amount of micro detail. AI is great for getting on top of that.
SRE was always about understanding; understanding data flow, understanding bottlenecks, understanding logs, understanding application architecture, understanding end user functionality, understanding break points, SLIs, SLOs. Understanding how to mitigate, gracefully deteriorate, restart, restore, recreate, scale, etc.
There is no understanding with AI-generated applications, full stop.
idk, I develop a lot of connected systems: Pis, Macs, lights, Google speakers, custom ESP32 boards. Understanding the capabilities and communication methods to pool them together is still very much part of what I do. If you're thinking one prompt -> production via AI, sure. But I don't think we are anywhere near that because there are so many moving pieces which still require human sign off: legal, design, QA, etc. To me, it just shift ups the level of understanding rather than replaces it.
> To me, it just shift ups the level of understanding rather than replaces it.
This just sounds like convoluted way of saying you understanding much less than you would otherwise.
Seems like an odd criticism given that programming has always been about abstraction. But sure, I don’t know tree-level details (functions) but I also wasn’t doing leaf-level (bits) understanding because that became obsolete. Now I’m on forest level and expanding to eco systems.
You can always know more but at a certain point, after you’ve built enough validation and safe guards, you find it’s not the best use of your time and the things you can make beyond your specialization is quite remarkable.
Also, I think the toothpaste isn’t going back in the tube with this
AI is not an abstraction.
Thank you!
The internet is full of people doing the “AI = compiler” comparison and it’s just wrong.
Maybe on a really small scale they can resemble a compiler because the number of possible outcomes is so small that it’s nearly deterministic.
But at any larger scale AI will not be deterministic anymore and at that point it will be relevant what the resulting code looks like. Because it can and will be slightly or very much off from your instruction.
If all inputs match a spec then why does it matter that the LLM is non-deterministic? I agree we might not be there yet, but the timelines for progress is shrinking and we are more likely to get there in a matter of months than not.
> You can always know more
But the direction is reversed.
I figure out how to do something, wrap it in a bunch of functions, and then kinda just use it. If it just keeps doing what it should, I may kinda forget how it works. But I don't forget that I made it, and that I understood it, that will never not have happened. So in a way, I am able to "forget" some things because I still know I knew them. And I trust that programmers who write programs that I dream about writing also know what they're doing.
With LLM, you accumulate stuff with from knows where, ultimately fished from a vast ocean named question mark, by whatever turns of phrase you and involved agents happened to use, and what the temperature was at that moment. Which is of course where all the harnessing and looping and running code on the output comes in, but still, ultimately you have no idea. Which is fine for prototypes, but long term? Good luck.
I guess I'm just lost on things not being solved to a degree. Maybe its that I'm an iOS engineer where things are so locked down as far as goals and efficiencies that AI doing the work is verifiable by running the program, setting up some pre and post verification of code, and writing tests that look for certain edge cases. To be frank, it just seems like a trust thing. I trust it to the degree I am willing to to verify it.
“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger Dijkstra
I've thought a lot about this quote, and the best abstractions I use are absolutely this. I love array indexing as an abstraction, but in the back of my head I have pointer + index * size. I like having the kernel do memory mapping, but I know there's a buddy allocator and TLB cache there.
> There is no understanding with AI-generated applications, full stop.
Full slop, you mean.
For the last 10 years, I don't think I met a lot of senior architects or dev who could understand all that.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field. It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
Really? I think that in most orgs up to medium size, you expect senior and especially staff engineers to have that scope
They wouldn't be able to draw it all onto a whiteboard from memory. But give them a half day and they will have mapped out the data flow
The bigger the org, the more specialized they want you to be.
I've seen these people rapidly disappear in the last 5 years at a big tech co. Old UNIX heads that had experience back to the 80s being made redundant. Who is replacing them? Mostly offshore 20 something year olds on a fraction of the salary a tenured person had, being told to just follow runbooks, pasting the runbooks into Claude or ChatGPT. These people can't explain anything, didn't really cut their teeth on doing things out of curiosity, and if asked a question, will just be playing copy-paste-back-and-forth with a LLM.
Agree up to certain levels but your examples are too specifc.
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
There's nothing unique about junior SREs. They're replaceable with almost any "vanilla Dev". You don't really build a unique experience or expertise until Senior+.
That's where my analogy breaks as all of them do.
My point is that @pluc comments suggests too specific and I wanted that his examples break down in different levels of expertise.
Nothing I mentioned deals with disposability of engineers.
The difference between a junior SRE and a dev is that during an outage the dev will take responsibility for their own code only[1]
While the SRE will follow the problem to the next layer or component and try understand what is going wrong there even though they never heard of the system 10 minutes ago.
[1] "I know I'm on call for this team but the guy who sits next to me wrote that code" is something I've heard way too many times.
Fair.
The same understanding most of the devs lack because they aren't.... curious. And thus, why they aren't SREs.
Have the AI explain it to you. You probably won't understand it as well as if you created it yourself, but there is (the potential for) lots of understanding.
Or it could be just better instrumentation...
What would you think of easier-to-debug infrastructure building blocks? I'm building an easier-to-debug web server and reverse proxy, and I want to know how this would compare to trying to use AI SRE agent to reduce debugging friction. Or maybe both complement each other?
There's a lot of chatter about how AI is advancing mathematics at the expense of human understanding. This is going to be true for software industry also at some point. The main gripe of mathematicians is how AI is discouraging young people from persevering. Right now job market is rough for junior SWE and unlike mathematics professors, nobody gives a damn. A few years down the line we'll see a real shortage of software engineers who "understands" and it will come back to bite the industry.
SRE: https://wikipedia.org/wiki/Site_Reliability_Engineering
Thank you! I wish people would define acronyms more.
Agreed. Neither the subject article nor the one it linked to defined this acronym, leaving the reader to guess.
initialisms*
TLAs and eTLAs.
Join me in my War On Acronyms! (WOA) They are opaque, a form of in-group signaling, and the number of collisions is ridiculous. They hinder the flow of information much more than they facilitate it.
Thank you. I have been a software dev a long time. More than 30 years. I did not know what this acronym stood for. I read the article and the article it linked to. Neither spent half a sentence explaining what the acronym means.
It actually means sysadmin.
Terraform deskilled us too. No one is saying, "SREs should configure their infra by hand sometimes to not lose that skill".
If AI can handle it, great! We have a new tool -- let's make the tool better.
Let's figure out how to use AI to make us detect and resolve issues faster, and even better, prevent issues. I've always said we can't consistently hit 5 9's without automatic detection and remediation. We are there now.
Let's get everyone to 5 9s with AI!
> SREs should configure their infra by hand sometimes to not lose that skill
Actually... yeah, I'd totally say that. Some SREs in my company never wrote a single line of HCL that wasn't a call to a in-house, pre-approved module. They think they know Terraform - they don't, and they don't know what's under these modules either.
Terraform, as with any language, is mostly useless for humans now. It is for the agents full stop. You are correct though, we need to know the underlying architecture and most importantly why it is the way it is.
That is not true of Terraform, nor any other language. If you aren't at minimum reading and understanding the code, you are bad at your job, full stop. More realistically, you should still be writing a lot of code, because most of the time you will waste more time trying to work with LLMs than doing it yourself.
I don't even have experience with Terraform at all...
I'm using something different myself: a Docker Compose + reverse proxy config + some other support files, pushed to a private GitHub repo, which then triggers a GitHub Actions workflow that leads to the repository being pulled on the VPS and the infrastructure to be "rebuilt" (docker compose up, reverse proxy graceful restart).
It's unclear if you even know what terraform is, because the system you described has little to do with it.
Terraform would be the code that brought that VM into existence.
Along the same lines, Docker "deskilled" a lot of what used to be called "sysadmins."
AWS condensed 5 SMEs from the NOC into 1 expensive cPanel wannabe and so it goes, and will continue to go.
That ship (of Theseus) has sailed. Computing is going to become intensely personal again.
Opus 5.5 is quite effective at generating Terraform for a "typical" app that needs a load balancer, compute, and a database or two. That's the exact sort of cookie-cutter, rote work that AI is perfect for and no one is really going to miss.
SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
DevOps was a reasonable idea, but it rarely actually manifested. If a company had an actual "DevOps group" separate from the developers, by definition they weren't doing DevOps.
Part of the problem, and what ultimately drove me out of coding altogether, is that Ops is a specialized skillset completely orthoganal to the business of building applications. And, I was really bad at it due to a complete lack of interest. As cloud environments get more and more complicated one ends up spending more and more time dealing with that at the expense of building things.
Conversely, there are lots of folks who love ops type problems and are not well suited to building apps. So, it makes sense to put them on separate teams where they can each do the thing that they're best at, rather than fumble through the parts of the jobs they hate. But yes, at this point it's not really DevOps anymore, it's just ops.
At one point I was very hopeful that the progression from Infrastructure as a service to Platform as a service was going to simplify things. Instead we seem to have ended up with mind boggling complex services.
I think this one is really localized.
Some places do have really well built out platforms where logging, instrumentation, databases, deployment methodologies, IaC, etc have well-built happy paths that exclude a bunch of accidental complexity.
I think a lot of other places are either too small or grow too quickly with the wrong set of values and admit too much into their "platform" which had very little design to begin with.
IMO it was a worthwhile experiment that probably went too far before being rejected.
Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful. Especially for certain kinds of products, if executed with discipline you can get really good engineering reaction time with no degradation of quality, and often with much cheaper infra costs.
The problem is what you said though. People with both skillsets are rare, and combining them is legitimately a third skill on its own. And even "both" skillsets is a simplification once you start accounting for data, security, compliance, large scaling DB work etc.
It was mostly a dead end for these reasons but it was worth exploring for its potential. And some genuinely good engineering techniques went mainstream out of it, not a total waste.
Isn't that just anyone with an "architect" job title though? I don't think that kind of dev is particularly rare at all.
I've found that "ops" has become far too overabstracted and jargony. Their moat is endless "certifications". Their "experts" are the dogmatic personality types who try everything in their power to prevent their feet from ever touching the ground like it's lava. They lack the kind of dev or sysadmin experience where you have to stand under your own constructions like a Roman engineer.
Any formerly-known-as "full stack" dev who kept at it for the last decade and a half would run circles around anyone in ops. That's every senior dev working right now who doesn't like to manage people.
It used to be that the ones who went SRE just wanted a pay bump for a bullshit job of relatively light coding puzzles and parroting stuff in meetings, but I guess that well has dried up with "AI".
Right but the dream and promise of devops was that early-to-mid career line engineers could do these things, that these weren't rare skills and combining them wasn't particularly challenging. This turned out not to be the case, these skills are the requirement not the outcome.
But if you have the budget and connections to hire a department of people with these skills and experience, and a problem interesting enough to keep them engaged in the day to day building, and a leader capable and charismatic enough to align them all into the business's goals... you should do that.
Coming from the Ops side, “full stack” usually translates to “our entire infrastructure stack is managed AWS services, we don’t know how they work”.
It turns out about as well as if you set the Ops team to shipping features.
Eg I’ve seen more than one full stack dev team build a synchronous RPC-over-message-bus framework because they can’t figure out how to make HTTP or actual RPC calls reliable. It is hands down the most poorly and overengineered thing I have ever seen on a service that takes single digit qps.
The ops teams fight that shit tooth and nail because if it blows up, they’re the ones who get called. And it is 100% too late to fix it by then. Ripping it out will take months, and the dev team will say they’ll do it, but they never will because the VP is leaning on them about some feature.
You either say no early, or you are forever on the hook for whatever abomination the intern has cooked up.
> Having the maintenance/ops requirements directly feed back into the design and engineering of the product is genuinely powerful.
You’re so right. This is the key, whether a team is doing devops or not.
One of my major contributions when I took on a CTO role was getting dev, ops, and support teams to talk about the pain points. There were so many easy wins to make things run better because the teams didn’t understand the difficulties the other teams were going through.
it only really worked as part of a team.
Where it worked well is the "devops" or sre/sysadmin would sit in the same team and take away the rough edges of the infra.
conversly they would also keep an eye on the web people and see what they are doing and why, learning the tooling they use and try and rationalise it so they don't install 50 messaging queues.
> Ops is a specialized skillset completely orthog[o]nal to the business of building applications.
Exactly this was the idea: persons in a DevOps role have skills in both areas, i.e. are very wide skill set, so that one can assign them to very diverse tasks for more flexibility. Additionally, these persons can implement synergies to make things more efficient in areas that intersect both development and operation.
No, what you descrive is SRE. DevOps was the idea to give people that work 100% as developers the ability to do some amount of ops work (let say 5%), as a way to break the silo between Dev and Ops that existed in older organizations.
I am skeptical that either label is sufficiently well-defined and more importantly well-understood within the industry to offer definitive statements on the differences.
Of course it's misunderstood and misused, but that doesn't mean that more discerning people can't understand where it came from and why, and then used the term properly.
In practice though, at least at places I worked, it wasn't 5%, it was more like 80%.
For everyone ?
yeah, I am a software developer, not a system administrator.
In the early days of devops it was primarily focused on supporting developer productivity (CI/CD) but quickly morphed into sysadmin once cloud computing took off.
Imo a dev with no ops exp is like a chef who never tastes their own meals.
I think a more apt analogy is a chef who is not waiting tables. Which, by the way, are typically completely different roles in a restaurant.
Your comparison might work better for a dev with no testing experience. Write the code and throw it over the wall. But I fail to see how an application engineer should need to worry about the intricacies of, say, setting up virtual private gateways.
I disagree: ops is related to dev, because good dev work makes ops easy. But they are different: being good at ops requires a mindset of curiosity, risk management, and being willing to chase things to to last details.
Yes, big companies have a tendency to over-structure things. Devops? Sounds great, let's have a dedicated team for it. Agile? Absolutely, and to make sure it's done right, we need a rigid process around it.
The rigid process around agile is absolutely mind-blowing to me, and it seems like it's everywhere in corporate environments, but my current client dials it up to 11. Maddening.
Real DevOps is actually just Dev - in that the Devs are doing the Ops. The people called DevOps are just there to backstop the devs oncall when shit hits fan and you need someone more experienced in ops / incident management to escalate to.
I am consistently surprised that I'm unique for enjoying, and if I may be so bold, being pretty good at, both Ops and appdev. Now I'm not the greatest in the world, but I really do firmly believe now that most generalists should strive to be able to do both at a journeyman level.
To my understanding, this _is_ DevOps in the original sense of the word, and the version where they are a separate team is an offshoot bastardization.
At my last job, it was somewhere in-between where we had DevOps specialists who were regular software engineers who allocated the bulk of their time to building / running infra via Real Code (Pulumi vs. TF, minimal YAML wrangling etc) and built paths for app devs to self-service as much as they could. That, I felt, was a really excellent setup.
Devops definitely still exists, I've seen it. It is when people are responsible, even in the middle of the night, for the systems they fucked up engineering in the day.
SRE at google works closely with devs to architect and build systems together. Most of them are on the SWE ladder, are able to bug fix and scale directly. They're not just ops.
The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
If SRE cannot throttle the release cadence or fix bugs or change architecture in binaries, it's not really SRE.
Nice I started a timer for how quickly a "well actually, SRE at google" would show up and you did not fail to deliver.
> The point of SRE is to have a team dedicated to scaling/resilience to balance the tension between product pushing to ship features.
Sounds like a cultural issue. Google should look into that.
He's right you know.
It turns out it's easier to hire people who specialize in scalability, reliability, debugging of distributed complex systems, cost ops, efficiency gains, etc. than it is to instantly ensure every single engineer that works there is an expert in all those things, without anybody on staff who is an expert in them. Much like database skills, IT security skills, etc.
It's still something everyone can theoretically learn, of course, and someone who works in that role is most effective by instilling those disciplines in others as they go, and making everyone successful.
A cultural issue that many organizations face. Same reason most large companies have an infosec department that runs separately from IT.
Why is your response so pithy? Google invented the term SRE, and I think how they use SRE is pretty relevant to the discussion.
Oddly the industry mostly rejected googles application of SRE even as they adopted the term. I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
> I've only seen Google use it to refer to engineers maintaining / operating someone elses code.
You didn't look around very hard then. I acknowledge that certain areas of the company can consistently use subsets of a role, so you may not have encountered it personally, but your view is definitely wrong about the role in the company as a whole.
It’s cause they don’t know what they are talking about.
Easier to shit on people and things with hot take then actually have a grounded and coherent argument or realise that different companies/teams have different constraints, problems and environments they operate as well as different solutions.
When I worked at Google SREs only handled legacy products that were maintained but not in active development. SRE was its own vertical separate from SWE and they would only take on maintenance of a new service or product if SWE could prove to them it was stable.
I know Google SREs who do feature development on the infrastructure support they support alongside their devs.
ZIRP? That's not an acronym I'm familiar with.
Zero interest rate policy
https://en.wikipedia.org/wiki/Zero_interest-rate_policy
"Zero" Interest Rate Period, during the COVID pandemic, where most governments offered very cheap bonds or printed money to reduce interest rates. This caused a lot of investment as money was "cheap" for investors and needed to find a worthwhile investment in order to grow faster than inflation.
This caused a lot of money to be poured into all kinds of projects, not all of which had a positive return on investment.
It wasn't COVID that kicked off ZIRP, it was the global financial crisis in 2008.
ZIRP goes back to 2010 actually, when interest rates dropped so much in an attempt to stop the great recession.
We should do it again but the right way this time.
Give literal zero interest loans from a state bank based on social value and economic priorities, just like they do in China. And set up an extreme progressive tax (90% top bracket) on returns that are paid out to owners to make sure everything gets invested back into production.
More investment is the driver of economic growth. All the bad parts of ZIRP came from letting companies use the money for unproductive activity. If it's used efficiently the downward pressure from higher product supply will be stronger than inflation from more money in the system. And you get more demand in the labor market leading to higher wages.
Sre seems to be disappearing anyway in favor of infrastructure and platform engineering. Which seems to just be kubernetes admin jobs using terraform.
Continuing the never ending trend of Sys Admins/DevOps/SRE job title holders with skills in specific tools rather than foundational skills always wondering why they are out of a job every 5 years.
You have no idea what you're talking about if that's what you think SREs are.
That's reinforced by how hiring is done though. I rarely if ever run into companies that hire for fundamentals. Only the Google tier of companies do that.
That’s what companies hire for. I’ve learned those skills and no one ever really wants them.
If you build tools with that context, they’ll have to replace you with someone that has that context. There’s few of them, and they’re expensive.
It’s easier to hire folks that should be interns or juniors that know that one tool, and then replace the whole department every decade.
It’s why everything is YAML now, despite it being horribly verbose and clunky. Can’t have our people needing to know how load balancing works, just add another 95 lines of YAML boilerplate so they don’t need to learn 5 lines of nginx config.
I never understand this sentiment. I do SRE/DevOps/Platform Engineering/Infrastructure Engineering, and I definitely do need to know how underlying technologies work. Even when everything is kubernetes and YAML.
If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that. Your company can also choose to outsource complexity to the cloud, and you’ll just be writing YAML. Then there’s also companies where you host everything yourself AND you have large/high workloads, then you still need people with understanding.
With your type of comments, it feels like you’re pretending a developer who writes WordPress websites does the same work as someone doing Erlang for instance. You’ll have different skill sets for every type of engineer, and they’ll all be necessary.
I also work in the space.
I think you’d be surprised if you look at the younger (post-2000-ish) companies. Specifically about this part:
> If you run simple workloads, it’s probably not necessary but then you’ll probably be paid for that.
You would be surprised how many workloads either are simple, or can be ham-fistedly shoved into a “simple” architecture.
Eg I’ve seen a lot of Postgres-on-k8s solutions that are basically just Postgres on bare metal with a bunch of useless abstractions. It writes to local disk, moving the pod to a new host is basically restoring from backup. Wrap it in some tool that ingests YAML and call it a day.
k8s is sort of the king of this. I hear people say they understand k8s and I can’t help but furrow my eyebrows. I don’t even pretend to, there are so many insanely complicated edge cases.
Eg if I’m sending UDP packets to a service and scale the deployment from 2 pods to 4 and the host for one of the new pods crashes and comes back up with a bad iptables rule, what are the range of potential behaviors k8s would show? What’s the difference between blocking say all traffic vs only traffic to one or two of the etcd instances? What if it’s blocking half the hosts in the cluster?
Storage has its own weird vagaries around single vs multi mounts and race conditions therein.
k8s is the king of “I don’t have to know that, the platform will do it”. I’m not sure I’ve ever heard anyone outside networking talk about whether a pod will saturate a NIC. Just chuck it in the cloud and let it figure itself out.
Kubernetes is probably more responsible for deskilling, or at least constraining, the expertise, of SRE/devops/platform/infrastructure engineers than anything.
I don't think that's a bad thing, it makes infrastructure more consistent and less weird (assuming you understand k8s) but it is a very high level of abstraction and for a bunch of people is just a black box.
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Yea cause you wouldn't escalate to 50 people on the weekend when one person can do it. Op is basically saying they never worked on a big enough system.
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
That doesn't apply when your org is big.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
To me the best compromise is put a price on it.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
there is always infra/platform underneath the system, they are the SREs
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
I call that job SysOps - caring about systems and operation
SRE just came from the big guys calling their sysadmins SRE
If you look at the history of SRE, it was distinctly NOT sysadmins, Google specifically wanted software engineers hired to automate scaling as well as sysadmin type ops work.
their work is entirely what sysadmins did tho
RCA of complex bugs? Writing infra software?
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
On-call problems are mostly just a result of Kubernetes' "cattle not pets" brainrot ideology where people throw local simulate-ability out the window because the Rube-Goldberg web of pods and operators and cluster state gets too complex to simulate locally or keep in your head (esp. if code is located in 50+ microservice repos).
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
>Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen".
Uh, I can literally see thousands of crashes happening all the time. You think there isn't a .01% chance of hardware going bad or hitting kernel bugs?
It's not that crashes don't happen, it's just that they don't "randomly" happen. The error codes from bad hardware are predictable (against kernel bugs... well I use an "allnoconfig" kernel, which disables like 95% of the kernel). Bad hardware, bad network, bad RAM, all of these are simulateable in QEMU or Linux (tc netem / blkdebug).
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
i'm not questioning your setup, I'm saying that if you need a 10,000 machines to run a system you're going to have kernel bugs and hardware bugs and you have no choice to treat machines like cattle.
> Real ones are oncall for the systems they build.
Code of Hammurabi's Law 229: "If a carpenter has erected a poorly constructed house, so that the walls cave in and kill the homeowner, that carpenter shall be executed."
> Real ones are oncall for the systems they build.
What the fuck do you think the DevOps movement was about?
AI isn't the only cause of deskilling. I've seen a few people go from coder to architect roles. Architects, being the decision makers, talk to vendors, CxOs, consultants,... and mainly spend their lives in powerpoints and visions.
When an engineer stops coding, managing servers, ... it takes about 6 months to transform even a really smart person into a buzzword factory. It is scary. People like Oracle or Microsoft sales are really good in decoupling all of us from the relevant day to day concerns. It is reversible too, with former architects getting really angry if they have to deal with the software they themselves procured or envisioned.
My take is that you need non stop contact with reality, both the computers and the customers, or you start to drift. I fear AI is causing this in all of us.
This has been going on in other professions for a long time. See the "children of the magenta line" talk about pilots losing the ability to hand-fly.
Another oft-cited article here is William Langewiesche's article on the AF447 crash (https://archive.vanityfair.com/article/2014/10/the-human-fac...):
> To put it briefly, automation has made it more and more unlikely that ordinary airline pilots will ever have to face a raw crisis in flight—but also more and more unlikely that they will be able to cope with such a crisis if one arises. Moreover, it is not clear that there is a way to resolve this paradox.
I bought Sully Sullenberger's book in early 2012. He's the pilot who perfectly ditched his Airbus A320 in the Hudson River after losing both engines on takeoff to bird strikes.
One of the things from the book was how Sully spent a lot of time investigating other pilots' accidents: 'was there anything that pilot could have done to have survived?' When he got to his own emergency situation over the Hudson, time slowed down and he decided over the course of a few seconds that ditching in the river was their only option.
"One way of looking at this might be that for 42 years, I've been making small, regular deposits in this bank of experience, education and training. And on January 15, the balance was sufficient so that I could make a very large withdrawal." https://en.wikipedia.org/wiki/Sully_Sullenberger
s/ditched/expertly landed on an improvised water runway/
I like this analogy that I've come up with: Non-writers can produce impressive articles that are unbearable to read the same way non-programmers can produce impressive software that is unbearable to use.
I think people are getting funneled into a new career path of being a technologist or technology director. People who can ask the right questions at the right time and direct attention to the appropriate areas given the goals and feedback.
People don't appreciate how much of engineering is just as you've described.
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
Admittedly, a lot of my identity was wrapped up in those hard earned skills. I have empathy for those struggling to chart a new path for themselves. It does not, however, change the fact that tools evolve. Ultimately, using tools effectively is how we are measured and while you may have needed a shovel yesterday, today you have an excavator.
> "...imagine the vibe-coder who has the same curiosity and inquisitiveness that you had..."
We're required to imagine them because they don't seem to exist in the real world. The vibe-coders seem almost entirely composed of the "I commit 20 kLOC a week without reading a single line, lol" or the "LLMs are like calculators/basic arithmetic skills are obsolete" lunatics.
This has always been the most valuable part of the job if you ask me. Its just that the other parts of the job were quite hard to get right letting people build a career out of that specialization.
I dug into general AI deskilling a couple days back[1] - no good signs anywhere, but there's a few things we can do like make concerted efforts to do programming (like a coding hour every week), do a Leet code question, and getting the agent to quiz before merging something (stolen from Notion).
[1] https://www.youtube.com/watch?v=EWuf2EekRSM
> make concerted efforts to do programming (like a coding hour every week), do a Leet code question
I'll probably get downvoted for this but: At 15 YoE, hell no.
Day-to-day coding lost it's allure to me about 3-4 years ago; it's just a means to an end. I think AI came at the right time for me.
At 40 YoE, it seems like your heart was probably never in it. I really miss coding at my day job. I still code plenty for my personal projects.
Let's not rush to such judgment - our desires change as we age, the field has changed (including what nose-to-the-grindstone coding looks like) and burnout affects many.
Plus, some people enjoy the rigors of solving in-the-weeds code problems and others enjoy building systems-level designs. This was always true.
You guys act like the React component nonsense from the past 10 years is the same as the code I was learning at age 16. It's by far and away completely different. And none of it has been audited by Daybreak.
That's precisely my point when I say the field has changed even for "true code writing."
I'm very happy to let the robot handle any incidental React or other extraneous tools that end up adjacent to the core work I'm doing. It avoids getting blocked without interrupting teammates or spending time and cognitive capacity on stuff that has no lasting value in my work.
Exactly, thank you. I'm much more interested in how systems interact gracefully, addressing failure modes and building for resiliency/scale.
Do you need good quality code to do that? Sure, but that's just another line-item on the laundry list of things to make it happen.
I've been having this same gnawing feeling. I just started reading the book "The Collision" by Eric C So which promises to cover this subject. I haven't finished it but seems decent so far.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I'd love to hear what others are thinking/doing.
I've actually been thinking that designing a Zachtronics-like would be both great practice and great fun!
I've been doing more networking recently. But any skill building is probably worthwhile.
I don't know if it's true. I didn't program at all between 2000 and 2015 and I've been able to restart without too much effort. I forgot many things, but I found it relatively easy to recover...
I'm more worried about people that did never program and started directly with LLM support...
I’m personally using it to augment my coding more than replace it. I find joy in it.
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
Nice video, but I have said it before, the difference between just “vibe coding” and reliably using AI is the specs you need to write first and the tests after. If you start prompting the end goal/s the AI will basically guide you on what needs to be done and you just accept it as long as it runs, and this is following what most software developers were doing before LLM, the agile way, the quick ship way, the daily standup and ship first discus later way, the code pairing way, which worked mostly (even though I hated personally) in teams, so these developers are replacing what they were used to do with an agent and that’s the mistake. However, if you go with traditional engineering ways, waterfall, specs based, blueprints, and then tests, you control the agents and guide them not the other way around, in your specs you define the tiniest possible thing to what library to be used or excluded, the methods and approaches, among others, the agent here will produce the syntax and help you in the tests, just like in other engineering aspects how machines didn’t take over, they follow your specs and frameworks. The syntax is the past, AI can produce better syntax than any human alive or all combined, if you think otherwise you are delusional and still didn’t move on, it’s time to stop the “developer” mindset and do the engineering one.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
We use AI at work, Claude specifically, and it’s faster than avg SRE for trivial tasks like cross signals correlation. But it requires a lot of effort to build a harness from your company contexts for it to work like an SRE. And it’s just a tool so if you have to build your knowledge to make it useful, otherwise it’s just a blunt. Or in other word, incompetent people are incompetent no matter of tools
Is it really that complicated to set up? A basic version is make everything work from a CLI in terminal coding agent of choice (probably already mostly there), hook it up to alerts, and let it run in a corner?
Not OP, for me the issue is nuance of all of your infrastructure, application and outside layers. There's a lot of documentation and such, but often times things don't work as expected to a degree and it requires filling in the blanks correctly. The amount of times I've had an AI Agent misinterpret an alert and assume it was XYZ when it was some random layer it had no idea about is often.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
From what I've seen the more complicated the deployment and the more tedious the investigation the more useful the automation is. The agent shouldn't be interpreting the alert much beyond gathering symptoms and identifiers for drilling down into all the underlying data and code. The security side of this is very tricky because the speed that makes the tooling useful also makes it hard to supervise.
What contexts would the AI model need for analyzing, let's say, cluster-wide incident other than the observability data that such systems anyway collect already?
The entire company code base, design docs, realtime threat intelligence feeds, all the vendors' status pages, etc...
Not sure how that relates to incident debugging?
Very often the causes of an incident aren't visible in metrics because the relevant metrics aren't collected. Sometimes an incident is caused by external actors, etc...
Yes but IME it is never possible to collect everything. It's part of the job to deduce the missing parts from what we have available.
You're not sure how the system's code relates to incident debugging for that system?
Yes, "entire" company codebase is a stretch as is design docs, status pages etc Helpful yes but not a hard requirement per se. In many places you won't find such level of details available and yet the incidents are still being solved
"I'm as sharp as ever says SRE whose skills declined 30% in 2 years of AI use" - if there was a tech space Onion
I don't even know if this is an AI thing. I've worked at places that have varying levels of operational maturity, and some of the ones that had very old SRE orgs also had a lot of their SREs operating mostly in the tier 3 given by the article.
One of these orgs had basically all of the noisy pages under control through automated remediation and a culture that pushed towards aggressive automation of pages that had the same root causes. Almost everything I was paged for was some novel or uncommon fault condition. Developing an intuition about how the system worked (well, more specifically, how the system failed) took a lot longer because most things I had to look at were some edge case failure.
> Tier 3 incidents: Novel, complex, cascading failures stay human-led, with AI providing supporting context.
> But here’s the catch.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
How did they do this before AI? I have done a bunch of complex production level debugging and these Tier 3 incidents are always "starting from scratch" never seen before issues. The main "intuition" I built is persistence and verifying/refuting random hypothesis. In other words, I think we can build an intuition for Tier 3 issues only by engaging with Tier 3 issues.
Although, psychologically, some success in T1 and T2 issues is what gives us the confidence to even approach the T3 issues.
When you're just starting out in a system, every incident is a novel, never-before-seen issue. That's putting aside existing sources of knowledge like coworkers and runbooks of course, but I've had to do "software archeology" on lots of stuff because coworkers are gone and docs are wrong. But that's where you build up that intuition, like how the company typically builds things, how deployments usually work, what's that terrible load bearing service installed 10 years ago and not updated since, etc. Yeah every T3 incident eventually is novel, but being able to cut out the non-novel parts helps a lot
Deskilling has been happening for centuries. One doesn't farm their own vegetables, hunt for their meat, slaughter their own livestock, maintain their own vehicles (ahem - remember the horses and carriages). We've even stopped shifting the gears on our cars. Why are we complaining about AI?
The biggest problem I see with AI isn't deskilling. It's the same dopamine hit one gets from social media but for their profession.
I haven't touched AI, so this SRE is still skilled!
Now if only companies wanted to hire people that still have skills and aren't just prompt engineers.
Hang in there, in a few years we'll be mining gold cleaning up the mess.
I work in an AI sceptical organization, it's not that we don't use AI agent or consult chat bots, but they are kept a good distance from production. Talking to other SREs in AI heavy and AI first companies I don't see that ending well. SREs struggled with operating applications from developers who cannot or will not understand how computers and the internet work. Now many of the junior developers are even worse and some of them are given the title of "DevOps" and they are using AI to create some of the most broken ass Terraform code and bastard Helm charts to run their broken code. Some SREs are spending more time fighting agents and agent written infrastructure code than they would have, had they been allowed to write it themselves.
If tokens becomes to expensive, we're left with a ton of broken code and infrastructure that can only be managed at an inflated cost. There will be good money to be made consulting and cleaning up the mess of AI boom.
The devops department at my company, which has around 500 services across a load of kubernetes clusters / regions are all amazed at how well gpt-6.1-luna is at their jobs. They've all been using it exclusively for 2 weeks and between them have spent like $400.
The take where in a couple years there will be a massive revert or that anyone cares about terraform indentation or a shitty yaml template is wildly optimistic.
I do hope you're right though, but I'm not seeing it from my side of the fence.
I assigned myself to re-read "Flowers for Algernon" based on my current frustrations with the team / cultural impacts of AI usage.
It's mildly orthogonal, but I think the metaphor is really quite apropos.
Learning to Be Me by Greg Egan is also apropos.
If you are a single-task person and never stop to think why you do things probably AI can replace you.
SREs are about super-thinking, context switching, recognize patters and workflows, data movements, etc..
I disagree, it gives me the ability to apply first principle knowledge unhindered by whatever fuckass idiotic idiosyncrasies a particular cloud platform considers “good design”
AWS, Azure, GCE, Alibaba Cloud - all the same crap, different names and slightly different jenga towers.
It applies to surprisingly a lot of crap and it’s a good substitute for where we can’t put automation in yet.
I think this works better when you have the underlying knowledge which was probably always a minority of the market ( but still a significant cohort )
Imagine SRE that very quickly triaged the incident, gave solution and it turns out that solution made system ever worse, and after that he/she said: "You're absolutely, right!"
Deskilling is not necessarily a bad thing, if what you refer to as skills are mundane things that should be automated. I do agree with the author in that people are becoming lazy when it comes to critical thinking and evaluating the decisions or work that AI presents to them. These are people who have little passion to do what they do, and for whom their role is primarily a chore.
I think that's one reason - another reason is that people are time-constrained, backlogged and expected to do more than ever.
I don't think this is as black and white. You skill up by spending time with the domain. Of course, you learn better when you are involved in situations or jobs that actually matter. But getting rid of all the "mundane" distances you from the nitty-gritty and you lose that feel for the problem.
Right, there's a reason students that never dealt with a real problem, even when they get As, aren't as good as professionals. You need to do real work to get really good.
Agreed. I think there's also an argument to be made that it's upskilling non SREs to now be able to do SRE things.
Yes. I bet it was really fun debugging punch card computers.
Also, skills can be re-acquired, if needed. People are acting like it's the end of the world for them, like their knowledge will become impossible to understand anymore. I'm not as fast at solving equations as I was in high school, but I still understand the logic behind it.
We will adapt just fine, like we did before.
As an SRE Engineer (RAS syndrome, I know), I was just having this talk with my supervisor.
My original career/learning path was for a traditional kind of SysAdmin role, because that's what I like. Doing Linux administration and such. Then our whole stack went AWS/cloud-native, so, ok. Seems like no one cares about someone who can actually SSH into a Linux server and figure out what the problem is--they want everything to be built into CloudFormation so that when a server is misbehaving you pull a switch to trapdoor it into oblivion and drop an identical one into its place, in the hope that this one works. I didn't relish the idea of moving away to the increasingly abstract, but alright, I can deal. Reformulating my learning path into being a Certified AWS Full-Stack Engineer or whatever.
As that was in progress, my company drank the AI kool-aid and went all in on AI. Now I honestly don't know what I should be training for. They've been pretty consistent on "No new staff, just AI", "You don't need X, you just need AI"--so I am honestly not even sure at this point if my company even wants me spending time diving more into the AWS stack, or if their attitude is "The AI knows everything--put all of your time into AI prompt engineering and just push whatever buttons the AI tells you to do."
The desired company skills continue to drift from my "personal" desired skills, but if this is an industry trend I'm not sure I have much of a choice, because I need my job.
Same happens with SWE
> Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures
AI is already pretty good at this in "SRE" workflows.
> AI-driven change risk assessment
And half decent at this.
Ops is bifurcating in two:
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
Org 1 will probably survive better as they will adopt the right primitives - logging, compliance, etc.
Ref: Agent management by Microsoft, AI Gateways springing up everywhere, etc.
If it deskills that guy who reaches for complex bash when they should reach for golang but they don't want to bother with writing the build layer in the image, then I'm all for it.
Plenty of people citing aviation. Aviation is deeply regulated at a global level. This is never going to happen with software engineering. Even where it already happens “locally” like in some military sectors, the results are self revealing. It gets crushed by equivalent unregulated competition in pretty much all metrics.
I stopped reading halfway because it doesn’t describe competent SREs in an organization where the culture values reliability.
> Alert noise reduction and intelligent grouping, runbook execution for known issues, log pattern detection and anomaly flagging, and basic root-cause suggestions from historical incident data, and auto-remediation for well-understood failure modes like restarting a crashed pod or scaling up a service that’s running hot, are all fairly automated today.
Let’s stop here and ask, why do we not fix the underlying issue? Alert noise? Design better alerts. Runbook execution? Make it a cronjob. Known issues? Why aren’t people fixing these known issues? Root cause suggestion from historical data? Sorry if root cause A is in the historical data you expect problem A is fully solved so it wouldn’t appear again in the next incident. Restarting things? Why doesn’t your health monitor restart things automatically?
SREs also need to climb up the abstraction ladder. Instead of using AI to assist with grunt work, use AI or human thinking to build more reliable systems with fewer incidents. Every incident should be novel.
I'm glad you apparently live in an ideal world where every incident is fully root caused and fixed at the origin, and all your coders produce perfect, bug-free code.
Most of us, however, do not.
Every incident having a root cause is required. If your organization doesn’t do this, it basically means they don’t value reliability at all. The best SREs quit. Well either that or the SREs have the ability to transfer oncall responsibility to SWEs. Whether or not that can be fixed at the origin is an entirely different matter. Bug-free code doesn’t really matter: the best incidents are the kind that reveals hitherto unknown bugs.
I've started to use the term "downskilling" for what a number of companies are going through in the age of agents: layoffs of key employees in the name of increased efficiency, developers being incentivized to outsource their thinking when writing code, tests, specs, docs or reviews, agents increasingly replacing communication between team members...
I already sucked at incident handling before, it was more of a panic attack mixed with the fear of getting fires. So Agents have a good place there.
Reminds me of Children of the Magenta Line: https://youtu.be/5ESJH1NLMLs?si=wozPZOKAFKNcjPSm
Fly the plane, not the automation.
This has been my concern since the moment it became clear how capable LLMs had become and were likely to become.
I feel like I've been going insane as people have ignored this for short-term gains.
if you want a vision of the future, winston, imagine 'X isn't Y, it's Z'... forever
I'm wondering if it would be a good idea to have the AI occasionally throw a wrench in the works of the dev environment Chaos Monkey style to keep the SRE team accustomed to handling responses.
No it's forcing everyone to realize all the jargon was made up to sell books and land speaking engagements. Money from patents (2016 Alice ruling upending that whole scam).
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
10000% agree. I am seeing this first hand. SRE and NOCs folks are turning into meat proxies.
Is HN upvoting just because they agree with the title?
Clearly the "author" didn't even put in the effort necessary to make a real title. Not "SRE persists but a skill crisis looms" not "SRE careers remain but competency lowers in spite of AI" not "AI lowers SRE skill but demand is steady". Moreover, hasn't this type of insufferability been dramatically reduced in recent models? It screams that in addition to being slop, it's outdated.
Why would anyone take this at face value?
I don’t understand what this title is saying, feels a bit circular. You could say this about so many jobs that are or are not being replaced by AI.
Do LLMs “deskill” us in general? I suspect we are just replacing old skills with new skills.
I wanted to use this post as a source to write on my web server project's blog on why debuggable web infrastructure matters even if AI SRE agents exists, but the idea got delayed and I ended up posting it on HN... :')
Here's a thought experiment: why is it that I can't do long division?
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI makes stuff that was hard - someone's bread and butter - possible to ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for tasks that used to take forever to become something to steamroll through on your way to a much greater outcome. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
This is a poor analogy because I don't think your inability to do long division will cause as much damage as an inability to bring the site back up in an emergency situation.
If a team can't bring the site back up in an emergency situation that's poor planning, not poor debugging.
Planning for mitigation before resolution lets a team address production issues at leisure (because the customer impact is mitigated)
All this knowledge is in your agent - ask it for ideas!
the analogy doesn't work unless your role means having to occasionally do long division, on the spot, without a calculator
Just to be clear are you suggesting one might find oneself without a clanker?
Where're you doing your debugging? The tundra??
Except as every damn AI app says: AI is prone to making mistakes. If AI deskills you with a non-zero chance of getting the problem wrong, that seems deeply, profoundly bad.
You can type the same long division problem into a calculator 65,000 times and you will get the right answer 65,000 times.
Can AI get even CLOSE to that?
Computing tapped out on sources of leverage for solving deterministic problems a while ago. Computers got fast and our algorithms got really good.
We're on to the next frontier, and it's a bit weird. Jagged frontier and all. But it's opening up new worlds.
Luckily loads of problems are amenable to a non-deterministic approach - even better, many have lots of solutions. SRE is in that bucket. There are as many ways to SRE as there are SREs
(source: one of the first SREs at Uber, co-invented Chaos Engineering as a sub-branch of SRE, hosted first chaos eng conf at Uber, played a key role in Uber's global SRE and incident response "Ring0" initiative, managed many SRE teams over a period of roughly 7 years before a career pivot. You could say I know a thing or two about SRE.)
To SRE well you look at the situation and pull from a huge range of tools. With such a broad range of solves, you're better to focus on the outcome: did we achieve our SLOs? Are they aligned with customer expectations? Focus on defining that and providing the right support for it to be achieved and you'll be fine - you might even get a surprise.
You can own a calculator for $2, good luck owning your own frontier model
Of all the comments on this post, this landed the most.
I feel there is a potential to "distill" the frontier models into useful tools that can run on smaller hardware. Or distill that knowledge to produce hardware, compute or otherwise. Iff one day the frontier model become inaccessible, the tools that were carved with them may remain valuable.
They will remain most valuable if they are designed in a way that is maintainable even without LLMs. If not, they can be studied as artifacts, or reference designs and then recreated using human intelligence.
The analog I'm playing with:
You can rent time at a high end machine shop and use the best CNC machines to produce extremely valuable tools. Once you leave, you don't have access to those tools, so it is advantageous to produce durable artifacts that are valuable on their own — on a different frontier.
Imagine you're going beyond the frontier physically, exploring new parts of the earth or space. You can't bring the giant machine shop, let alone the machinery of society, the entire economy etc. What do you bring? What do you build now while you have access to the best machines to take with you?
Nobody can read a map anymore, either. Or operate a loom, etc, etc.
I recently won the red ribbon, second place, at the Lambtown fiber festival, for my lovely skein of undyed handspun yarn. Weaving it is next. However, I am probably an outlier, and should not be counted.
(It was a full 250g of this fantastical kempy Herdwick, absolutely unlike anything you can get commercially, and I'm honestly more proud of it than of anything I built in six years working at NVIDIA.)
> why is it that I can't do long division?
> The Calculator!
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
Edit: "HackerNews" and people can't do long division??
> in the macro, humanity becomes far more capable as a species
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
I think a closer analogy would be forgetting basic arithmetic as a concept. Imagine how hard it would be to work with more complex math without understanding what division is. How do you prompt the genie, or understand its outputs, without the foundational knowledge?
Is basic arithmetic something that can be forgotten? Studies have suggested that even infants have an understanding of basic arithmetic. There is a reasonable likelihood that it is something innate.
As we're only imagining anyway, let's ignore the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
You're taking the analogy too literally. The point that I'm trying to make is that in math, and a lot of areas, is that concepts build upon one another. To understand something at a 7/10 of complexity it first needs you to understand the 6/10 concepts that it builds upon.
I believe it is more than you've missed the forest for the trees. You haven't accurately explained what understanding means. One might not rigorously understand x÷x=1 but still 'naturally' see that having two apples and taking one leaves one. And that's the thing: The genie, as in LLMs, are analogy machines so you can approach problems without a formal understanding as long as you can grasp the analogies. We have real world proof with 'vibe coders' producing incredibly complex math without much formal understanding of it. They know how to describe the input and interpret the output using words that relate to everyday things that they are familiar with, much like the apples on the table, and it accomplishes their goal.
Perhaps there is a limit, sure, but I suppose that's where you lead with a concrete example to explain where that line is.
You still need to be able to do long division so you can tell when you screwed up on the calculator.
How do you guys fight deskilling ? I can see my brain rot from reviewing $current_llm approaches to solving jira tickets.
Honestly, as the expectations change I don't see you really have a choice unless you're in the top 1% of engineers and can essentially dictate your terms. Even then I'd say the clock is running out.
At some point (we might already be there) someone insisting on doing everything manually will be seen like a carpenter who does traditional joinery. They might exist, but they're never going to be a significant percentage of the market compared to 4 guys with nail guns knocking walls together in hours.
Because for the vast majority of cases, "good enough" is good enough.
I can use nail guns fine, I just don't want to live in a world of people who only know nail guns, and keep my brain / body stimulated.
So they get deskilled which means they're no longer considered professionals but merely technicians which means they're now elligible for overtime.
I had never heard about SRE before and suddenly it's all I see. Modern problems (slop fest) require modern solutions (someone to babysit broken systems)?
https://en.wikipedia.org/wiki/Site_reliability_engineering
SRE has been around for at least a decade or more.
Obviius fix:
AI Isn't Replacing SREs. It's Destilling Them.
I work on a major distributed system at a faang. We are currently in the process of letting ai be the first responder on all incidents
I’d love if author mentioned SRE is site reliability engineer. I’m still not sure if that’s it
But I'm not a factory worker sitting around waiting for something to break. I'm so busy. I'm solving more and ever more complex problems. If anything, I'm becoming more skilled...
At least part of it is that I waste less time debugging and testing, and now I have more time to learn new skills and technologies (faster).
> And to answer the burning question in your head, our goal isn’t AI that replaces SREs. It’s AI that supercharges SREs.
Ah, that bullshit reasoning again. No, if you build something that "supercharges SRE", someone will be made obsolete.
I'm geniuinely sick of this framing, seeing people at my company building company-wide skill repositories and shoving contents of their brains into slick Markdown documents. Well thanks, now our AI knows everything, good luck with your future career (if there is any left).
This neglected to say what an "SRE" is.
Site reliability engineers respond to emergencies and keeps systems online in real time. They are perpetually on call. As a swe, it's much better than having on call rotations.
Previous discussion on an extremely similar article: https://news.ycombinator.com/item?id=49574167. This one even name-checks The Ironies of Automation in pretty much the same way.
Thought about this one too. I was expecting to re-read this same article to see how I feel about it now.
I feel like I did re-read the same article, or at least a version of it. Is this how content marketing operates these days? Find a popular blog post, rewrite it and attach your company's name to the end?
It's fairly hard to understand to me, but also entirely unsurprising that swapping one occupation for another would be interpreted as a net loss at individual skill level.
You guys are still doing things, correct? Is my experience so outside the norm, or is stuff still challenging?
This is happening to SWEs too. If you have a certain amount of experience, you are capable of critically analyzing AI output. In Big Tech parlance, these are the Senior SWEs and higher. You require little to no supervision and can work independently.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
The point I got from the article was that AI is a cause for skill atrophy for any level of seniority, and tends to worsen over time. I can see this happening to myself in some areas, while in others I’m able to learn new skills I’m interested in.
- "not you OP but to the other guy linked on your post "writing code was always the easy part of the job"
- if writing code was that easy, i request you to come and write call of duty modern warfare 5 from scratch without a game engine in c++
idk.. it’s upskilling many people. AI seems to be very kind to people with ideas. It takes away the boring work and leaves the interesting part. So yes, it replaces and deskills those who only do boring work
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
Not deskilling all of us. Some of us with greybeards are immune to the hype train.
I had an incident in prod years ago, that no one was able to solve from my team, and it took days for me to figure out what is happening. It was a tricky one.
For Claude it took 10 minutes to solve it.
IDK... it's upskilling me ...
Yeah that. Watched our lot spending 2 hours using Claude to debug shit that was blindingly obvious if they actually didn't go to Claude first.
If Claude goes down or stops existing when we go down, we're fucked.
The best is when it's just two devs blindly sharing output from Claude back and forth during a major outage instead of USING THEIR BRAINS
Something tells me they were not very big on doing that even prior to 1 BC (Before Claude).
Our lot were. It's like they're now a 5 year old that has been exposed to Italian brain rot for 6 months. https://www.youtube.com/watch?v=DaJ0JAsJEAc
Why just SRE's it's deskilling everyone in this industry.
Deskilling is good, actually, as long as it's the right deskilling. "Skill" in software development is partly raw IQ and partly knowledge of complexity. Programming author Fred Brooks (author of The Mythical Man-Month) categorized complexity into "essential" and "incidental". Examples of incidental complexity are: React/Rails/Django/Laravel makes a backward-incompatible change to their interface and you need to scurry around the codebase fixing invocations. Or a senior developers on your team have a weird in-house coding style and all the junior devs need to learn it to get their code approved. Or management wants to move everything from one cloud provider to another so you have to change a bunch of resource URLs. Examples of essential complexity are: Self-driving car algorithms. Making your code run faster on the CPU. Adding support for a new web standard like SPDY.
I'm not lamenting the days when you needed to pull a seasoned (read: exquisitely expensive) COVOL developer out of retirement to make a change. Now AI can figure it out well enough, I just need to know how the basic concepts work.
I'm (still) a software engineer, and I've found it crucial to solve some LeetCode problems from time to time. It's honestly refreshing to write code yourself, make your own mistakes, and not end up low-key mad at a robot for failing to understand your lousy description because, apparently, it still can't read your mind.
Seeing this posted a week after I had to emergency push cve fixes to 1200 servers in production is pretty funny.
Every person on my team is capable of networking, product engineering, database, job processing and we have command of the entire stack. We will push updates for CVEs with stuff we manually compiled. We own the build system and have the ability to change product as needed. We all have experience in product and we're all capable of pushing fixes as needed. Frankly the SRE team is by far the smartest team I have ever worked with at any company I've ever worked at.
Is your org at the point where it would make sense just to upstream patches for FOSS projects and pull rather than patch and push to just your infra?
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
I generally don't reveal much about my current job on a blog post for security reasons but I can explain some of the reasoning for having your own build system.
We have many packages internally so inevitably we need to build them which necessitates an entire build matrix on it's own. But once you have your own build matrix adding a new package is trivial and most build in under 10 minutes in github actions. One issue is we don't want to use Canonical's default gcc flags so like Gentoo I setup a make.conf that is specific to the suite (22.04, 24.04, etc). This has allowed me to target a higher tier instruction set which is it's own micro optimization. Folks that upgrade blindly from 22.04 to 24.04 will see their performance drop from gcc compile flags alone. Also we have people who have access to CVE fixes before they are generally available so that's also another important reason. Anyway we DO use upstream most of the time for setting up a project folder or normalizing it against a new suite but then building it is using different gcc flags and might be pulling in our version of curl or something.
The ubiquity of this rhetoric is not an indication of its correctness. Repetitive tropes like "deskilling", "losing the ability to think" reflect a deep seated distrust of AI alone. No matter how many people parrot these fears the fact remains that you choose your attention. My manager has started coding, heavily, aided by frontier LLMs. He instinctively knows his boundaries and defers much of the more complex work to his engineers. We use the clankers too. Have we been deskilled? Hardly appropriate. Trans-skilled, or learning a new emphasis built upon our long experience? Much closer to the truth.
Wow. I wonder how the tech bros are gonna handle this thoughtful article.
I work a "dev"ops job, and like yeah, I'm pretty sure this field is even more fucked than software engineering is.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
Eh, I think they're still terrible at architecture. They'll find bugs but they'll try to make them go away instead of building a sustainable system.
They also have a propensity to try to rewrite gigantic amounts of code in one go making changes riskier and harder to scrutinize... without using more LLMs.
At this point a company can effectively fire you by taking away access the Enterprise LLMs.
As long as I receive my paycheck I don't care, be passive aggressive to me as much as you want!
I am more worried that an LLM company could destroy your entire company by removing access or losing enough cash to go under.
AWS, Azure, GCP, etc could already do this more effectively. Your bank even more effectively.
At least for LLM absolute worst case scenario you can run models yourself.
Yeah don't start me on using the cloud. I am well aware of that concern.