So now I am expecting a major GitHub Actions incident (worst case, the whole of GitHub) to go down every single month at least once. Last time this went down was last month. [0]
I now doubt if they can consistently manage more than a month without a major incident like this one.
Isn't this normal with hosted solutions?
IIRC, GitHub, Bitbucket and GitLab.com are all unusable for a few hours at least once a month as far back as I can remember.
Isn't this just commonly accepted as a tradeoff for not having to manage the servers yourself?
I would say it's normal for businesses which are engaged in competitive feature development.
Stability & reliability are relatively easy to achieve if you aren't changing the software frequently.
I'm pretty sure they're constantly working on their software (at least GitLab is), so "aren't changing anything" does probably not apply.
That was my point. If you want stability, look for someone who is no longer funding a substantial development team to work on the software.
This is a profound insight.
Nope, this isn't normal at all for products on AWS, GCP, or pretty much any other cloud provider. Azure is simply a subpar product and its time to stop attempting to sweep its awful downtime under the rug.
It could be interesting to track the uptime of such cloud services. Two decades ago, companies prided themselves with 4 or 5 nines (99.99% uptime). Not anymore. Worse is better won yet again ;-)
Companies and programmers should be aware what they get into when they build such dependencies. Distributed git is a thing, but distributed CI/CD that you could also run locally isn't (yet?).
The fact that it’s down is not the problem, but it’s painful that there’s nothing I can do to fix it.
That’s the part I like about self hosting. It may ultimately be down more often, but I never have to tell someone “Nothing to be done, we wait.”.
Which is better for you, but for your company, "wait, I'm on it" and "wait, they're on it" does not make a world of difference.
What's better, on the other hand, is that you can schedule expected and possible downtimes to a time that causes the least impact to your company; with a SaaS, an update might cause you problems any time.
You're second point exactly nails it. Hosted solutions break because they're busying pushing new features I may or may not care about. When self hosting I can decide when upgrading is worthwhile to my needs and then plan when to make risky actions according to my own organizations time line. Any single org can probably get away with 90% uptime just so long as the downtime is at the correct time.
While I sort of agree, I guess that’s that’s arguable. My bosses really like telling theirs that we are doing something about it too.
In these kinds of situations, you often end up in a situation where they say ‘the problem is resolving itself in region x’, where region x is not relevant to you at all. If you are fixing your own setup you can focus on exactly what is most important (to you) first.
Easy to explain. They switched to the MS Azure cloud for actions.
You won't get high availability from Microsoft you are used to as from proper cloud services. Plus privacy issues. But it's cheap, in this case for free.
Hopefully this serves as a good dogfooding excecise for MS now and help them improve things.
Every time I tried azure I was disappointed. But that doesn't mean they can't fix it; I bet there are now tons of talented engineers working there. My best wishes for them to up the quality of azure. I think diversity / alternatives are a good thing.
I was amused by reports when Microsoft was in talks with Discord that one of the reasons why Microsoft wanted to buy Discord was because they wanted Discord on Azure. Like, was the grand customer acquisition strategy for Azure just acquiring the companies and then migrating over?
My wife works as an AWS/Azure consultant, and she mentions that in our area it's much more common for the non-technical management to push Azure than it is for technology to choose it. Sounds quite IBM-ish/Oracle-ish.
When you open a Microsoft account for a new company, they do a lookup to figure out your area of activity and if you're a good match you get contacted by a sales representative asking you if you want to become an Azure re-seller. Basically for services you sell to third parties you get Azure credits meaning your Azure usage is "free", and your clients pay the premium. I know this because I used to work for a company that did this, and from personal experience when starting a company.
Edit: Here it a tip; if you see a "Microsoft partner <TIER> Cloud Platform" badge on a outsourcers website stay away.
I've had a couple of Fortune 500 companies as clients. Microsoft/Azure is usually brought it as a place that is treated like VMs in the cloud. The setup and management is often handled by Accenture and Infosys. Impossible that that decision was made by Engineering. In fact those Accenture managed setups are almost unusable for engineers. I can't even begin to fathom how much these companies spend on Accenture to setup Azure in a fashion that you can't do anything.
The worst part about Azure for me is always the list of undocumented bugs you run into. On the surface it looks like everything started as an AWS equivalent, but when you have to drill down on something it almost always has some weird issues that you then find as unresolved complaints on some MS managed github issue list.
But hey, maybe I was just luckier with the other cloud providers.
I worked on a project automating some parts of an Azure infrastructure for a big company. Half-way through development, JSON integers returned by Azure changed from strings to ints, back to strings. E.g., "42" became 42, then a few weeks later went back to "42".
This and other API weirdness gave me such Azure PTSD that I promised myself I would never touch it again.
That's exactly the situation I'm in at my current work environment now. All Azure, and everyone in Engineering/Devops hates it. It's a business decision though.
Sounds like MS, that is the only way they can get organic customers. All their popular products are originally built by someone else except Windows of course
Azure, .net, office, office 365, teams, ml.net, ...
Not a single one i could manage building ( eg. Teams has bots, quick to create apps and pretty advanced cam features)
Teams UX is a hot mess; it's just astonishingly bad.
The thing that made it the worst for me was the inconsistency across devices, such as only being able to directly reply to messages on the web and phone app but not the desktop app (at least on OSX).
I sometimes think MS product owners are on coke all the time or something. Last week or so i was in the middle of a conference call and MS teams just decided to restart itself with a new update.
Buying products though is very different than buying customers; if we try to map this hilarious customer "acquisition"--which is now a double entendre ;P--strategy to a more typical product, it would be akin to saying "no one is using Excel, so let's start buying large accounting firms currently using VisiCalc to migrate over".
Down ~4~ 9 hours so far today.