I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
> If the Linode Manager database only stores the hash, how would they know the password?
Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently.
The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters).
This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts.
But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present.
But the most effective "rainbow table"-like table is a look-up table with the followings:
* leaked password in plaintext, associate with email and any ID (forum username??)
* hash all of those passwords without salt
* hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed.
If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.)
I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available.
It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
If we presume the attackers had access to the system handling authorization, then the attackers introducing code to ship passwords offsite as users log in isn't really a stretch of the imagination.
That's true and great point. Let's play devil. You can't really be sure if Google engineer is sloppy and logging username and password on entry and then the SRE reading the log sees everything. I am not sure if their build system has plugin to detect such big red flag. Stacktrace is another place with potential leak of credentials. All of these are reasons when someone claims software is open source and auditable but they run the service themselves, it's really important to note you can't audit the actual server. They can log your username and password behind the scene while the client side appears to be 100% the same as the one on the server side.
What you describe is not rare, can be done with cross-site scripting. How it happens depends on the injection method (perhaps SQL injection).
12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better.
I wouldn't consider an 8 char password secure, no matter what the entropy is.
Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked).
As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps).
The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession.
They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me.
We would've appealed but there was no point as the sentence was essentially nothing.
Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.
The other side to that is that if you appeal, and lose, you can come off worse. So even though a criminal record is "something", depending where the lawyers feel your chances are, it may be best going along with it.
This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it.
Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985
I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.
Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.
Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.
I know we left them 4 years ago because how they implement bandwidth caps on private IPs, which a few years later another company also had the same problem with and nicely wrote it up on their blog: https://docraptor.com/blog/gone-in-60-seconds-how-we-moved-f...
Linode's attempt to keep us (spending $5k a month at the time, but we've grown substantially with AWS now. Linode were at $22m in 2011, so we'd have been 0.3% of their total revenue): "We will certainly be sorry to see you go."
That may just be because they were fed up of us after we opened 27 support tickets about the same networking issue over the course of 11 months though.
When I worked there support was under intense pressure to respond to tickets quickly.
Not nearly as much pressure to respond correctly.
To clarify and reiterate--employees who responded to tickets quickly were praised, even though there response contained half truths or outright falsehoods. If someone took 15-60 minutes ( or more ) to deep dive into an issue for a real fix for a customer, they were shamed and got a talking to.
This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack?
Edit:
1.The bitcoin hacks, March 2012
2.HTP hack, April 15, 2013 (CF exploit)
3.Second HTP hack April 16, 2013 (Another CF exploit)
4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014
5.This hack
I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.
Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
>we're still doing routine audits of the existing stuff.
This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
Today's attack is some kind of unspecified or unknown breach involving Linode manager.
I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues.
Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
I know of at least one other of the alleged HTP members who is not only not arrested, but still actively involved in the information security community.
Then there are other personalities that went dark, whom are presumably also not arrested.
Unless Thomas Asaro can name names, that was a bluff.
A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.
I'd avoid VPS providers in general, but AWS is on a whole different level than linode. They actually understand what they're doing well enough to do live xen patching etc.
But yeah, people get hacked through their hosts all the time. Best approach is colo with minimum access for the dc staff.
That's been my recommend for a long time. Plus, I liked obfuscating with unusual CPU choices and network guards (esp for protocol layers). Worked wonders with about no effort outside setting up guards. Opponents throw so much x86 shellcode at your Alpha, etc boxes while never quite getting stuff to run.
"Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.
"A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.
"This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.
"The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.
"You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.
"The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.
"We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff.
Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.
Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks.
Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to linode made this rather front and foremost in our minds) - so nothing that's happened at AWS has ever been anything more than an inconvenience - we've managed five nines since the move - before, we managed one.
Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.
Yeah... our issues mostly arose from the fact that at the time, they were advertising 1Gbps node interconnect - which actually turned out to be 1Gbps HOST interconnect, with the nodes actually throttled down to 50Mbps. We use memcached extensively, and this was absolutely crippling for performance. It didn't help that they furiously denied that they were throttling until we demonstrated it beyond all doubt.
They did obligingly increase these caps when we begged them to do so, but at that point the writing was on the wall, and we kept on bumping into other weird and wonderful limitations and issues, such as the fact that someone running an intense job on the same host could bring our VPS's to an absolute crawl.
It really is a shame, as we desperately wanted to make it work, as we liked Chris's hands on approach (very much like ours), but ultimately our confidence in them was so eroded by the point that things started going genuinely wrong on their end that we had no choice but to leave.
As I said, we kept random small single-server stuff (wordpress sites mostly) there, as if you're not dealing with their networking, performance is generally OK - but the network limitations were an absolute clincher for us, and it was at one point literally every day that we'd find that one of their switches had broken, or we couldn't ARP IP's for no apparent reason, etc. etc.
ATL was much worse than Dallas in this go around. You did not get 3 9s in the last month if your service was located solely in ATL. They were completely null routed for the majority of 2 days.
Strange how this works. I moved everything to cloud providers because of availability issues. Now a few years later, I'm moving everything back to dedicated hardware in several different datacenters because of availability issues. Thanks, Docker!! <3
We are leasing hardware, and we are leasing a lot of OVH (in addition to a number of smaller providers). OVH does a good job, and if you build your infrastructure to load-balance around failure (which does not happen often), you'll hardly notice it. Often we get an email from OVH telling us that support techs have been automatically dispatched to a node which we hadn't even alerted about yet. Docker is basically a miracle drug for this kind of thing. Our hardware costs are less than half of what they were with EC2. Huge savings.
At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...
Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others.
The source of this is typically two fold:
- For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety.
- For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers.
The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.
Github was also targeted by China to try to get them to delete certain repos that were unfriendly to China. Some people also saw it as showing off their power.
Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.
Be grateful they posted something to Slashdot at all. Back in 2013 they didn't do that for DAYS after the event happened and someone else posted what happened.
(Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.
Not to throw fuel on the fire but grumbling ex-employees aren't surprised. I've heard a few stories about not investing in infrastructure or even bug fixes. "They wont even look at bug fixes with major support costs." I know it's hard to triage what to do when running a company but when employees are proud of their product a breach like this isn't surprising.
I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here.
Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.
In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.
In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015.
We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.
In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it.
To clarify, we did not leave the users enabled in the Linode Manager. When you delete a Linode user, Linode shadow deletes them in the DB by setting an inactive date on the row.
The honeypot user would not have been able to access the account had the credentials been valid, but based on the information given by Linode we did see someone attempt to log in as that user only once around the time of the compromise.
That's incredibly poor handling by linode, it also quite possibly resulted in many other systems being compromised via the same route or a similar one in the meantime.
Saw your tweet (https://twitter.com/theckman/status/684484772316360705) that linked to this post. Did a quick search to get your technical background and your LinkedIn profile states you used to work for Linode? I think it's important to share that info when you're telling your side of the incident. Your past relationship, if you left on bad terms, could play a role in your motivation to post.
I can absolutely understand your concern. I originally had it in my post, but removed it because I was worried it would detract from the details of our compromise at PagerDuty.
I worked at Linode for just under three years, and worked on quite a few different things there. I started on support and moved on to a development role (including writing ColdFusion). I left Linode on good terms. California is much more enticing than NJ, so I wanted to relocate. Plus I was interested in doing more of an Ops role, instead of working including customer-facing web applications. I'm still enjoying it. :)
I think there are lessons that can be learned whenever a company has some sort of security incident. This is especially true if they are willing to publicly disclose details of the incident. We've wanted to provide what limited information we had, but wanted to wait until we had confirmation that Linode was the vector.
While there is some relief in finally determining what we believe to be the vector of our attack, it's very unfortunate that Linode engineers are dealing with the fallout right now.
so you're saying you wrote the code that was responsible for the breach(es)? It also sounds like you knew about these issues before you left Linode but didn't point them out while you were there?
Where are you reading this? When he says he was writing ColdFusion, he's referring to the framework/programming language. He's not saying he wrote every single line of Linode's management interface.
Isn't this like the third Linode hack? Back in 2012 one of their cust tools was compromised in order to steal bitcoins. In 2013, cc# and password hashes were compromised, Linode denied it until the hackers showed proof, then did a piss poor job of handling it afterwards.
Why were people still using Linode after their poor handling in the 2013 hack?
I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking.
1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very least.
2) Being able to blame Linode for your own data breach is a fantastically easy (although lazy) way to pacify customers about the fact that their personal data was just pilfered by someone on your watch.
All that being said, what have you presented that can be proven? All that can be proven is that you're an ex Linode employee. Everything else is hot air that we're all meant to take your word. Tons of appeal to authority in your explanations. You keep invoking some mysterious third party "expert" security group who conveniently agrees with everything your own company "discovered." If you were actually confident in your own abilities and that of your team members, there wouldn't be an immediate appeal following every attempted assertion.
Plus, even if you really did hire someone, what company isn't going to just say "yes" and agree to whatever PR campaign their customer is saying while dumping wheelbarrows of cash into their pockets? Frankly I don't believe you, and I find your consistent drumming against Linode to be highly suspicious in the wake of these attacks. You're not involved, are you?
The Linode post you're referring to is just saying that they expired everyone's passwords. That's not admitting anything, especially not admitting anything about a separate incident from a year ago. What lawyer would ever take this and say "okay, you can legally publicly blame linode now?" No lawyer worth his salt. In other words, you're full of shit. Your story if full of holes, tells, and I think you should stop posting so much garbage before you're on the receiving end of a lawsuit or are considered a suspect.
Are you Jesse Nicholson? Is Linode paying you to write this? This reads like a Linode executive wrote it. That's the kind of thing you should disclose. Your only two HN comments are regarding Linode; pretty suspicious. You also commented on their blog post:
TechnikEmpire January 6th, 2016 at 10:28 pm. It's hilarious watching all of these armchair experts criticize Linode for the actions of another.
PagerDuty and WP Engine were both compromised 'inexplicably' during the same timeframe at the same hosting provider. Seems pretty self explanatory. Linode didn't disclose their "security firm" so why should PagerDuty? Linode couldn't explain how accounts were accessed and it isn't the first time! Linode is hacked once a year; it's a feature. They need to get their shit together and stop pretending security is a game.
I enabled two factor on my Linode account recently and saw this worrying copy:
> If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account.
> Should you need us to disable your Two-Factor Authentication, the following information is required:
> An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card.
> An image of the front and back of the matching government-issued photo ID.
There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.
> There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
Sadly, this is quite common, especially with organizations that provide phone support[1].
Agreed - especially having read this earlier (not being able to fix it at the time), still not getting the email (like 6 hours have gone by) and I'm resetting because I should and still no communication from Linode.
Concerning since I had to reset my password and regen my 2fa using only my old password...
There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.
That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere.
I just don't think the performance is good enough for pure web hosting. Much slower/$ than Linode and DO.
Their uptime and infrastructure is great, but you're totally on your own (e.g. if you're getting DDoS'ed). Also their traffic cost is not as easy to predict than DO/Linode/Vultr.
OVH is hard to beat when it comes to raw performance/$. Hardware and network has been quite reliable for me (using SoYouStart and OVH), and their DDoS filter is excellent. I'm not really familiar with their cloud/VPS offering.
That being said, they only have one data center in NA at the moment, and they recently suffered a rather nasty network outage in that data center. Would definitely look into them if your target market is primarily EU, but if you're mostly NA-based, YMMV.
Blog is down. Maybe they should host it on WPEngine. (:
This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.
Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.
With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?
VPS providers that are typically pitched as Linode Alternatives. Not 1:1 equivalents, but may work depending on your needs.
Vultr, Ramnode, Wable, iWStack
Or, depending on the number of VPS's you have, you might like:
- Aliyun, the cloud service from Alibaba
- A dedicated server from OVH, or their mid-tier brand, SoYouStart. This is my personal favorite. They have real DDOS protection, Data centers in North America and Europe, reasonable web interfaces, lots of available IPV4 space, and DIRT CHEAP prices. Run proxmox as the distro, and you get a decent interface to create and manage VPS instances.
The most important piece would be to try and split instances across at least two of these providers so that you have some fast recourse if something goes wrong. For the things I'm running, doing a nightly rsync of the data from one provider to another suffices as reasonable insurance.
This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?
so... I completely agree that shared secret authentication is a bad idea, and I use public key authentication wherever I can (password auth is disabled for ssh on every server I control; I do everything with ssh public keys.)
However, I've yet to set a public key authentication scheme that users would find acceptable for web applications. Do you really expect all users to setup x.509 auth in the browser?
What is your public key solution to authenticate the web-applications that customers demand?
OK, I believe you, though I don't know any of the details. It's an important misunderstanding to prevent, though. In this case, including enough of the details to make the comment substantive would probably have been enough to clarify your intent as well.
you could have made the comment more substantive in a way that was clearly not a personal attack, and that would have solved the problem.
This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.
Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.
Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.
I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
If the Linode Manager database only stores the hash, how would they know the password?
How did you find out about the illicit login?
If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
If the Linode Manager database only stores the hash, how would they know the password?
Weak password? Weak hashing algorithm?
> If the Linode Manager database only stores the hash, how would they know the password?
Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently.
The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters).
This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts.
But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
> This only works if the input password has low entropy.
If you're generating every single possible password up to e.g. 8 characters the password's quality doesn't matter, only the length does.
A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present.
But the most effective "rainbow table"-like table is a look-up table with the followings:
* leaked password in plaintext, associate with email and any ID (forum username??)
* hash all of those passwords without salt
* hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed.
If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.)
I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available.
It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
If we presume the attackers had access to the system handling authorization, then the attackers introducing code to ship passwords offsite as users log in isn't really a stretch of the imagination.
That's true and great point. Let's play devil. You can't really be sure if Google engineer is sloppy and logging username and password on entry and then the SRE reading the log sees everything. I am not sure if their build system has plugin to detect such big red flag. Stacktrace is another place with potential leak of credentials. All of these are reasons when someone claims software is open source and auditable but they run the service themselves, it's really important to note you can't audit the actual server. They can log your username and password behind the scene while the client side appears to be 100% the same as the one on the server side.
What you describe is not rare, can be done with cross-site scripting. How it happens depends on the injection method (perhaps SQL injection).
12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better.
I wouldn't consider an 8 char password secure, no matter what the entropy is.
Yes length matters. That's why pretty much all "randomly generated" passwords are long. Mine are 20 characters.
They don't need to know the password with db write access.
In fact, depending on how the sessions are managed the attacker might just need read access to log in without a password.
Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked).
< naqod> alexf: any legitimacy to this https://news.ycombinator.com/item?id=10845619 ?
<@alexf> naqod: again I'm not in ops so I don't have the deets, but my gut reaction is to say No effing Way
You'd imagine that by now they wouldn't be so quick to deny this stuff.
As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
Double jeopardy, I was previously convicted of 50700 counts of unauthorized access to various coldfusion sites. Among those was linode.
This one?
http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-te...
Yeah.
How exactly did you get caught? opsec/comsec violation?
Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps).
The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession.
They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me.
We would've appealed but there was no point as the sentence was essentially nothing.
> We would've appealed but there was no point as the sentence was essentially nothing.
But you do end up with a record, which is not 'essentially nothing'.
I suppose it depends on what that means for a minor in Finland.
Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.
How are you in a position where looking for a job in the future is not really necessary?
Bitcoin.
That's an awful lot of faith in virtual currency and its speculative nature to be betting the next 50-60 years of your life.
I can't even tell you how many pen testers I know with criminal records. Somewhere, someone will hire him to do security.
Who is to say xe didn't buy a bunch of it at $1 and sell it off, making millions, when it hit $200-$300?
Diversification is pretty important if you want to keep any kind of currency for 50-60 years.
Talented hackers are not exactly unemployable.
The other side to that is that if you appeal, and lose, you can come off worse. So even though a criminal record is "something", depending where the lawyers feel your chances are, it may be best going along with it.
This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it.
Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985
I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.
Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.
I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.
Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.
And of course your concern is pure as new fallen snow and you have no other motives.
They were expecting to do $60m in 2014 - http://www.sramanamitra.com/2014/07/11/bootstrapping-a-web-h....
I know we left them 4 years ago because how they implement bandwidth caps on private IPs, which a few years later another company also had the same problem with and nicely wrote it up on their blog: https://docraptor.com/blog/gone-in-60-seconds-how-we-moved-f...
Linode's attempt to keep us (spending $5k a month at the time, but we've grown substantially with AWS now. Linode were at $22m in 2011, so we'd have been 0.3% of their total revenue): "We will certainly be sorry to see you go."
That may just be because they were fed up of us after we opened 27 support tickets about the same networking issue over the course of 11 months though.
When I worked there support was under intense pressure to respond to tickets quickly.
Not nearly as much pressure to respond correctly.
To clarify and reiterate--employees who responded to tickets quickly were praised, even though there response contained half truths or outright falsehoods. If someone took 15-60 minutes ( or more ) to deep dive into an issue for a real fix for a customer, they were shamed and got a talking to.
This is at least the fifth time I can count that linode has been hacked, really? Maybe it's time to ditch that coldfusion stack?
Edit:
1.The bitcoin hacks, March 2012
2.HTP hack, April 15, 2013 (CF exploit)
3.Second HTP hack April 16, 2013 (Another CF exploit)
4.MySQL server that allowed anonymous logins (?!?!) January 19, 2014
5.This hack
I'm not counting their domain name and various other parts of their infrastructure provided by third parties being compromised, but if I was the list would be significantly longer.
>Maybe it's time to ditch that coldfusion stack?
[Linode developer here]
Been working on (re)writing things in Python since I started. It takes a while, but I think everyone recognizes that the CF codebase is difficult to maintain. The good news is that significant progress is being made and we're still doing routine audits of the existing stuff.
>we're still doing routine audits of the existing stuff.
This doesn't work, auditing coldfusion code is impossible without auditing the entire platform. The whole platform is so full of bugs and strange behaviour that it's actually impossible to produce secure coldfusion code.
I can't really deny that CF is that bad, but it'd be irresponsible to just let the codebase rust as we rewrite it - and we are rewriting it.
When did you start rewriting it? It doesn't take years to replace this stuff.
I was hired in July and have been driving most of this effort and we're shipping rewritten versions of some parts of our infra soon™
That sounds good, at least, thanks for letting us know.
I was just thinking the same thing. I've been a customer for >10 years but this is getting ridiculous.
First 2013 attack was apparently exacerbated by cleartext password storage for LISH (their management shell) and API tokens https://marco.org/2013/04/16/linode-hacked
The 2012 Bitcoin attack involved a breach of Linode's customer service portal http://arstechnica.com/business/2012/03/bitcoins-worth-22800...
Today's attack is some kind of unspecified or unknown breach involving Linode manager.
I guess the obvious commonality here is that all the attacks target the "soft" Linode layers AROUND managing deploys of Linux and Xen/KVM/UML rather than the "hard" targets of those widely used systems. This also happens to be the layer where Linode should be adding value (as opposed to the cheaper VPS providers out there) and I think it's increasingly troublesome that they continue to have such severe security issues.
Is this company (CEO - Christopher Aker) not investing in security staff, security training, best practices etc, or are they investing tons and just getting breached because they host so many sites? Unclear. But it's easy to imagine it's the former, from the outside, given all these incidents.
Actually the 2013 hack was caused by linode running blatantly misconfigured CF installations, like doing stuff that the manual has big warnings about.
For those not in the know, ryanlol was one of the people on the team involved in the 2013 hacks.
Really? Thomas Asaro told us they were all in jail.
Nobody went to jail, I'm the only person being prosecuted. I won't go to jail.
I know of at least one other of the alleged HTP members who is not only not arrested, but still actively involved in the information security community.
Then there are other personalities that went dark, whom are presumably also not arrested.
Unless Thomas Asaro can name names, that was a bluff.
A customer warned Linode team about the exposed CF folder. CEO aggressively shrugged it off. "That doesn't matter, it's nothing, that's a non issue." Dev who was a bit of a suck up parroted the same telling support to shut up about it. This was six months before HTP happened.
We were aware of it for probably an year before anyone bothered to spend 10 minutes looking at coldfusion source. That's all the time it took.
V interesting. Do any of the other VPS providers strike you as more secure alternatives?
I'd avoid VPS providers in general, but AWS is on a whole different level than linode. They actually understand what they're doing well enough to do live xen patching etc.
But yeah, people get hacked through their hosts all the time. Best approach is colo with minimum access for the dc staff.
That's been my recommend for a long time. Plus, I liked obfuscating with unusual CPU choices and network guards (esp for protocol layers). Worked wonders with about no effort outside setting up guards. Opponents throw so much x86 shellcode at your Alpha, etc boxes while never quite getting stuff to run.
The best thing about the 2013 hack was that news of it was on Slashdot days before it was mentioned to customers.
I've been consistently saying this for years. Linode is a joke and you would be crazy to use them for anything other than toy/non-critical use cases.
Has anyone got more information on this? Various Google searches keep pointing me at the other four hacks.
https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...
They try to downplay it pretty hard.
Thanks for this link. Although it talks about:
So I'm not sure "anonymous login" would be an entirely accurate description.
The server would in fact accept any credentials as it had been started with --skip-grant-tables. Tested it myself.
The page was slashdotted for me, here's a copy:
"Effective immediately, Linode Manager passwords have been expired. You will be prompted to set a new password on your next login. We regret this inconvenience, however this is a necessary precaution.
"A security investigation into the unauthorized login of three accounts has led us to the discovery of two Linode.com user credentials on an external machine. This implies user credentials could have been read from our database, either offline or on, at some point. The user table contains usernames, email addresses, securely hashed passwords and encrypted two-factor seeds. The resetting of your password will invalidate the old credentials.
"This may have contributed to the unauthorized access of the three Linode customer accounts mentioned above, which were logged into via manager.linode.com. The affected customers were notified immediately. We have found no other evidence of access to Linode infrastructure, including host machines and virtual machine data.
"The entire Linode team has been working around the clock to address both this issue and the ongoing DDoS attacks. We’ve retained a well-known third-party security firm to aid in our investigation. Multiple Federal law enforcement authorities are also investigating and have cases open for both issues. When the thorough investigation is complete, we will share an update on the findings.
"You may be wondering if the same person or group is behind these malicious acts. We are wondering the same thing. At this point we have no information about who is behind either issue. We have not been contacted by anyone taking accountability or making demands. The acts may be related and they may not be.
"The security of your data, the functionality of your servers, and your confidence in Linode are extremely important to all of us. While we feel victimized ourselves, we understand it is our responsibility, and our privilege as your host, to provide the best possible security and service. You can help further enhance the security of your account by always using strong passwords, enabling two-factor authentication, and never using the same password at multiple services.
"We sincerely apologize for the recent disruptions in your Linode service. Thank you for your patience, understanding and ongoing trust in Linode."
Sadly I trust them as far as I can throw them, and we moved everything important from them to AWS a few years back. We had left a few static sites there but after the shenanigans over the holidays, we're moving our remaining stuff.
Sad. We spent >$10k/month with them for a while, before their shit started falling apart. They didn't appear to care at all when we left, so I suppose they have an awful lot of large customers who just don't mind their stack disintegrating beneath them every sixty days at most.
So, what happened during the AWS outage this past Fall? Or did you restructure to provide failover beyond just moving platforms?
Well, at linode you can't have a structure that is immune to failover, as they have single points of failure within their infrastructure, apart from anything else - all their London kit for instance lives in Telehouse East, in a few adjacent racks.
Once we'd done the initial up sticks and move to AWS, our first priority was to use their redundancy and failover to the fullest (six months of sleepless nights due to linode made this rather front and foremost in our minds) - so nothing that's happened at AWS has ever been anything more than an inconvenience - we've managed five nines since the move - before, we managed one.
Only one 9? Even through this crap during the holidays I've managed 3 9's on my service hosted on several servers in Linode Dallas (the most hard-hit region in this DDoS attack). I would have moved to AWS by now if Linode didn't have such cheaper bandwidth.
Yeah... our issues mostly arose from the fact that at the time, they were advertising 1Gbps node interconnect - which actually turned out to be 1Gbps HOST interconnect, with the nodes actually throttled down to 50Mbps. We use memcached extensively, and this was absolutely crippling for performance. It didn't help that they furiously denied that they were throttling until we demonstrated it beyond all doubt.
They did obligingly increase these caps when we begged them to do so, but at that point the writing was on the wall, and we kept on bumping into other weird and wonderful limitations and issues, such as the fact that someone running an intense job on the same host could bring our VPS's to an absolute crawl.
It really is a shame, as we desperately wanted to make it work, as we liked Chris's hands on approach (very much like ours), but ultimately our confidence in them was so eroded by the point that things started going genuinely wrong on their end that we had no choice but to leave.
As I said, we kept random small single-server stuff (wordpress sites mostly) there, as if you're not dealing with their networking, performance is generally OK - but the network limitations were an absolute clincher for us, and it was at one point literally every day that we'd find that one of their switches had broken, or we couldn't ARP IP's for no apparent reason, etc. etc.
ATL was much worse than Dallas in this go around. You did not get 3 9s in the last month if your service was located solely in ATL. They were completely null routed for the majority of 2 days.
I don't think Telehouse East is correct. They're with Telecity, and in their PowerGate facility, i.e. out in Acton: http://www.telecitygroup.com/our-company/news/2010/linode-ex...
Unless they've moved?
Ah, that may be it - was a good four years ago, so I guess they were there. Telesomething!
Strange how this works. I moved everything to cloud providers because of availability issues. Now a few years later, I'm moving everything back to dedicated hardware in several different datacenters because of availability issues. Thanks, Docker!! <3
Are you buying your own hardware and colocating, or leasing hardware? If the latter, which provider(s) are you using?
We are leasing hardware, and we are leasing a lot of OVH (in addition to a number of smaller providers). OVH does a good job, and if you build your infrastructure to load-balance around failure (which does not happen often), you'll hardly notice it. Often we get an email from OVH telling us that support techs have been automatically dispatched to a node which we hadn't even alerted about yet. Docker is basically a miracle drug for this kind of thing. Our hardware costs are less than half of what they were with EC2. Huge savings.
Heh, and here "often" means "often during failures". ;)
We've been watching the unfolding situation closely as well and have decided to migrate/duplicate on AWS. Glad it worked our for you.
At what point should Linode start thinking damage control? DDoS, breaches, lack of transparency...
It's ironic how most of the "yeah me too" posts end up being AWS related.
> The page was slashdotted for me
Likely the server hosting the blog is being DDoSed too. I feel bad for them, but they should know a bit better already...
Looks like the reason their blog is down is because... it's now being targeted by a DoS:
http://status.linode.com/incidents/kldhjpjnfnkj
Attacking a blog talking about the hack? It sure seems that someone has a grudge against Linode. :-/
At this point I'm starting to wonder whether this isn't a competitor putting their investors money to work. It's otherwise utterly bizzare that someone would be so obsessive in damaging Linode. I really hope they make the details of the investigation public...
It makes more sense than you might imagine...
Linode, Github, Stackoverflow, Imgur, they've all been targeted. But what do they have in common? In a word: popularity. The core reason these sites are targeted is because it is impressive to others.
The source of this is typically two fold:
- For the lolz. Someone with a botnet just wants to show off, taking down something known gives them more notoriety.
- For a sales pitch. Someone has botnet capacity that they want to sell, and "I took down Linode" is a great way of demoing that to potential buyers.
The first one is more common when someone finds a new method of traffic multiplication and just wants to show it off (i.e. they trick a third party into DoS/DDoSing a target). The second one is legitimate criminal enterprises.
Github was also targeted by China to try to get them to delete certain repos that were unfriendly to China. Some people also saw it as showing off their power.
Based on the number of negative reviews on glassdoor, it could also be a former employee.
For those not keeping track, that's the third major breach I've heard of where user data was exposed.
Previous Incidents:
April, 2013: https://blog.linode.com/2013/04/16/security-incident-update/
January 2014: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-... (they mention the breach almost in passing...the swatting is unrelated)
I really want Linode to do well, but this may be it for me.
Sure wish they had sent out an email notification to users instead of a slashdotted blog post. Now the question is how long can Linode stand in the face of these sorts of hacks and network attacks in the face of stiff VPS competition.
Be grateful they posted something to Slashdot at all. Back in 2013 they didn't do that for DAYS after the event happened and someone else posted what happened.
(Linode Employee) Already got it covered, we are sending out an email to everyone in batches, but pushed out the blog first since it can be seen by everyone right away.
I'm still waiting for mine (6 hours since this was posted to HN).
I'm still waiting on an email too.
How come, still, no one I know, including myself, has received this email?
Comments from Linode employees, Linode customers who were hacked, and a guy who hacked Linode (different incident)
I love HN.
Not to throw fuel on the fire but grumbling ex-employees aren't surprised. I've heard a few stories about not investing in infrastructure or even bug fixes. "They wont even look at bug fixes with major support costs." I know it's hard to triage what to do when running a company but when employees are proud of their product a breach like this isn't surprising.
I'm glad to see that this information has now been publicly disclosed. In July 2015, we suffered a compromise at PagerDuty via the Linode Manager. I hope that we can provide a bit more of an official in-depth post-mortem of our compromise, but I'd be happy to disclose some of the details here.
Using the access gained within the Linode Manager, the attacker reset the root password on a few systems, and used Lish to gain root access. We were alerted to this activity and fully revoked the attacker's access within 60 minutes of the first node being compromised. Working with Linode support, we discovered which user account was being used and completely deactivated the user. We also isolated the VMs, and performed forensics on read-only copies of their disk images.
In our situation the attacker knew one of our user's passwords and MFA secret. This allowed them to provide valid authentication credentials for an account in the Linode Manager. It's worth noting that all of our active user accounts had two-factor authentication enabled. An interesting data point was that the user who had their account compromised was no longer in possession of the MFA secret themselves. Their cell phone had been reset (thus deleting all data) 8 months prior. The user could not log in to the Linode Manager if they wanted, so it was our determination that the key could not have been obtained from the user and was more likely on Linode's side.
We also have evidence from access logs provided by Linode that the attackers tried to authenticate as an ex-employee, whose username ONLY existed in the Linode database. It was absolutely unique and was not used elsewhere by the employee making the username an accidental honeypot. This was another piece of data supporting that Linode was the source of our compromise.
We immediately reached out to them not only to inform them of their compromise, but to assist them in investigating it. We were confident that the Linode database had been breached, and that the secret key used to encrypt information in the database had been compromised as well.
In addition to reaching out to Linode, we also worked with a third-party security firm to audit our work done during the incident. Likewise, around the same time we reached out to law enforcement to assist in investigating the attack. I believe our public disclosure includes this information[1]. This was in the middle of July 2015.
We did not get confirmation in July that there was a breach of the Linode Manager or any associated credentials.
In the end, we migrated away from Linode because of this breach (even before it was publicly disclosed) in Aug 2015. We also never were able to confidently disclose that Linode was the vector due to lack of confirmation from their end. While all of us who responded to the incident were confident they were the source, we now thankfully have the data to confirm it.
[1] https://www.pagerduty.com/blog/july-2015-security-announceme...
Yikes...This comment has me seriously thinking about moving my VPS elsewhere. Thanks for spreading this information, this is definitely good to know.
Nice writeup.
Keeping logins of ex-employees on 3rd party systems is a no-no though I admit full removal might pose some hurdles.
To clarify, we did not leave the users enabled in the Linode Manager. When you delete a Linode user, Linode shadow deletes them in the DB by setting an inactive date on the row.
The honeypot user would not have been able to access the account had the credentials been valid, but based on the information given by Linode we did see someone attempt to log in as that user only once around the time of the compromise.
Thanks for the clarification. Makes more sense now.
That's incredibly poor handling by linode, it also quite possibly resulted in many other systems being compromised via the same route or a similar one in the meantime.
I had almost exactly the same situation in 2013.
I honestly don't understand why anyone would be stupid enough to use Linode.
They continue to (a) have incidents and (b) fail to disclose them in a timely and transparent manner.
Saw your tweet (https://twitter.com/theckman/status/684484772316360705) that linked to this post. Did a quick search to get your technical background and your LinkedIn profile states you used to work for Linode? I think it's important to share that info when you're telling your side of the incident. Your past relationship, if you left on bad terms, could play a role in your motivation to post.
I can absolutely understand your concern. I originally had it in my post, but removed it because I was worried it would detract from the details of our compromise at PagerDuty.
I worked at Linode for just under three years, and worked on quite a few different things there. I started on support and moved on to a development role (including writing ColdFusion). I left Linode on good terms. California is much more enticing than NJ, so I wanted to relocate. Plus I was interested in doing more of an Ops role, instead of working including customer-facing web applications. I'm still enjoying it. :)
I think there are lessons that can be learned whenever a company has some sort of security incident. This is especially true if they are willing to publicly disclose details of the incident. We've wanted to provide what limited information we had, but wanted to wait until we had confirmation that Linode was the vector.
While there is some relief in finally determining what we believe to be the vector of our attack, it's very unfortunate that Linode engineers are dealing with the fallout right now.
so you're saying you wrote the code that was responsible for the breach(es)? It also sounds like you knew about these issues before you left Linode but didn't point them out while you were there?
Where are you reading this? When he says he was writing ColdFusion, he's referring to the framework/programming language. He's not saying he wrote every single line of Linode's management interface.
Isn't this like the third Linode hack? Back in 2012 one of their cust tools was compromised in order to steal bitcoins. In 2013, cc# and password hashes were compromised, Linode denied it until the hackers showed proof, then did a piss poor job of handling it afterwards.
Why were people still using Linode after their poor handling in the 2013 hack?
I'm curious what kind of losses you've seen since your own data breach. I'm sorry but you have far too many plausible ulterior motives for taking the position that you are taking.
1) As someone else pointed out, you're an ex-employee of Linode. You went out of your way to hide this fact. I'll refrain from listing all of the very obvious reasons why your word on Linode should be taken with a grain of salt at the very least.
2) Being able to blame Linode for your own data breach is a fantastically easy (although lazy) way to pacify customers about the fact that their personal data was just pilfered by someone on your watch.
All that being said, what have you presented that can be proven? All that can be proven is that you're an ex Linode employee. Everything else is hot air that we're all meant to take your word. Tons of appeal to authority in your explanations. You keep invoking some mysterious third party "expert" security group who conveniently agrees with everything your own company "discovered." If you were actually confident in your own abilities and that of your team members, there wouldn't be an immediate appeal following every attempted assertion.
Plus, even if you really did hire someone, what company isn't going to just say "yes" and agree to whatever PR campaign their customer is saying while dumping wheelbarrows of cash into their pockets? Frankly I don't believe you, and I find your consistent drumming against Linode to be highly suspicious in the wake of these attacks. You're not involved, are you?
The Linode post you're referring to is just saying that they expired everyone's passwords. That's not admitting anything, especially not admitting anything about a separate incident from a year ago. What lawyer would ever take this and say "okay, you can legally publicly blame linode now?" No lawyer worth his salt. In other words, you're full of shit. Your story if full of holes, tells, and I think you should stop posting so much garbage before you're on the receiving end of a lawsuit or are considered a suspect.
Are you Jesse Nicholson? Is Linode paying you to write this? This reads like a Linode executive wrote it. That's the kind of thing you should disclose. Your only two HN comments are regarding Linode; pretty suspicious. You also commented on their blog post:
TechnikEmpire January 6th, 2016 at 10:28 pm. It's hilarious watching all of these armchair experts criticize Linode for the actions of another.
PagerDuty and WP Engine were both compromised 'inexplicably' during the same timeframe at the same hosting provider. Seems pretty self explanatory. Linode didn't disclose their "security firm" so why should PagerDuty? Linode couldn't explain how accounts were accessed and it isn't the first time! Linode is hacked once a year; it's a feature. They need to get their shit together and stop pretending security is a game.
I enabled two factor on my Linode account recently and saw this worrying copy:
> If you lose your token and get locked out of the Linode Manager, email support@linode.com to regain access to your account.
> Should you need us to disable your Two-Factor Authentication, the following information is required:
> An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card. > An image of the front and back of the matching government-issued photo ID.
There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
EDIT: I also find it odd that you have to manually generate a scratch code, and they don't automatically generate it for you. Again, all of the other 2FA setups I've gone through have done this.
> There doesn't seem to be a way to say "I have my big boy pants on, don't let anyone in under any circumstances". This is the first 2FA setup I've seen that still allows bypass by contacting support.
Sadly, this is quite common, especially with organizations that provide phone support[1].
[1]: http://krebsonsecurity.com/2015/12/2016-reality-lazy-authent...
Why the hell have they not emailed their customers about this!
This is not the kind of thing I want to learn from HN.
They're doing so as we speak. It takes a while to send 400,000 mails if you actually want them to be delivered to inboxes.
400k emails isn't that many though. :(
I haven't received any email. It's over 12 hours since the notice went up on status.linode.com and the Twitter account.
Agreed - especially having read this earlier (not being able to fix it at the time), still not getting the email (like 6 hours have gone by) and I'm resetting because I should and still no communication from Linode.
Concerning since I had to reset my password and regen my 2fa using only my old password...
Not the first time they poorly handled a security issue: https://blog.linode.com/2014/01/19/an-old-system-and-a-swat-...
There are also a number of reviews on Glassdoor where supposed Linode employees say they were asked by management to lie to customers about security breaches.
We were.
That's it. I'm done with Linode. I will be keeping one server on Linode, but my new startup where I need several load balancers, Web servers, DB replication will be hosted elsewhere.
Do you guys know any good providers? Not AWS.
Can I ask why not AWS?
I just don't think the performance is good enough for pure web hosting. Much slower/$ than Linode and DO.
Their uptime and infrastructure is great, but you're totally on your own (e.g. if you're getting DDoS'ed). Also their traffic cost is not as easy to predict than DO/Linode/Vultr.
OVH is hard to beat when it comes to raw performance/$. Hardware and network has been quite reliable for me (using SoYouStart and OVH), and their DDoS filter is excellent. I'm not really familiar with their cloud/VPS offering.
That being said, they only have one data center in NA at the moment, and they recently suffered a rather nasty network outage in that data center. Would definitely look into them if your target market is primarily EU, but if you're mostly NA-based, YMMV.
Don't use Linode.
Don't lecture me.
Blog is down. Maybe they should host it on WPEngine. (:
This sort of thing almost became background noise when I worked there. I can say that things improved somewhat after the HTP compromise, at least for awhile.
Wow, these guys can't get a break.
It's not as if this is just bad luck...
You make your own luck.
If they can't be bothered to invest in their tools and processes then this is the sort of thing that happens.
Yeah it's not like ColdFusion powered web sites enabling billions of dollars in commerce and what with all other web development tools being free of security vulnerabilities and all.
The merits of ColdFusion are irrelevant. The fact of the matter is that their software has continuously been breached.
They've been using a CF stack that is fundamentally broken from the foundation up for years and was aware of it.
Its part of why they hired someone in July to rebuild it in python.
You signed an NDA and have done a full analysis of their code? Cool.
Go look through how often CF has zero days compared to competing platforms.
You don't really need to sign NDAs to read through java classes.
With Linode's extended DDoS I have not been able to get into the Manager for a couple weeks. I'd really like to cancel my account with them (and they do keep billing) but I don't appear to have any tools short of a chargeback. Anyone else in the same boat?
You don't mention contacting support. Shouldn't that be your second avenue after trying to access the account dashboard?
I have always contacted support by opening a ticket in the Manager. The main website is also down. Is there an out-of-band way to reach them?
Phone? 855-454-6633 (+1-609-380-7100)
Why not call them on the phone? They have a toll-free number for support.
The main website is down for me and has been flakey but http://manage.linode.com hasn't been a problem for me over the last few weeks.
So, outside of the major cloud providers, what are the good alternatives?
I like Bytemark.
VPS providers that are typically pitched as Linode Alternatives. Not 1:1 equivalents, but may work depending on your needs.
Vultr, Ramnode, Wable, iWStack
Or, depending on the number of VPS's you have, you might like:
- Aliyun, the cloud service from Alibaba
- A dedicated server from OVH, or their mid-tier brand, SoYouStart. This is my personal favorite. They have real DDOS protection, Data centers in North America and Europe, reasonable web interfaces, lots of available IPV4 space, and DIRT CHEAP prices. Run proxmox as the distro, and you get a decent interface to create and manage VPS instances.
The most important piece would be to try and split instances across at least two of these providers so that you have some fast recourse if something goes wrong. For the things I'm running, doing a nightly rsync of the data from one provider to another suffices as reasonable insurance.
This is a perfect poster child for decentralized authentication. Centralized stores of passwords and secrets are just asking for trouble. When will we learn?
so... I completely agree that shared secret authentication is a bad idea, and I use public key authentication wherever I can (password auth is disabled for ssh on every server I control; I do everything with ssh public keys.)
However, I've yet to set a public key authentication scheme that users would find acceptable for web applications. Do you really expect all users to setup x.509 auth in the browser?
What is your public key solution to authenticate the web-applications that customers demand?
Glad they posted it. I just logged into one of my linode boxes that I don't use for very much but keep around, and it was rooted... doh.
Could be just a coincidence if you keep SSH open on standard port. SSH bots don't sleep.
The host was only accessible via private key or Linode's LISH shell. That's what seems most suspicious.
There is some minor evidence remaining in the .bash_history that is curious.
I'd be interested in a write-up about anything you find in that .bash_history or logs, would you consider writing one?
Here's the main bulk of the obviously bad things:
and now getting server errors & timeouts when I try to get to the manager to change my passwords.
this has been a hellaciously thorough attack.
And of course there's the ongoing ddos attack that is preventing me from reaching manager.linode.com...
The actual answer is much more sinister than that.
Which is kind of hilarious.
What is the actual answer?
Last I heard, compromising Bitcoin exchanges for lulz and mad profit. Allegedly.
So if the rumor is true, he wasn't technically lying.
I don't think that if all you have is 'allegedly' it is up to you to make such accusations here in public.
This is stuff he shares openly on IRC. In public channels.
Whether any of it is true or not remains to be seen. He's not ever given me any reason to believe he's lying, however.
That crosses into personal attack, which is not ok, even if it's just insinuated.
We detached this subthread from https://news.ycombinator.com/item?id=10847715 and marked it off-topic.
It's not a personal attack, it's something that ryanlol publicly claims in ##security on freenode FFS.
If it wasn't, I wouldn't have referenced it
From what I gather, he thinks law enforcement is hilarious.
OK, I believe you, though I don't know any of the details. It's an important misunderstanding to prevent, though. In this case, including enough of the details to make the comment substantive would probably have been enough to clarify your intent as well.
you could have made the comment more substantive in a way that was clearly not a personal attack, and that would have solved the problem.
Questionable.
This is Yet Another Reminder to use unique, unguessable, unmemorable passwords for all online services. It's a question of when, not if, any particular password database will be compromised. While 'password1' and 'this is my long unguessable password' and 'm4r1g0ld' and 'false lemur capacitor paperclip' will all eventually be guessed, 'sF0PSQMwK85fe9xanqJRm9nty9cJGHJsVmti' will never, ever be.
Why was this downvoted? It's true. The example password is 36 characters; a mixed-case-and-numbers alphabet has just over 6 bit per character; the example password is thus 217 bits. It's not going to be guessed.
Except passwords aren't even a relevant part of this breach. They're worth absolutely nothing compared to the ridiculous amounts of customer data linode controls.