I'm fairly confident that Linode has been compromised since July, if not earlier. PagerDuty moved off of Linode after an incident in July. We've been under strict gag orders from legal about that incident until today when Linode finally announced their compromise. Really, the only way I can see that this attacker could have gotten in the way they did (they logged into our Linode Manager account on the first try using a username that wasn't used anywhere but in Linode Manager, using valid 2fa and valid password) was if they had access to the Linode Manager database. I'm pretty sure that the recent WP Engine compromise was achieved through the same attack vector, too
This was my post from SomethingAwful, it was copied/pasted here. No idea who copied/pasted it.
Rather than writing up a huge reply with more info, I'll link to another reply with more details on this same thread since someone else already did the writing: https://news.ycombinator.com/item?id=10845985
Interestingly, while I know for a fact that you're correct. When someone asked about this on #linode the ops immediate reaction was to deny it (as it was every other time they got hacked).
< naqod> alexf: any legitimacy to this https://news.ycombinator.com/item?id=10845619 ?
<@alexf> naqod: again I'm not in ops so I don't have the deets, but my gut reaction is to say No effing Way
You'd imagine that by now they wouldn't be so quick to deny this stuff.
As a former Linode employee, I find all of your comments very interesting. But if it's true that you're being prosecuted, why are you incriminating yourself here?
All of the people involved have been very open about their identities and what they've been up to. It's sort of their MO. Pure speculation on my part, but I imagine it's basically a giant middle finger to everyone else.
Double jeopardy, I was previously convicted of 50700 counts of unauthorized access to various coldfusion sites. Among those was linode.
This one?
http://www.upi.com/Top_News/World-News/2015/07/08/Finnish-te...
Yeah.
How exactly did you get caught? opsec/comsec violation?
Difficult question to answer. There was a complete lack of "technical" evidence against me (e.g .bash_history files, wiretaps).
The only evidence the prosecution had against me were a list of compromised sites and several coldfusion 0days I had in my possession.
They could never prove that I generated the list of compromised sites, but the judges felt that the possession of said list was enough evidence to convict me.
We would've appealed but there was no point as the sentence was essentially nothing.
> We would've appealed but there was no point as the sentence was essentially nothing.
But you do end up with a record, which is not 'essentially nothing'.
I suppose it depends on what that means for a minor in Finland.
Since I'm not really planning to look for a job, my main worry was potential visa issues. But I haven't had any troubles visiting the few countries I do need a visa for.
How are you in a position where looking for a job in the future is not really necessary?
Bitcoin.
That's an awful lot of faith in virtual currency and its speculative nature to be betting the next 50-60 years of your life.
I can't even tell you how many pen testers I know with criminal records. Somewhere, someone will hire him to do security.
Who is to say xe didn't buy a bunch of it at $1 and sell it off, making millions, when it hit $200-$300?
Diversification is pretty important if you want to keep any kind of currency for 50-60 years.
Talented hackers are not exactly unemployable.
The other side to that is that if you appeal, and lose, you can come off worse. So even though a criminal record is "something", depending where the lawyers feel your chances are, it may be best going along with it.
I find this very interesting because I think it brings up a specific question. If this is actually from July, and they knew about it, why would Linode choose this specific time to announce the breach publicly? It seems very curious being that they are already getting punished from the DDoS attack and if they waited 6 months they could have waited until the dust settles a little bit.
Unless, of course, the group involved in the breach are also the ones unleashing the DDoS attacks. Which would also make me think there has been communication between the group and Linode, in contrary to what Linode stated.
I think since multiple customers were hit by this and are presumably all putting pressure on them about it, their hand may have been forced.
Could be, but the timing still seems really weird to me. If they did an investigation like they said they did, I just don't see why you would come out with it right now. Because from Pager Duty post it seems that no one could do anything to make them disclose it sooner.
I can't speak for the other folks that were compromised this way, but we decided to just cut our losses and move on at PagerDuty and spent the 30 days after the compromise migrating everything that was running there over to Azure. No point in putting pressure on a company that stonewalls you.
That's a good point. Not worth your time for a company like Linode that doesn't really care about its customers. I think people mistake the quick support responses to basic questions as them caring, but when it really comes down to the important things like security and communication during a crisis, it's clear that there is a huge lapse from the leadership level down. Someone in this post wrote about how they stopped $10k worth of Linode service a month and no one tried to get them back or retain them. That was very odd to read, especially for a business of Linode's size ($22 million in revenue isn't that big where you can shrug off $120k/year.) Par for the course it seems, and everything is starting to make sense. Thanks for sharing your story and sorry that happened to your team.
And of course your concern is pure as new fallen snow and you have no other motives.
They were expecting to do $60m in 2014 - http://www.sramanamitra.com/2014/07/11/bootstrapping-a-web-h....
I know we left them 4 years ago because how they implement bandwidth caps on private IPs, which a few years later another company also had the same problem with and nicely wrote it up on their blog: https://docraptor.com/blog/gone-in-60-seconds-how-we-moved-f...
Linode's attempt to keep us (spending $5k a month at the time, but we've grown substantially with AWS now. Linode were at $22m in 2011, so we'd have been 0.3% of their total revenue): "We will certainly be sorry to see you go."
That may just be because they were fed up of us after we opened 27 support tickets about the same networking issue over the course of 11 months though.
When I worked there support was under intense pressure to respond to tickets quickly.
Not nearly as much pressure to respond correctly.
To clarify and reiterate--employees who responded to tickets quickly were praised, even though there response contained half truths or outright falsehoods. If someone took 15-60 minutes ( or more ) to deep dive into an issue for a real fix for a customer, they were shamed and got a talking to.
If the Linode Manager database only stores the hash, how would they know the password?
How did you find out about the illicit login?
If they've been more compromised than something like a simple SQL injection vulnerability, it could be something like added code to log passwords or post them off somewhere else.
If the Linode Manager database only stores the hash, how would they know the password?
Weak password? Weak hashing algorithm?
> If the Linode Manager database only stores the hash, how would they know the password?
Hashes can be turned back into plain text, it is just computationally expensive to do so. Hashing only slows down an attack (and or increases the cost), it doesn't not mitigate one. In particular if the hashes aren't salted then a rainbow table is an extremely effective way of breaking all of the hashes concurrently.
The main method of doing so: Generate the hash for every combination of typable characters up to a given length (e.g. MD5() A-Za-z0-9 & specials up to 8x characters).
This can be mitigated using a more computationally expensive hashing routine (or increasing the work factor on a less computationally expensive one) and salts.
But given enough time OR computing power, all hashes will be broken. AWS makes breaking hashes a lot cheaper as you can bid on spare capacity and perform the operations relatively cheaply.
This only works if the input password has low entropy. You would think that people using Linode are savvy enough to be using long, randomly generated passwords.
> This only works if the input password has low entropy.
If you're generating every single possible password up to e.g. 8 characters the password's quality doesn't matter, only the length does.
A good secure password hash should be generated with a salt to slow down the process, in addition to using strong (slow) KDF function like scrypt. State actor like NSA could have store all possible 8 character combination today in a massive storage facility. God knows. But the size of such rainbow table is only effective if there is no salt, as with salt the hash is now different despite the underlying password is the same, thus there will never be enough storage if salt is present.
But the most effective "rainbow table"-like table is a look-up table with the followings:
* leaked password in plaintext, associate with email and any ID (forum username??)
* hash all of those passwords without salt
* hashes (with salt) of known leaked passwords (you try pas$w0rd and found a match for some hash with salt) - this only works if your attack succeed.
If you do a quick count you won't be surprise most passwords are fairly short and simple. If two complex passwords appear to be very similar, you can assume with a good probability they are used by the same person. You can learn some private data from just looking at password (e.g. birthday, pet's name, door number, company they worked for, sport team they root for, which many turn out to be the crucial hint or actual answer to security questions.)
I have never opened or downloaded any leaked data and don't know if it legal for use at all, but the black market probably has over petabyte volume of such data available.
It would be very interesting to see the whole world attack couple hashes per day. Imagine you go to a website, it gives you some plaintext, and you run a couple quick scrypt with random salt, and return the response. Now with a billion online users, run this every day once, you may end up finding one successful match of "this password == this hash with this salt" once in a while. But hey, that's what botnet can do...and then bitcoin!
If we presume the attackers had access to the system handling authorization, then the attackers introducing code to ship passwords offsite as users log in isn't really a stretch of the imagination.
That's true and great point. Let's play devil. You can't really be sure if Google engineer is sloppy and logging username and password on entry and then the SRE reading the log sees everything. I am not sure if their build system has plugin to detect such big red flag. Stacktrace is another place with potential leak of credentials. All of these are reasons when someone claims software is open source and auditable but they run the service themselves, it's really important to note you can't audit the actual server. They can log your username and password behind the scene while the client side appears to be 100% the same as the one on the server side.
What you describe is not rare, can be done with cross-site scripting. How it happens depends on the injection method (perhaps SQL injection).
12 character random strings are an absolute minimum for a secure password, because brute forcing and tabling start to become impractical. Longer strings are even better.
I wouldn't consider an 8 char password secure, no matter what the entropy is.
Yes length matters. That's why pretty much all "randomly generated" passwords are long. Mine are 20 characters.
They don't need to know the password with db write access.
In fact, depending on how the sessions are managed the attacker might just need read access to log in without a password.