"As a respected hosting provider, I hope they do the correct thing and refund me for this liability due to their error. Many people trust Linode, and they have proven themselves as a serious contender for hosting critical sensitive operations on the internet. I would hate to not see them live up to that reputation."
"hosting critical sensitive operations" in particular. If you are doing "critical sensitive operations" you need a more secure solution and process which will cost you more money.
Under no circumstances can a hosting provider assume the liability for something like this.
The tradeoff you make for the low cost you pay is that you might have an issue like this because someone screws up.
You pay more for a safe to store your money (and for a safe deposit box to store your valuables) because it's important and you understand the risk involved in not doing that. If you have valuable jewelry many times the insurance company will only insure if you keep it in the safe when you are not wearing it and even the amount of days is specified when it can be out of the safe.
It's unreasonable to expect (and linode's contract clearly states as other's have mentioned) a hosting provider to have a liability over what you are paying them. Edit Add: Unless you specifically have an agreement in advance or that is what they promised or charged you for.
Before anyone reacts to this with any harsh criticism please think for a second what liability you would want for any mistakes that you make with your web startup or idea. You could either be charging zero or charging a small $5 to $20 per month charge. You might make a mistake. Are you willing to accept and even be able to insure for thousands or even millions in liability for those mistakes?
It's certainly a grey area, but at what point is it safe to assume that if you get hacked, it's not going to be because your ISP got hacked? Say this happened to Amazon and it affected a company like Heroku or dropbox, both users of AWS? Regardless of what terms of service says, I'll bet there's some liability somewhere. And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!"
If a bank gets robbed, I'm not liable for the cash they steal. But how about if I've got cash in a safe deposit box and someone uses a fake id to get into it, and the bank doesn't recognize the fraud? That's trickier. And if someone robs my house and I've got a bunch of cash under my mattress, that's another story too. I know the analogy doesn't quite hold up because it's kind of like a bank and a customer engineering a safe together (eg both could be at fault for a break in), but there's got to be some responsibility on Linode's part.
I'd say this is more akin to stashing a bunch of money in a self-storage unit instead of a bank account. One explicitly insures against theft, the other does not. The onus is on you as a customer to decide what to go with.
> One explicitly insures against theft, the other does not.
I'd take issue with that summary. If I put things in a self-storage unit, and it gets robbed because some employee left his master key under the door mat, regardless of how bad of an idea it was to store my money there, that's still their issue.
Think about it this way. I could store my money under a table at McDonald's, in a self-storage unit, or in a bank. Clearly the self-storage unit should provide me some more security than McDonald's. So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault; it doesn't matter that there was a better security option, because it's still below what I was paying for.
"So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault"
A good point but keep in mind that courts don't have the level of expertise to judge what in terms of security is trivial and what is not.
The person trying the case and/or the jury may very well be someone who uses "football" as a password.
Also there are multiple cases of the very best companies with supposedly the highest levels of security getting hacked on a regular basis (might be a small percentage but it always makes the news). Consequently any company defending could make an argument that "this stuff happens even with the best and brightest" and it might be believed. (Well anyway that's what I would argue if I was a lawyer..)
So the public could easily be convinced in the case of a technology company something that would never fly as far as a screwup at the self storage - something physical that they can relate to (like leaving a door unlocked which is easy to understand).
A bank or safe deposit box business will most definitely have clear terms about how liable they are for if something gets stolen.
For a hosting company this is different. Especially because--it's hard to draw a line but I feel it's there--there's a difference between storing sensitive data and storing (what are practically) valuables/money.
I wonder though, I'm not clear on his set up (or business, even), but basically he was literally storing his bitcoin money on a Linode server? Since they're not a bank, nor a safe-deposit service, or are in the business of storing valuables (as opposed to sensitive data). I can't come up with a realworld analogy (they usually break down anyway), but wouldn't you want to wrap this data in an extra layer of encryption or something? It's not that hard to come up with some scheme so that people with root access to the Linode server can't do anything with it either. Since this is about (almost) real money, that's what I'd do.
"And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!""
Well of course that's never going to happen and the truth is the contracts of the more expensive provider no doubt also limit liability. (And all of this is in the TOS/contract etc). So what we are really talking about is who will do a better job protecting what you have AND more importantly who has more to loose if they screw up. (Small linode has more to loose but they also don't have deep pockets to pay. Amazon has deep pockets but access to magnitudes greater legal help to prevent having to pay.)
I can put a "not responsible for stolen items" sign in my restaurant, but if the coat check employee bolts out the door when you hand them your coat, I'm buying you a new one.
<devil's advocate>Yeah, but could the (ex) coat owner hold the restaurant's landlord liable? Isn't it the restaurant _managers_ problem?
I think there's a _lot_ of "grey areas" here, and while I feel sympathy for the guy who's out ~$13k worth of bitcoins, I can't help but think he was "doing the wrong thing" relying on the security of an inexpensive vps to keep them safe…
What would you recommend? Would you say the same thing if he'd been colocating and a data center employee had stolen his bitcoins? Because that seems far more analogous than any restaurant analogy, and I don't see any reasonable way for somebody who's not a huge corporation to avoid this kind of risk.
You have to trust somebody at some point unless you're keeping the server locked in your own closet. It seems really bizarre to me to say that a hosting provider doesn't have a responsibility not to steal your stuff.
Firstly, I'd start asking whether a $19.95/month shared hosting* account is a "reasonable" place to store $13k worth of (effectively) cash. I'd be _very_ careful if I had that sort of folding-money-type-cash on hand, and would under normal circumstances automatically deposit in a bank account to mitigate the risks involved with carrying it around. And I'd usually take steps to not ever have that sort of value of cash build up or be required - the only transaction I've ever done of that sort of value in cash is selling or buying a car from an individual - and that's always been a direct from transaction to the bank type of arrangement.
If I had enough bitcoin value that it'd hurt to lose it, I would not (at least now in hindsight) store that on a machine that other people I don't know/trust have root access to. Maybe I'd keep my wallet on a usb stick in my pocket or in a safe at home? I think though that at somewhere near the $13k value the "right" thing to do is convert it to cash and take advantage of the existing banking system and its time-tested security and insurability.
(* Which is fundamentally what a linode VPS is, at least from anyone with access to the hypervisors point of view.)
Then he should Colo with a hosting provider with a contract provision that specifically holds them liable for any losses related to problems caused by the host, and enumerates those possible losses beforehand.
Simply not going to happen at any reasonable rate.
There's a saying in many businesses with different variations:
"Price, quality, speed" pick any two.
So this would be:
"Price, security, bandwidth" pick any two.
People pick on price and bandwidth security is taken for granted to be commercially acceptable. The colo can easily figure out price and bandwidth they are clearly defined. Losses from a security breach not as easy - to many variables. Same reason insurance companies love to write life insurance but hate to write disability insurance. Life insurance is absolute (you know when someone is dead and tables exist to compute probabilities on when they will die). Disability is open to interpretation, fraud and other things. It's not black and white.
Would have to meet several tests. 1) Hosting company knew and agreed to the value of the things they had control over. 2) Employee did it and they were negligent in hiring that individual. 3) Reasonable and customary for that type of stuff to be in that situation given pricing and the practices of others.
With respect for #3 it would be reasonable for a bank safe deposit box to contain a $100,000 ring maybe but not to contain a 10,000,000 ring.
In an actual court case, the reasonableness of everyone's actions would be evaluated, but it's hard to imagine a court finding it unreasonable that someone placed data worth $13,000 to them on a respected VPS provider.
That doesn't mean Linode has any legal liability in this case, just that your analogy is off the mark.
Second, where are you getting $19.95/month from, anyway? I haven't seen the plan in question mentioned, and even if this particular VPS happened to be Linode's lowest-end, the last time I looked (a while back, granted), slush had multiple large VPSs with Linode.
Third, really, what does the price of the VPS have to do with it? You think as the cost of the VPS goes down, we're entitled to less assurance that an employee isn't going to bolt with our data?
Finally, HIPAA and PCI regulations are ginormously complex, but violations of them almost inevitably cost a hell of a lot more than $13k.
I'll freely admit I've only been thinking about this since reading this article, so I'm both not-fully-informed and I'm thinking about it as a response to some guy losing ~$13k, but to me it's _not_ reasonable.
(And my $19.95 number is perhaps hyperbolically chosen from their least expensive vps offering - but my assumption would be that the management/hypervisor back end would be shared across their entire infrastructure, so I think my argument holds, in that I'd expect the higher priced offerings to "only" have the employee-reliability-assurance of the cheapest vps…)
Why are you looking at it as "X has the same assurance as Y" instead of "Y has the same assurance as X"?
If you've got a vault that holds a massive diamond, and a little gold ring, do you become concerned because the diamond "only" has the protection of a little gold ring?
Linode plans (like those of other hosting/vps providers) are differentiated on storage/memory/bandwidth. Security isn't a factor. Compare that to an auto where "security" is definitely mentioned and part of the selling proposition and what you pay (they mention theft devices, crash stuff, airbags etc.)
PCI doesn't specifically say anything about usage of a VPS. It does however speak about access to data.
If you have encrypted credit card information, you'll be asked to list those that have access to the encrypted information, they encryption key, and the key encrypting key. Then you'll be asked to justify their access.
I'm sure I could come up with with several other major violations, but this alone is severe enough that I can't envision a way that you'd pass a PCI audit.
Sure but there is legal precedent for that as well as it's an easy concept for the everyday man to understand. (Employee and seeing them makes it a open and shut case.)
As an aside, signs like that are the same as ones that appear in auto shops that say "insurance regulations don't allow you in the shop". I owned a company that did manufacturing and said a similar thing because I didn't want anyone in the machine area. I've dealt with insurance companies for many years they don't dictate things like that at least from my experience.
Your example is correct though the restaurant would be liable in the situation you cited because you actually saw an employee steal the coat. If you saw someone else NOT connected to the restaurant AND the coat was expensive you are probably out of luck. It's your property and there is no assumption that a restaurant protects you from acts of god.
The key is was their negligence on the part of the restaurant.
On the other hand if there was a large party at the restaurant and normally a coat check girl that's a different story (devils in the details with this stuff).
"please think for a second what liability you would want for any mistakes that you make with your web startup or idea"
It seems to me that bitcoin wallets are a relatively new and not well enough understood risk. There are very few other "files" like them, in that an attacker copying them can deprive you of their value in a way that you cant protect with backups. I feel a big part of current "internet security best practices" are about minimising the risk of getting exploited - but with a pragmatic limit to how much effort you invest mediated by the excuse of "if we _do_ get rooted, we can always reinstall and recover from backups". It'll only cost you time, and perhaps some reputation, and may put assumed-private-to-you information in someone else's hands, but it hasn't deprived you of access to any of your data. That doesn't apply to bitcoin wallets, and example like this are pointing out flaws in assumptions people are making about appropriate ways to manage them.
It'd suck to be "that guy" who provides the object lesson in why we need to think differently about bitcoin wallets to just about any other file type we might put on an internet accessible machine, but we _do_, and I don't know whether we have an answer to the question "Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"
I _think_ the answer is "if you can't trust those people, you can't risk storing your bitcoins there". There's a reason people keep their money in banks, and not in train station luggage lockers. I'm guessing inexpensive commodity VPS's should be considered closer to storage lockers than bank vaults. I suspect the finance sector and/or fortune500 companies have hosting arrangements with companies offering bank-vault grade protection and reserve bank style insurance - but sure as hell not at $24.95/month.
Don't store your money on servers you don't own. Don't have a web interface. Don't allow anyone to connect. Don't use passwords, use public key encryption. These are the basics.
An interesting point is that for receiving coins and for long-term storage, bitcoin wallets do not need to be online, and in fact do not need to ever have touched a machine that has been online. While slush "just" lost his "hot wallet", another user lost 200k. When will it become common best practice to store high-value accounts entirely off computers? You can print out a bitcoin wallet and put it in a safe deposit box for storage, and still add money to it.
I don't see what's so conceptually new about bitcoin wallets. They're just plain text that you don't want people getting access to. It's no different than storing passwords in plain text: if someone copies them, they're completely compromised (until the user changes them). The solution is pretty simple: encrypt your own bitcoins with your own password (or more ideally, your private key). Then, if someone hacks your server, they don't get anything.
Keys normally grant access so worst case once they're compromised you can take the server offline physically. Even private signing keys can be revoked though some damage may have been done.
With Bitcoin, once it's copied, you can consider it gone forever, irrevocably, in totality.
I thought the idea was if someone copied your bitcoins it was essentially a race to be the first to spend them. I guess if you're stealing bitcoins, though, you're going to be quick to run them through a couple of transactions to claim ownership of them.
(Of course, my understanding might be completely off-base.)
A 'bit coin' is a space on a block chain that everyone has a copy of. You lock the coin with a cryptographpic key, which you need to store. Whoever has a copy of the key can unlock the bitcoin and re lock it with a different key, such that now only the new owner has a copy of the key. Everyone can still see the entire block chain, but only one account (that no one knows the owner of) has the ability to move that coin.
So they got access to these peoples keys and transferred ownership of the coins.
Most money supplies are regulated, but bitcoin isn't regulated. No one has the ability to say 'reverse that transaction', but it also makes the currency safe from inflation and interference by money printing governments and privacy snoops.
There's an economic myth that inflexible supply of a commodity gives that commodity when treated as money, stability. It does not, as looking at this graph of US inflation/deflation time shows (1944 is when the dollar stopped being gold convertible):
1. The money supply around a currency not only contains the mined/minted instances of that currency, but also liquid currency-denominated assets, like customer bank balances. So money supply is not necessarily bounded.
2. Inflation/deflation can be considered measures of the change in demand for the currency. In times of deflation, holding money is valuable because it becomes more valuable.
3. From the above graph, you see that during the gold standard, inflation tended to be mostly balanced out by deflation in the long-term, so long-term inflation was low. But in the short term, prices were very unstable as inflation jumped all over the place, and far more unstable even than fiat money in the past three turbulent years that we've seen.
4. From the point of view of an economy, inflation and deflation are not symmetric; because of the value of sitting on money during periods of deflation, savers do not tend to invest their money but move money from investments to cash savings. This undermines economic activity. But in an economy with a rich range of investment opportunities, moderate inflation does not penalise acquisition of money and does encourage investment.
If you want a non-performing store of value and don't mind big fluctuations in value, gold is there and we know how to secure gold rather well. Bitcoins are another non-performing store of value with far more drastic fluctuations in value, and securing it involves the double vulnerability: physical security of storage media, information security of computations involving bitcoins. And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
> physical security of storage media, information security of computations involving bitcoins
Only funds that you have daily access to need be vulnerable to the latter point, as physical security (air-gapping) is sufficient when you do not need to -send- funds.
> And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
Strongly disagree. Can you keep N redundant copies of your gold? Combined with secret splitting, you could require that at least K of N secure locations be accessed.
> Only funds that you have daily access to need be vulnerable to the latter point
I'm talking about protocol risk: e.g., the software that implements the protocol on some machine is flawed, so the cryptography can be effectively breached. Or there is some issue with the protocol, like but worse than the issue Kaminsky found with anonymity.
>Can you keep N redundant copies of your gold?
Have you ever found that your backups didn't contain what they were supposed to contain?
Gold gets stolen, but besides such things as costume jewellery, I think it doesn't often get lost.
Not exactly. In order to spend bitcoins (transfer them to other wallets) you need them not encrypted.
So if you are a bitcoin business making bitcoin payments, at moment you cannot avoid the risk of having the wallet stolen if someone gains unauthorized access to your operating computer.
"Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"
Probably not, _maybe_ trusted computing could help[1]. But it might secure enough if you had a special piece of hardware that stored the bitcoin-key and did all the signing operations. I believe things like these exists for Certifications and other signing-keys. I doubt VeriSign stores their root certificates on just-another-box :)
[1] In theory, TC could "anchor" your data to the hardware chip on the motherboard, and breaking it would require physical tampering.
You're thinking of a hardware security module (HSM). These are standard for e.g. certificate authorities. To the best of my knowledge, they have never been used by bitcoin outfits, but that is certainly possible, in principle.
Still, the amount stolen in Bitcoins here is probably small potatoes next to the value of all the credit cards and personal details stored in other Linode instances. If someone can hack their admin panel and get root access to 8 accounts, they can get access to all the accounts. This time it happened to be done by someone who was going after the Bitcoins. But how can they claim anything in the way of security?
People who host on a VPS typically do so because they want to run complex applications that scale up, and do so without the cost or headaches of adding more physical hardware. Most people who use Linode probably do not do so to host static webpages, which can be done just fine on a shared server for a quarter the price. While Linode might be cheap, they do claim to be secure, and if they didn't it's doubtful people would host their apps there. And most apps do take some personal information; and a large number of them take financial details.
No one would use Linode to host their apps if they thought all accounts were rootable from a master login panel. And no one would use them to host static pages. So who would their customers be if people didn't expect their instances to be secure?
So, a customer service interface was compromised via stolen credentials and used to access various Linode instances. A couple questions that immediately come to mind:
1. Can this interface be accessed from anywhere on the Internet? If so, why? If not, does that mean other systems owned by Linode were compromised as well?
2. Why can customer service representatives access and update servers without the client being notified and with minimal logging?
Regarding #1, an update from Linode was just posted:
"Our investigation has revealed a customer support interface was used to access your account. The compromised credentials have been restricted and we are discussing policy changes to prevent this from recurring."
I'm a Linode fanboy, but we need maximum transparency on what occurred and what's being done. What support interface? How compromised? Who's credentials, etc.
Hopefully they're working on it, and will give a post mortem once they get it sorted out. I'm inclined to show patience and not demand they do anything other than ascertain the scale of the breach, alert those affected, and secure their systems at this point. Later, they can get into what happened and how they will avoid it in the future.
That update had already been released when I made my original comment (hence why I said "a customer service interface was compromised via stolen credentials"). It doesn't reveal how the credentials were compromised, nor how the attacker managed to use them to log in.
Reading the ticket slush posted it shows no password change logs, if linode was compromised either the whole infrastructure was compromised (unlikely) or a rouge admin or a admin comprimised account accessed the vps and stole the $, as per the bitcoin forums.
Total stolen is roughly $16,000 USD
> if linode was compromised either the whole infrastructure was compromised (unlikely)
That's funny. I know from experience in the script kiddie part of the Internet that it was sometimes exceptionally easier to hack entire datacenters (even ones worth millions of dollars) just to get into a few of their customers, especially if those customers secured themselves.
Hosting companies have very sophisticated websites sometimes, meaning that they're almost always vulnerable to something.
I know of an SQL injection in an very large U.S. datacenter's administration panel which has been there for at least six years. Six years and it has not been fixed, and maybe a dozen people have independently discovered it. The deeper you delve, the more you realize that at least a handful of people also have access to important upstreams/backbones.
It's a lot bigger of a mess than anybody realizes. A bit of advice: if you say you're secure, you're either lying or colocating.
So far there are 3 people who've reported their Linodes compromised. They all had popular Bitcoin services running on their Linode.
3 compromised emails? Very unlikely. They are all major contributors to Bitcoin, I think they know a little more than using the same password everywhere.
Linode will only send you a confirmation email if you enable the feature, otherwise tough luck. It's also been confirmed by the vice president of Linode to be a fault on their side.
> Linode will send you a confirmation email if you access the admin panel from a "new" IP. This guy must have had his email address compromised as well.
The attack was not via the consumer facing admin panel. It was the internal Linode customer service interface.
The OP's tone clearly indicates that he expects some compensation, Linode's TOS are pretty clear:
Therefore, subscriber agrees that Linode.com shall not be liable for any damages arising from such causes beyond the direct and exclusive control of Linode.com. Subscriber further acknowledges that Linode.com's liability for its own negligence may not in any event exceed an amount equivalent to charges payable by subscriber for services during the period damages occurred. In no event shall Linode.com be liable for any special or consequential damages, loss or injury.
This also provides an interesting dilemma when it comes to such events. In this case the damage is relatively easily quantifiable, he got X bitcoins stolen so the damage is X times the bitcoin value at that time. Still, it could have easily been user personal data or credit card information, which would have made an evaluation harder to make.
One of the risks of using such a platform I guess and something that anyone who does it should consider.
That is very interesting indeed. Holding Bitcoins in a wallet on your server is the equivalent of having actual cash on your server. When someone takes it, there is a definite value of the damages vs. ID theft where the damage could be drawn out over a period of time.
As I consider launching my own Bitcoin business, I have wondered about where to host the bitcoind that serves as my business bank account. This event certainly makes me reconsider just any VPS.
This especially seems hard to certify. Bitcoin aims to be quasi anonymous. Users could just log into their own systems and transfer coins to another anonymous wallet elsewhere and then try and claim robbery. Or be rooted (by say script kiddies) and then take advantage of that, transfer coins, and try and claim for them.
They end result is something like bitcoin would seem nearly impossible to insure for in any reasonable way.
Isn't this akin a bit to storing a bag with $12K in a storage locker in a public space and then asking to reimbursed after robbers broke into the locker and stole the bag.
I think the author is hoping for way too much. The world doesn't work that way, nor probably can or should it.
It's actually a bit worse than this -- basically this is analogous to putting the $12k in a storage locker and then posting on the internet the exact location of the locker.
Given that a bitcoin client was almost certainly running on this box, basically anyone could have connected to the bitcoin network and established this guy's IP address as a target.
If you run bitcoin software, you are advertising your IP address as a high value target.
Insurance won't insure for what they don't understand and build a risk model for. I can assure they won't understand something like this for a very long time.
I'm not sure they need to understand bitcoin specifically to build a risk model, wouldn't they just need general data on losses suffered by various internet hosts due to hacking attacks? Doesn't really matter exactly what is stolen as long as they have a corpus of data on the value of everything that is stolen in this manner. Bitcoin wallets probably fit somewhere in the payoff curve for that.
I've never heard of one, but there may very well be a market for that sort of thing. The problem with Bitcoins will be the legal jurisdictions involved, since these services are regulated nearly everywhere, right?
I write software for use by high risk insurance carrier. There is coverage for just about anything. One of the inland marine insurance types might cover this. (possibly Electronic Data Protection or Valuable Papers)
Ideally, install rkhunter on fresh system, right after updates, APF, & BFD. Then update the binary check-sums with this command, if you know server is secure:
Update file properties:
# rkhunter --propupd --sk
Run a system check to make sure it is known clean:
# rkhunter --check --sk
Lastly, sign up for the security alert mailing list for your version of linux on your server.
If you want maximum security, be sure to password protect your boot loader and use an encrypted file system. This will make it very difficult for ISP to work on your server however!
And, I'd add "if any of this is news to you, you should _seriously_ question whether you're skilled/competent enough to be admin-ing publicly accessable servers with files (like bitcoin wallets) that can be valued in the thousands (or tens or hundreds of thousands or more)".
Hmm, for a customer of a cloud provider, this sort of thing will be very hard to defend against.
Maybe if the customer service system had had two-factor security, this might have been avoided (i.e., customer service can access your account only if you read them your hardware token's code).
Requiring SSL/SSH client certificates even for intranet accesses might have deterred this attack.
I hope other cloud providers take note of this incident. This is a very interesting incident.
> customer service can access your account only if you read them your hardware token's code
At the very least, I'd hope Linode implements two-factor authentication for their own logins. A customer-provided OTP would be great but you'd need a customer service reset tool for that when people forget, which would put you back where you started...
Not necessarily if the reset tool is manually driven and audited. The vulnerability we're worried about here is an automated attack against many customers of a single hosting provider.
There will always be ways to human-engineer your way into any single host. Having a hosting provider just increases the attack surface a little.
> The vulnerability we're worried about here is an automated attack against many customers of a single hosting provider.
This was an attack against Linode's customer service systems, which allow their support reps to reset root passwords. There's no reason for that system not to be protected by two factor authentication on top of heavy logging.
Actually not. Just use a loopback cypto FS to store the sensitive stuff. The reason they had to reboot the machine is that they just had access to the HDD where they could change the password, as opposed to having live root access.
how do you transmit it to the server in such a way that it can't be intercepted by someone that compromises the bits of your virtual that boot before the key is uploaded?
Yes, but will that help you if the attacker trojans whatever it is that is doing the decryption?
I mean, I'm very clearly not a crypto expert, but I do believe that this would be quite a lot like what Bruce Schnier calls 'the evil maid' attack. Instead of having a bootloader, you have a minimal Linux install, then you get a key to that minimal linux install, and that minimal linux install uses that key to decrypt your encrypted disk.
I believe that if that minimal linux install that does the decrypting is compromised before you log in, in theory, the attacker could then insert something in the code that runs after the data is decrypted to, say, send some of that data elsewhere, or, say, open a back door for them to log in and examine the decrypted data.
I mean, certainly, you are making the attacker wait for you to log in (assuming that they've only compromised the admin interface and don't have full root on the dom0) and that's another step you are forcing the attacker to take, and you do at least have a chance then of detecting the compromise and /not/ sending the key, so I'm not saying that it's worthless.
Of course, this is all protecting against a compromise of the admin tools that does not lead to a compromise of the dom0. If the attacker compromises the dom0 without rebooting it or otherwise disturbing the guests? they have access to your ram. They can snapshot both the ram and the disk and take apart the system at their leisure.
I don't think I've ever met a professional sysadmin who could defeat the evil maid attack. All the ones I've met would refuse to think about an attack vector if it implied they could not boot their servers.
Yep. Classically, we abdicate responsibility as soon as physical security is breached. With virtual machines the problem is mostly the same, but it moves from "true physical" to "virtual physical" -- once the host environment is breached, all bets are off.
Any sysadmin that claims to be able to protect against a physical access attack or its contextual equivalent is either lying or incompetent. In neither case should that sysadmin be considered "professional".
It won't stop a private individual, either, not even one with a budget of approximately $0. Cold boot and similar attacks, even just yanking the memory or forcing a CMOS reset, are trivial. Please tell me you don't do this for a living.
We are talking about live servers. Monitoring should detect server going down, raising suspicion of a physical attack.
The evil maid attack would be useful against the system admin's workstation. Securing NOCs is beyond the scope of this discussion, but it is not difficult.
That's not strictly true. If you're careful and imaginative, with moderate effort you can commit a fair amount of highly random data to memory. You just can't expect to change it every month.
This may be oversimplified, but it's the correct horse battery staple.
Eh, it depends, a random fragment of a very long poem for instance can be quite easy to memorize but at 80-90 words/300-400 characters long is pretty damn secure, doesn't even need to be written down then.
What the? If you write it down, it's a bad password. Make it a pass phrase, make it long. And possibly store it in an encrypted password database, protected by a master pass phrase.
I really doubt that "only HDD access" is a secure position in cases where your cloud provider CSR is pwned.
If that's even the case here. We don't have much indication that the attacker couldn't have taken a snapshot of a running system (or that that will be how it happens next time).
So, sure, use a loopback crypto filesystem. It just doesn't provide much in the way of a security guarantee (but it adds some measurable costs).
Yes. Tracing this back to one of their own employees, the company is clearly liable for any resulting loss. If they do not find themselves liable (and a court does not) this will be a big problem for people doing shared or cloud hosting, and traditional, physically secure datacenters will see a resurgence.
How many times does something like this have to happen before people learn to encrypt? Any serious business or financial data should be encrypted, period. Almost all of the major hacks we read about could have been minimized if not entirely avoided if the data was encrypted.
I just read the release from Bitcoinica where they explained that the server accessed contained _only_ Bitcoinica's "hot wallet", and that no code, services, customer data, or other wallets were stored on the server.
If this was the case, why couldn't every access to that wallet, which, assuming the above is true, necessarily occurs on other servers, run a decryption on the file first? Even if you keep the passphrase and/or secret key in plaintext on the machines that run the code, the separation should prevent this kind of rogue access as long as the intrusion is isolated as these people claim.
There is really no excuse just to have a plaintext wallet sitting around anywhere anymore (the official bitcoin client now supports symmetrical encryption). Like credit card numbers, when a wallet is accessed it should be decrypted in ethereal storage like RAM and promptly discarded; it should never hit disk as plaintext. At least the same practices used for PCI compliance and credit card data should be used for btc wallets; preferably better since there is no recourse if your btc wallet is compromised.
I do not believe implementing something like you describe is viable right now.
Although, AFAIK, bitcoin 0.5+ support keys encryption. I am not sure if this available only through GUI or through API as well, but even if it is not available through API, it might be good idea to implement.
>If this was the case, why couldn't every access to that wallet, which, assuming the above is true, necessarily occurs on other servers
From the sounds of it, this was that other server. All it did was operate on the wallet. And if they used other servers, then those would have been the target of the attack.
And, no matter how much damn encryption they have, they rooted the box that operates on the decrypted data, thats game over. The only attacker you would be able to thwart with more encryption would be the one who is able to root a linnode VPS, but unable to extract the key or decrypted wallet from from software running on that box. Sure, there is probably some number of attackers in that space, but security is a game of diminishing returns, and there are different security measures to take that are a much better investment of time than stopping that small slice of people.
The encryption wouldn't really help, because the software manipulating with the wallet is on the server anyway.
The server software was used for automating bitcoin spending, too, so you couldn't just save public parts of the wallet on the server.
The only thing that could be done would be obfuscating the encryption key to the data very hard in the code, but that's hardly a security, just obscurity.
I'm not really sure why people are trying to store bitcoins on a VPS in the first place. You can't process credit cards on a VPS and be PCI compliant (it's against the rules), but any moron can do what they want with bitcoins.
This strikes me as a really good point. The onus for proof of merit is on the decentralized currency.
Simply put, people trust Visa and MasterCard to safely manage transactions, for better or for worse. Regulation like PCI helps ensure that this trust is sound.
The Bitcoin community at large could really benefit from a set of published best practices for managing transactions. Anybody possessing an insecure wallet is ultimately a liability to the credibility of the currency.
It'd be worth it if more people used Bitcoins. No, we don't NEED regulation. We need competence and standards, which can come about without incompetent government intervention.
Show me your wallet with a good amount of cash and leave the room for a while.
Afterwards, let's talk about your comparison. Is 'can be stolen' really something that the state can protect you against? Let's discuss it over dinner. Depending on the contents of the wallet I'd pay.
On a more serious note: Your mockery, while amusing, is unrelated to the problem at hand. 'Stealing amounts of $currency from private persons' is not a new idea or something that bitcoin is supposed to change?
1) I don't think PCI is relevant here. If you store bitcoins somewhere and they get stolen then this is, in my world, cash. It's your very own digital cash. Not a credit card. That's why I constructed a (probably poorly implemented) example of someone leaving a wallet full of shiny $currency notes out there.
2) 'What state?' WTH? Can I reply with 'What kind of question is that?' The state I'm coming from is called 'Northrhine-Westfalia' [1]. Now I'm living elsewhere and there are no 'states' here. I can offer the district 'Tel-Aviv'? The point is, 'what state' is invoking aggressive feelings towards your US-centered mindset.
First and foremost: I'm sorry. We clearly didn't talk about the same thing and I misunderstood what you wrote.
My take: Someone was mocking Bitcoins with "But all that regulation is evil and it's the freedom of bitcoin that gives it the power" and I tried to make a point saying that _no regulation is involved here_ (laws? certainly). This is a wallet, it got stolen. Your credit cards are protected, your cash is gone for good.
You invoked PCI and I was (and am) unable to make the connection, maybe again because of a misunderstanding? I'm talking cash. Bitcoins are cash in my world (or - at least their value is equivalent to cash, if you choose to sell them).
From there we went downhill and I overreacted. Yes, for me 'state' is exactly what you posted. Again, sorry for the lapse.
Are you sure? I think that you may be mistaken. The bar is just set higher in a "virtualized environment"...
"In a public cloud environment, additional controls must be implemented to compensate for the
inherent risks and lack of visibility into the public cloud architecture. A public cloud environment
could, for example, host hostile out-of-scope workloads on the same virtualization infrastructure
as a cardholder data environment. More stringent preventive, detective, and corrective controls
are required to offset the additional risk that a public cloud, or similar environment, could
introduce to an entity’s CDE.
These challenges may make it impossible for some cloud-based services to operate in a PCI
DSS compliant manner. Consequently, the burden for providing proof of PCI DSS compliance for
a cloud-based service falls heavily on the cloud provider, and such proof should be accepted only
based on rigorous evidence of adequate controls."
Amazon getting a PCI compliance pass was a big deal. The last time I looked, you needed to be able to ensure secure access to the facility, enumerate who has physical access to the hardware and when, and things of that effect. And you need to be able prove all that in the event you're ever compromised.
> The last time I looked, you needed to be able to ensure secure access to the facility, enumerate who has physical access to the hardware and when, and things of that effect. And you need to be able prove all that in the event you're ever compromised.
None of that should be particularly difficult for a VPS provider as large as Linode.
I'd argue this isn't about bitcoins.
A (popular) VPS provider, according to that article, had a security problem that allowed some idividuals to access the VPS management interface for any machine they cared for.
They could've defaced your site in high traffic times. They could've logged in and delete your projects on the VPS. Depending on your setup (they had root) they could've searched for your backups. They could've read your mail, if that machine is your infrastructure service - and continue from there (password reset, amazon, buy expensive stuff. password reset, twitter, damage your reputation).
In this case the bits modified were part of a virtual currency and had a more or less clear value. I'd say there could've been worse results of that security hole though and 'don't put anything on a vps' is not a solution.
I'm sorry, but you just said the same thing again that I was arguing against. 'Money' is not as clear cut as you'd like it to be.
Bitcoins are no real recognized currency. So you can trade them for USD -> Don't store it?
What about this great project I'm working on? All my stuff on the VPS, because that's convenient and accessible from everywhere. I spent a double digit number of days on it. I have a daily rate for working as a programmer. Don't store it?
You totally ignored (so hard, that I think you didn't read it fully) my post about issues that are harder to value even. Access to your mail can be devastating. Even if you don't store 'money' on that VPS. Putting a dent into your online reputation by messing with your life on the net is hard to value, but certainly damaging. Again, no 'money' stored.
Bottom line: You ignored my point or didn't read my post at all. You picked a line out of context and refuted it with a pointer to the argument _I explicitly tried to prove wrong_.
Bad comparison in my opinion, BitCoins can be stolen (taken away and become unrecoverable) whilst your project you've been working on is recoverable.
Also the value something is worth is what someone else will pay for it. You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.
1)
Recoverable: Only if I have backups (which I excluded in my comparison, and would be a fault on my side. But go with me here..). Otherwise I'd need to invest (see the word I used here?) time to create it from scratch. That's equivalent to an amount of money (the exact amount is hard to define, granted).
2)
'You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.'
Right. But it's totally okay to value ~worthless~ stuff you have according to market rates, although you didn't sell them yet? Why are we talking about ~12k USD here? That's just a couple of bits and bytes on a disk. Yes, he _could've sold_ that at a specific time for a specific amount equaling ~12k USD. He didn't. Why do you assign this value to a highly fluctuating 'currency' on your disk, but don't like me assigning value to a 'yet to be successful' project on disk?
Point 1 isn't really debatable in my opinion, you make work, you back it up and if someone deletes it off your server you just move it back on. Bitcoins by design don't allow you to recover them.
Point 2, back in the day when currency was backed by gold reserves would you have said the coins and notes people had were also worthless? As they are not 'sold' yet into gold?
Also I don't say a project on your disk has no value, a project on your disk has worth, the amount it's worth is how much someone will pay for it. Because the project on your disk doesn't really abide by any fixed standards you will probably find it's quite difficult to sell it.
Then this conversation should be about the security breach and the BitCoin aspect should be a side note. The data lost is not what is important, just how the breach should be resolved and prevented in the future yes?
They were coins of a mining pool (slush pool - mining.bitcoin.cz - one of the largest three). And they were only the 'hot coins' left on-line for user withdrawals. The majority of the coins are kept offline/cold, as a common security measure for any bitcoin service.
Update: the Linode compromise first reported was that of the "slush" mining pool (mining.bitcoin.cz), reporting a loss of 3094 BTC. Second report was the donation-funded bitcoin faucet, reporting a loss of all of its 5 BTC.
Third report is the biggest, Bitcoinica.com which is arguably the second-largest exchange. Their main site is hosted at rackspace, but their 'hot wallet' was hosted at Linode, and contained 10,000 BTC which were stolen.[1]
EDIT: Those not following this incident on the bitcoin forums might be amused that the attacker used the stolen bitcoins to form a transaction with a size of 1337 bytes. That's probably not a coincidence, since the size of bitcoin transactions are usually under 1kb.
I already asked in the comments of the original article how many rounds of sha1 are used. SHA-1 still isn't the best, since it yields to FPGA attacks, but a single round can brute-force all 8 character passwords in less than 2 days on a GPU. My guess is that 10k rounds of sha-1 would probably not be feasible for non-dictionary attacks without specialized hardware.
You can salt all you want, but an 8 character password with a single round is going to fall very, very fast. Salt, being public, has nothing to do with it.
It does however mean you have to spend two days per password, rather than two days for the entire user base, or ten minutes with a pre-existing lookup table.
In my opinion it is not. Here is the common use case for people: 66% of them use 8 character or less passwords. Forget about the common longer passwords like "letmeinplz" just doing a brute force would give them 66%. In terms of loss, even 1% of passwords being compromised is bad since many people use the same password for multiple things.
Now, how long would it take for people to take down bcrypt? in the scale of time bcrypt is all the way over there -> from SHA1 and cleartext. Bcrypt will protect you even as computers get faster. SHA1 a couple decades from now will be be compromised for 9 or maybe even 10 characters. No way with bcrypt.
Your second point makes no sense, bcrypt factors are manually increased and require a new write for every item in the store. If we time capsule the SHA1 hashed db of passwords and the Bcrypt db for 10 years you cannot prove that the SHA1 database will be more easily compromised than the Bcrypt database. There may be a cryptographic break in blowfish in the time.
They don't mention a work factor for their SHA1 hash storage, they could be using PBKDF2 for all we know.
If you're looking for something to pick on about Linode here, go after something else, this isn't the worst thing to come out today.
How did the attackers know what they were looking for. I'm going to assume that it's a small minority of linode users who have bitcoins on their machines. How were just these users targeted so accurately? What tied together knowledge they used bitcoins to those VMs and their linode accounts?
Also, was the nature of the attack just that the were able to login to your linode admin panel and from their root the machines and then loot your wallets?
Simplest answer is probably the right one in this case:
Someone at Linode did it. Ran a script to see how many bitcoin files there were on all the machines (they probably do these types of queries for anti-virus/whatever anyways) and took a customer support password to log in and get the coins. If he did it right he still might be working there, as it is easy to get credentials from friends/coworkers (even though it should be really really hard).
I'd be very surprised (and suspicious) if they were running any kind of diangostics over their customer's data without an explicit signed contract. The liability worry there alone is scary.
you do realize that in this theory, the same person then went and stole thousands of dollars in bitcoins, right? I don't think they were worrying about liability...
You misunderstand. Not the thief's liability, Linodes. If someone, say, engages in insider trading because of something they saw in Linode's own analysis system, Linode can be sued for failing to protect that information. If they have a policy of never reading customer data (and can prove it) that becomes much harder. The posited "anti virus checker" would throw that promise out the window.
How so? Google credibly claims (somehow) that they don't "read" customer email, even though they run the largest-scale automated email reading system on the planet.
Sorry, there is just no way that this is the case. Please don't throw such a serious allegation out there without any evidence. To even suggest that this is technically possible for an employee to do is a serious allegation, let alone suggesting that someone did it maliciously. This spreads all kinds of FUD.
I'll happily eat my words if that turns out to be what happened, but it is definitely not the simplest answer.
> To even suggest that this is technically possible for an employee to do is a serious allegation
It is technically possible for an employee to do it because it seems (from the linked pastebin above) that is how it was compromised, an elevated account for linode manage was compromised.
As for an employee being the one that did it, that is probably the least likely cause.
I believe he's referring to the part about employees (at least the ones that have access to the customer dashboards) being able to run a script to scan for bitcoins.
Up-vote to the original comment because it's not stupid or impossible, just unlikely.
We can only speculate at this point.
The simplest answer is probably that one of the staff was subject to a targeted hack and a 3rd party gained external access to the CSR tools.
Possibly for an extended period of time. <-- This is the concerning part.
It's relatively unlikely an internal staff member would do something this dumb (but, not impossible. we've had this happen _here_ where I work, with credit card numbers, but obviously the person responsible was caught almost immediately).
They could have looked at the ip's of bitcoin services and noted those running on a linode. Or collected ip's from bitcoin nodes that their own node has seen and looked up those allocated to linode.
>Also, was the nature of the attack just that the were able to login to your linode admin panel and from their root the machines and then loot your wallets?
The way I understand it the attackers were able to get access to the admin panel and invoked some kind of 'change root password' emergency stuff. The machines were rebooted it seems, which makes sense: The interface of Linode has probably/hopefully no access to the root password. Maybe this 'Reset my root' feature (now I'm guessing) reboots the machine in single user mode or passes init=/bin/sh to the kernel to reset the password once and reboots again afterwards.
Only THEN the attacker had access. But yes, he had root. The good (if you want to call it that) part of it is that this procedure rings every alarm possible. The real owner doesn't have the password anymore, as he'll soon figure out. It's everything but sneaky.
I DO wonder why root is allowed to log in at all, though..
Probably. You disabled root login how, via the sshd_config file? If so, you're still screwed.
Even if you fully disable root, that's not going to stop the init=/bin/sh script.
Even if you fix that (securing grub?) you're still screwed because it's a virtual machine, and they can just mount the partition to another VM, and pull all your data/reset root that way.
So, maybe if you have an encrypted partition, no root access, secure grub, and real hardware (it's probably possible to dump the VMs memory by snapshotting it, then pulling the key out that way), you would be secure against attacks like this.
With a VM? No, it's not nearly secure enough for very important things.
Well, having the whole disk dm-crypted is kind of secure I guess. At least I still have no idea how I get at my ssl certification keys from startssl, although I have a dd of that drive in question from the vps provider. I was just too clever thinking of a long passphrase and too stupid to keep at least a hint around somewhere..
Total dataloss for me. But i fyou _do_ remember your dm_crypt password, I think you're safe against these kind of attacks
I would not be surprised if the Linode "reset root password" function shuts down your VM, mounts the filesystem directly on the host, and edits /etc/shadow, and maybe the PAM configs if they're feeling real nice. Using something we can't mount (e.g. encrypted)? Not using /etc/shadow nor PAM? Sorry, we can't help you beyond advising you try and reboot in single user and ssh in to your VM console.
It sucks that money was lost but I can't help but to shake my head at someone keeping something like that on a cheap VPS. It's just stupid to think that was at all safe. That's something you should do on your personal computer where you can assure your security.
I'm not really sure if the author of the article expects to be compensated but if so, he's dreaming. Just read through their terms.
Next time he won't be so ignorant as to put something so sensitive on a server like this.
Saw that too. In past discussions on HN, Zhoutong said he hosted it at Heroku, but apparently they keep the 'hot wallet' alone on Linode instead for some reason, and use that to enable instant withdrawals.
At least Bitcoinica is eating the loss, it's not client money that was directly stolen.
It's quite possible that the attacker has been using the support admin login details for much longer against Linode, without being noticed, until now.
What sort of defenses can developers put in place to protect against admin panels?
I've used these sorts of techniques in the past:
1) Separate username/password system compared to the regular website
2) IP whitelist of who may even access the admin panel
3) Failed login attempts send an e-mail alert with a log entry
We install a little script that runs at boot up to page us if /.expected-reboot isn't present (or removes it if it is). Then, to reboot systems, we run expected-reboot, which is a tiny script that touches /.expected-reboot before calling shutdown.
Wouldn't have prevented this but would likely have paged this unfortunate soul when his machine rebooted unexpectedly.
The comments calling for 'tainting' of stolen bitcoins and blocking their exchange will be the end of bitcoin. The anonymity of bitcoin is only due to general laziness. What happens when the market figures that out? Bitcoin's byzantine agreement is novel, but its crypto is crap.
This is obviously an unacceptable incident. I don't understand how the author can write:
Especially upsetting is that I went to great pains to keep everything as secure as possible.
When that's plainly not true. Surely having a wallet stored on a VPS is a really bad idea, what with admins potentially having full access to hard drive contents? Wouldn't a PGP'd local copy be a better solution, or am I missing a trick?
This reminds me of a situation when I first signed up for linode... my password on my account inexplicably changed one day(I use lastpass so no I did not type the randomly generated password wrong). I contacted support and they fixed it, but I still remember questioning why or how...
Without passing too much judgement........ it's common sense that as your revenue goes up, the time and effort put into ensuring you are on an appropriate platform should go up as well.
Because sh*t happens...... whether we like it or not. Even if the technical requirements are light and it runs fine on a tiny linode, that might not be the right place from a security or integrity point of view, depending on the value of the app.
(for me, a digital wallet worth that much, I'd want at my home..... where I can control it)
Since my $1,000 worth of bitcoins dropped in value to $150 over a period of weeks, I've become significantly less interested in using it as a currency.
You mean less interested in using it as a way to profit from speculation. As a currency it is not as critical that the value only goes up.
A person or merchant receiving bitcoins can easily convert them out to USDs and still lose less in fees than the same transaction would cost compared to accepting a credit card or debit card payment. For example, BTC -> USD at most exchanges is around half a percent.
"As a currency it is not as critical that the value only goes up."
True, but to be a practical currency it is critical that the value remains relatively stable. A currency capable of dropping from $1000 USD to $150 USD in a very short time is clearly not stable.
Well, no. Currencies as we know them are not only for exchange, but also are supposed to be reasonable stores of value. If a currency loses 85% of its worth in a period of weeks, it has failed at this fundamental task.
Now, for merchants this might not seem to matter, as they might be able to always convert bitcoins to a real currency immediately and never hold them long enough for depreciation to matter, but if the currency is excessively volatile there will never be a buyer at the exchange besides speculators, which prevents the currency from ever being really useful.
Which is not saying much, because if you try to avoid volatility by doing a USD->BTC->USD transaction you get hit with fees that are almost as much as credit cards, not to mention the complexity.
You were gullible and invested at the peak of the bubble at $30/BTC (now worth $5/BTC). Any bubble would have crushed you, eg the dotcom stock market frenzy. Your fault.
Bitcoin is up 400% over the last year (from $1 to $5/BTC), which has made it an excellent investment for other (smarter) investors not swayed by a bubble.
Has anyone solved the liquidity mess? Say I want to buy a car and have to unload $20k of bitcoins. Can I do that? With a latency of less than 24 hours? Without getting my PayPal account frozen?
Right. So no liquidity solution then. It's foolish, obviously, because it doesn't work: it looks like fraud. As it does to all the other banks out there. So bitcoins are illiquid. They're like oddball collections. "Valuable" to a few, but basically useless as an "investment" to people with liquidity requirements.
One corrolary you'll note is that because they are both illiquid and volatile, they even suck as a speculative investment: if you make a bad bet you can't get out of it.
Yes you can, but not in 24h. (Hopefully buying a $20k car is not an impulse buy you make in a day, ahem...)
Sell the BTC on MtGox and withdraw the USD via Dwolla directly to your bank account. No need to use Paypal!
MtGox's withdrawal limit can be raised to $10k per day if you provide a notarized government ID copy (IIRC). Dwolla's limit is $5k per transfer with as many txfer per day. So it would take 2 days for completing the withdrawal, plus a few days for your bank to actually post the transaction (thank the legacy financial system for these unexplainable delays).
Of course the very best way to do it is to actually buy a car in bitcoins... see the Bitcoin Market subforum and find a seller. I remember last year someone was happy to announce he was the first person ever to buy a used car with bitcoins.
thank the legacy financial system for these unexplainable delays
I understand your feelings on this. But the fact remains that the using the "legacy financial system" I can move my money between investments on my etrade account with a latency of minutes. I can buy that car on a credit card or with a personal check with zero latency. Bitcoins aren't remotely there yet.
There may be some privacy or social justice reasons behind pushing bitcoins. They may be fun (I'm sure they're more fun for a hacker to play with than mutual funds). But they're not a serious option for someone looking to "invest" their money, and claiming they are is doing the people you're trying to sell on the idea a disservice.
This latency is not a pb inherent to Bitcoin. It happens whenever you hold currency X and need currency Y, for any value of X != Y. You are going to waste time exchanging one for the other. This is one of the reason why I expect Bitcoin's adoption to take off for international trade where the 2 parties of a transaction use different currencies to begin with.
Also I doubt you can sell stocks on Etrade and withdraw dollars to your bank in minutes. When I do this with my stockbroker (TD ameritrade) it takes at least 2-3 days because the transfer is made by ACH which takes a while to clear.
This latency is perfectly acceptable for stock market investors, therefore I see no reason why it would not be acceptable for Bitcoin investors...
My understanding of when they move money around in minutes is that they're basically giving you a short-term, interest-free loan in exchange for using their service. In reality, it still takes a day or two for things to clear.
Right. Because it's a robust system with clear properties and sufficient risk tolerance to make that practical. The bitcoin economy lacks those properties, which is my whole point. And people who plus bitcoins as "investments" without awareness of these issues are hurting, not helping.
Linode compromised! That is important news that concerns me. If the headline didn't mention the BitCoin scam that HN is always pumping would it have made it to the front page? Certainly haven't heard anything from Linode :-(
Sigh. Please give me a definition of 'scam' that fits with bitcoin and not e.g. Apple or Google shares.
Hint: In a scam, there's deceit. The bitcoin devs never deceived anyone. The whole system is transparent, so if there's anyone who bought without understanding the risks, they have no one to blame but themselves.
Perhaps I am just cynical. I did not mean to suggest the devs were scammers. But I still believe the ecosystem as a whole reeks of pyramids and other scams and I am sick of reading about it. I also think Scientology and MLM are scams but there are people who think they are not and we can't all agree. But then I also think casinos and lotteries are scams so I am kind of outside mainstream opinion on a few things I guess.
I was probably too harsh, but frankly, just as you're sick of reading about it, I'm sick of every single thread on bitcoin having that inaccuracy.
What can I say, I'm literal minded - the misuse of words annoys me.
But I still believe the ecosystem as a whole reeks of pyramids and other scams
Oh, sure, that's kind of inevitable, it's a result of the lack of constraints and oversight. But personally, it's a part of why I like reading about it - it still has that feeling of a "wild west", populated by pioneers and thieves. Kinda like the Internet as a whole a few decades ago. Of course, it also means I wouldn't trust it with my money.
I am sick of reading about it.
Sorry, but then... why not just skip the link? There are a few topics I'm kinda sick of too, but I just ignore them.
But then I also think casinos and lotteries are scams so I am kind of outside mainstream opinion on a few things I guess.
Again, my literal mind jumps when I read that ; ) I can completely understand that you consider them immoral and/or predatory, but there's no need to call them a scam particularly - FSM knows there are plenty of other immoral acts.
The lesson repeatedly not being learned seems to be that it's not a good idea to keep wallet files on other people's servers, where you have no control over their security process.
For those of us late to the Bitcoin idea, how does one "steal" Bitcoins? Is it the equivalent of copying someones private key and then deleting all their copies of the key?
I know that bit pin is supposed to be annonomous but is there any way to get these back? I mean is there some logs or if they were signed by his account or anything?
One of the features of the next release of the bitcoin protocol is to allow things like multi-factor authentication (e.g. require a signature from the private key on your computer and your mobile phone before the bitcoins can be spent)
In this case, there's no one to authenticate - it's an automated system that transfers bitcoins. If the application is able to send bitcoins, so is anyone with root over the machine running it.
The attitude that Linode should refund the loss is a fragilising attitude. The more trust you keep pushing onto the provider the bigger everything is going to blow up when something goes wrong.
It had to be rebooted to reset the root password. I see no good reason not to have a decryption key held in memory and require you to log in and enter the key upon reboot for something this important.
The writeup of this is rather suspect. What happened is someone guessed slush's Linode account password, and used the root password reset feature from there.
What I don't understand is why does such a feature exist, why doesn't Linode require >16 character length passwords that are sufficiently random (or eschew password auth altogether), and why does slush (apparently from what I can tell) allow password auth for ssh AND allow root to login on ssh.
The response from linode says that it was a "a customer support interface" that was used to access the account. This seems to indicate an error in their support system rather than someone guessing slush's password.
Yes, the writing is a little incoherent. Maybe that's the reason that caused you to miss that, in fact, someone used Linode's 'Customer Service Representative' interface to get access to his account.
Don't stop reading and comment with 'I call bullshit'.
"As a respected hosting provider, I hope they do the correct thing and refund me for this liability due to their error. Many people trust Linode, and they have proven themselves as a serious contender for hosting critical sensitive operations on the internet. I would hate to not see them live up to that reputation."
"hosting critical sensitive operations" in particular. If you are doing "critical sensitive operations" you need a more secure solution and process which will cost you more money.
Under no circumstances can a hosting provider assume the liability for something like this.
The tradeoff you make for the low cost you pay is that you might have an issue like this because someone screws up.
You pay more for a safe to store your money (and for a safe deposit box to store your valuables) because it's important and you understand the risk involved in not doing that. If you have valuable jewelry many times the insurance company will only insure if you keep it in the safe when you are not wearing it and even the amount of days is specified when it can be out of the safe.
It's unreasonable to expect (and linode's contract clearly states as other's have mentioned) a hosting provider to have a liability over what you are paying them. Edit Add: Unless you specifically have an agreement in advance or that is what they promised or charged you for.
Before anyone reacts to this with any harsh criticism please think for a second what liability you would want for any mistakes that you make with your web startup or idea. You could either be charging zero or charging a small $5 to $20 per month charge. You might make a mistake. Are you willing to accept and even be able to insure for thousands or even millions in liability for those mistakes?
It's certainly a grey area, but at what point is it safe to assume that if you get hacked, it's not going to be because your ISP got hacked? Say this happened to Amazon and it affected a company like Heroku or dropbox, both users of AWS? Regardless of what terms of service says, I'll bet there's some liability somewhere. And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!"
If a bank gets robbed, I'm not liable for the cash they steal. But how about if I've got cash in a safe deposit box and someone uses a fake id to get into it, and the bank doesn't recognize the fraud? That's trickier. And if someone robs my house and I've got a bunch of cash under my mattress, that's another story too. I know the analogy doesn't quite hold up because it's kind of like a bank and a customer engineering a safe together (eg both could be at fault for a break in), but there's got to be some responsibility on Linode's part.
> but at what point is it safe to assume that if you get hacked, it's not going to be because your ISP got hacked
If you co-locate your hardware at a data center and your staff competently secures your systems.
Secure with what? A booby trap? There's always a level of trust you need to have for your provider. Even when you get a cage in their DC.
Set low level passwords and use filesystem encryption. eg: passwords on all networking devices, boot loaders, and BIOS.
Three letter agencies and foreign governments could attack your data if they took it off line, but you monitoring should detect that.
I'd say this is more akin to stashing a bunch of money in a self-storage unit instead of a bank account. One explicitly insures against theft, the other does not. The onus is on you as a customer to decide what to go with.
> One explicitly insures against theft, the other does not.
I'd take issue with that summary. If I put things in a self-storage unit, and it gets robbed because some employee left his master key under the door mat, regardless of how bad of an idea it was to store my money there, that's still their issue.
Think about it this way. I could store my money under a table at McDonald's, in a self-storage unit, or in a bank. Clearly the self-storage unit should provide me some more security than McDonald's. So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault; it doesn't matter that there was a better security option, because it's still below what I was paying for.
"So when an attacker gets access through some really trivial method that they really should be protected against, that's their fault"
A good point but keep in mind that courts don't have the level of expertise to judge what in terms of security is trivial and what is not.
The person trying the case and/or the jury may very well be someone who uses "football" as a password.
Also there are multiple cases of the very best companies with supposedly the highest levels of security getting hacked on a regular basis (might be a small percentage but it always makes the news). Consequently any company defending could make an argument that "this stuff happens even with the best and brightest" and it might be believed. (Well anyway that's what I would argue if I was a lawyer..)
So the public could easily be convinced in the case of a technology company something that would never fly as far as a screwup at the self storage - something physical that they can relate to (like leaving a door unlocked which is easy to understand).
A bank or safe deposit box business will most definitely have clear terms about how liable they are for if something gets stolen.
For a hosting company this is different. Especially because--it's hard to draw a line but I feel it's there--there's a difference between storing sensitive data and storing (what are practically) valuables/money.
I wonder though, I'm not clear on his set up (or business, even), but basically he was literally storing his bitcoin money on a Linode server? Since they're not a bank, nor a safe-deposit service, or are in the business of storing valuables (as opposed to sensitive data). I can't come up with a realworld analogy (they usually break down anyway), but wouldn't you want to wrap this data in an extra layer of encryption or something? It's not that hard to come up with some scheme so that people with root access to the Linode server can't do anything with it either. Since this is about (almost) real money, that's what I'd do.
"And if there's a cut off, maybe linode should advertise that? "Hey, we're cheap, but you get what you pay for!" rather than "You're getting ripped off if you go with amazon over linode!""
Well of course that's never going to happen and the truth is the contracts of the more expensive provider no doubt also limit liability. (And all of this is in the TOS/contract etc). So what we are really talking about is who will do a better job protecting what you have AND more importantly who has more to loose if they screw up. (Small linode has more to loose but they also don't have deep pockets to pay. Amazon has deep pockets but access to magnitudes greater legal help to prevent having to pay.)
I can put a "not responsible for stolen items" sign in my restaurant, but if the coat check employee bolts out the door when you hand them your coat, I'm buying you a new one.
<devil's advocate>Yeah, but could the (ex) coat owner hold the restaurant's landlord liable? Isn't it the restaurant _managers_ problem?
I think there's a _lot_ of "grey areas" here, and while I feel sympathy for the guy who's out ~$13k worth of bitcoins, I can't help but think he was "doing the wrong thing" relying on the security of an inexpensive vps to keep them safe…
What would you recommend? Would you say the same thing if he'd been colocating and a data center employee had stolen his bitcoins? Because that seems far more analogous than any restaurant analogy, and I don't see any reasonable way for somebody who's not a huge corporation to avoid this kind of risk.
You have to trust somebody at some point unless you're keeping the server locked in your own closet. It seems really bizarre to me to say that a hosting provider doesn't have a responsibility not to steal your stuff.
What do I recommend? I'm really not sure…
Firstly, I'd start asking whether a $19.95/month shared hosting* account is a "reasonable" place to store $13k worth of (effectively) cash. I'd be _very_ careful if I had that sort of folding-money-type-cash on hand, and would under normal circumstances automatically deposit in a bank account to mitigate the risks involved with carrying it around. And I'd usually take steps to not ever have that sort of value of cash build up or be required - the only transaction I've ever done of that sort of value in cash is selling or buying a car from an individual - and that's always been a direct from transaction to the bank type of arrangement.
If I had enough bitcoin value that it'd hurt to lose it, I would not (at least now in hindsight) store that on a machine that other people I don't know/trust have root access to. Maybe I'd keep my wallet on a usb stick in my pocket or in a safe at home? I think though that at somewhere near the $13k value the "right" thing to do is convert it to cash and take advantage of the existing banking system and its time-tested security and insurability.
(* Which is fundamentally what a linode VPS is, at least from anyone with access to the hypervisors point of view.)
Then he should Colo with a hosting provider with a contract provision that specifically holds them liable for any losses related to problems caused by the host, and enumerates those possible losses beforehand.
Does such a thing even exist?
Simply not going to happen at any reasonable rate.
There's a saying in many businesses with different variations:
"Price, quality, speed" pick any two.
So this would be:
"Price, security, bandwidth" pick any two.
People pick on price and bandwidth security is taken for granted to be commercially acceptable. The colo can easily figure out price and bandwidth they are clearly defined. Losses from a security breach not as easy - to many variables. Same reason insurance companies love to write life insurance but hate to write disability insurance. Life insurance is absolute (you know when someone is dead and tables exist to compute probabilities on when they will die). Disability is open to interpretation, fraud and other things. It's not black and white.
Would have to meet several tests. 1) Hosting company knew and agreed to the value of the things they had control over. 2) Employee did it and they were negligent in hiring that individual. 3) Reasonable and customary for that type of stuff to be in that situation given pricing and the practices of others.
With respect for #3 it would be reasonable for a bank safe deposit box to contain a $100,000 ring maybe but not to contain a 10,000,000 ring.
What if the coat was full of diamonds?
In an actual court case, the reasonableness of everyone's actions would be evaluated, but it's hard to imagine a court finding it unreasonable that someone placed data worth $13,000 to them on a respected VPS provider.
That doesn't mean Linode has any legal liability in this case, just that your analogy is off the mark.
"it's hard to imagine a court finding it unreasonable that someone placed data worth $13,000 to them on a respected VPS provider."
Really? (I'm reading that as saying you think it _is_ a reasonable thing to store $13k worth of effectively-cash-value in a $19.95/month vps account?)
Does anyone know what regulations like HIPPA or PCI have to say about the security of data stored on managed-by-3rd-party servers like VPSs?
First of all, yes, I think it's reasonable.
Second, where are you getting $19.95/month from, anyway? I haven't seen the plan in question mentioned, and even if this particular VPS happened to be Linode's lowest-end, the last time I looked (a while back, granted), slush had multiple large VPSs with Linode.
Third, really, what does the price of the VPS have to do with it? You think as the cost of the VPS goes down, we're entitled to less assurance that an employee isn't going to bolt with our data?
Finally, HIPAA and PCI regulations are ginormously complex, but violations of them almost inevitably cost a hell of a lot more than $13k.
Interesting.
I'll freely admit I've only been thinking about this since reading this article, so I'm both not-fully-informed and I'm thinking about it as a response to some guy losing ~$13k, but to me it's _not_ reasonable.
(And my $19.95 number is perhaps hyperbolically chosen from their least expensive vps offering - but my assumption would be that the management/hypervisor back end would be shared across their entire infrastructure, so I think my argument holds, in that I'd expect the higher priced offerings to "only" have the employee-reliability-assurance of the cheapest vps…)
Why are you looking at it as "X has the same assurance as Y" instead of "Y has the same assurance as X"?
If you've got a vault that holds a massive diamond, and a little gold ring, do you become concerned because the diamond "only" has the protection of a little gold ring?
Linode plans (like those of other hosting/vps providers) are differentiated on storage/memory/bandwidth. Security isn't a factor. Compare that to an auto where "security" is definitely mentioned and part of the selling proposition and what you pay (they mention theft devices, crash stuff, airbags etc.)
IANAL (or QSA)
PCI doesn't specifically say anything about usage of a VPS. It does however speak about access to data.
If you have encrypted credit card information, you'll be asked to list those that have access to the encrypted information, they encryption key, and the key encrypting key. Then you'll be asked to justify their access.
I'm sure I could come up with with several other major violations, but this alone is severe enough that I can't envision a way that you'd pass a PCI audit.
Sure but there is legal precedent for that as well as it's an easy concept for the everyday man to understand. (Employee and seeing them makes it a open and shut case.)
As an aside, signs like that are the same as ones that appear in auto shops that say "insurance regulations don't allow you in the shop". I owned a company that did manufacturing and said a similar thing because I didn't want anyone in the machine area. I've dealt with insurance companies for many years they don't dictate things like that at least from my experience.
Your example is correct though the restaurant would be liable in the situation you cited because you actually saw an employee steal the coat. If you saw someone else NOT connected to the restaurant AND the coat was expensive you are probably out of luck. It's your property and there is no assumption that a restaurant protects you from acts of god.
The key is was their negligence on the part of the restaurant.
On the other hand if there was a large party at the restaurant and normally a coat check girl that's a different story (devils in the details with this stuff).
What if the person had a $100,000 worth of cash in his coat pocket -- would you reimburse that, too?
He could never prove that he had the cash in his pocket. But you can prove that the bitcoins existed and had not been spent.
"please think for a second what liability you would want for any mistakes that you make with your web startup or idea"
It seems to me that bitcoin wallets are a relatively new and not well enough understood risk. There are very few other "files" like them, in that an attacker copying them can deprive you of their value in a way that you cant protect with backups. I feel a big part of current "internet security best practices" are about minimising the risk of getting exploited - but with a pragmatic limit to how much effort you invest mediated by the excuse of "if we _do_ get rooted, we can always reinstall and recover from backups". It'll only cost you time, and perhaps some reputation, and may put assumed-private-to-you information in someone else's hands, but it hasn't deprived you of access to any of your data. That doesn't apply to bitcoin wallets, and example like this are pointing out flaws in assumptions people are making about appropriate ways to manage them.
It'd suck to be "that guy" who provides the object lesson in why we need to think differently about bitcoin wallets to just about any other file type we might put on an internet accessible machine, but we _do_, and I don't know whether we have an answer to the question "Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"
I _think_ the answer is "if you can't trust those people, you can't risk storing your bitcoins there". There's a reason people keep their money in banks, and not in train station luggage lockers. I'm guessing inexpensive commodity VPS's should be considered closer to storage lockers than bank vaults. I suspect the finance sector and/or fortune500 companies have hosting arrangements with companies offering bank-vault grade protection and reserve bank style insurance - but sure as hell not at $24.95/month.
Don't store your money on servers you don't own. Don't have a web interface. Don't allow anyone to connect. Don't use passwords, use public key encryption. These are the basics.
What if they steal the key?
The key is encrypted with a long and key-strengthened passphrase, so the Universe will run out of energy long before their computers get the key.
But then, you're back where you started. Why not just use a password in the first place?
An interesting point is that for receiving coins and for long-term storage, bitcoin wallets do not need to be online, and in fact do not need to ever have touched a machine that has been online. While slush "just" lost his "hot wallet", another user lost 200k. When will it become common best practice to store high-value accounts entirely off computers? You can print out a bitcoin wallet and put it in a safe deposit box for storage, and still add money to it.
I don't see what's so conceptually new about bitcoin wallets. They're just plain text that you don't want people getting access to. It's no different than storing passwords in plain text: if someone copies them, they're completely compromised (until the user changes them). The solution is pretty simple: encrypt your own bitcoins with your own password (or more ideally, your private key). Then, if someone hacks your server, they don't get anything.
Keys normally grant access so worst case once they're compromised you can take the server offline physically. Even private signing keys can be revoked though some damage may have been done.
With Bitcoin, once it's copied, you can consider it gone forever, irrevocably, in totality.
I thought the idea was if someone copied your bitcoins it was essentially a race to be the first to spend them. I guess if you're stealing bitcoins, though, you're going to be quick to run them through a couple of transactions to claim ownership of them.
(Of course, my understanding might be completely off-base.)
Sorry for asking a noob question.
But what really is a bit coin? I mean in physical existence. Is it just a file(plain text) with some data/metadata?
And stealing it means copying those files, and then deleting the source?
Which in case how is this any different than traditional bank account. My money in the bank is basically DB record. And that can be stolen.
The bank can then just say to every one 'look this transaction from db such and such is no longer valid'.
Can't bit coin do the same? I guess I'm missing something fundamental.
Can somebody explain this?
A 'bit coin' is a space on a block chain that everyone has a copy of. You lock the coin with a cryptographpic key, which you need to store. Whoever has a copy of the key can unlock the bitcoin and re lock it with a different key, such that now only the new owner has a copy of the key. Everyone can still see the entire block chain, but only one account (that no one knows the owner of) has the ability to move that coin.
So they got access to these peoples keys and transferred ownership of the coins.
Most money supplies are regulated, but bitcoin isn't regulated. No one has the ability to say 'reverse that transaction', but it also makes the currency safe from inflation and interference by money printing governments and privacy snoops.
makes the currency safe from inflation
There's an economic myth that inflexible supply of a commodity gives that commodity when treated as money, stability. It does not, as looking at this graph of US inflation/deflation time shows (1944 is when the dollar stopped being gold convertible):
http://en.wikipedia.org/wiki/File:US_Historical_Inflation_An...
A few points:
1. The money supply around a currency not only contains the mined/minted instances of that currency, but also liquid currency-denominated assets, like customer bank balances. So money supply is not necessarily bounded.
2. Inflation/deflation can be considered measures of the change in demand for the currency. In times of deflation, holding money is valuable because it becomes more valuable.
3. From the above graph, you see that during the gold standard, inflation tended to be mostly balanced out by deflation in the long-term, so long-term inflation was low. But in the short term, prices were very unstable as inflation jumped all over the place, and far more unstable even than fiat money in the past three turbulent years that we've seen.
4. From the point of view of an economy, inflation and deflation are not symmetric; because of the value of sitting on money during periods of deflation, savers do not tend to invest their money but move money from investments to cash savings. This undermines economic activity. But in an economy with a rich range of investment opportunities, moderate inflation does not penalise acquisition of money and does encourage investment.
If you want a non-performing store of value and don't mind big fluctuations in value, gold is there and we know how to secure gold rather well. Bitcoins are another non-performing store of value with far more drastic fluctuations in value, and securing it involves the double vulnerability: physical security of storage media, information security of computations involving bitcoins. And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
> physical security of storage media, information security of computations involving bitcoins
Only funds that you have daily access to need be vulnerable to the latter point, as physical security (air-gapping) is sufficient when you do not need to -send- funds.
> And it is much easier to accidentally lose bitcoins than gold, pirate tales notwithstanding.
Strongly disagree. Can you keep N redundant copies of your gold? Combined with secret splitting, you could require that at least K of N secure locations be accessed.
> Only funds that you have daily access to need be vulnerable to the latter point
I'm talking about protocol risk: e.g., the software that implements the protocol on some machine is flawed, so the cryptography can be effectively breached. Or there is some issue with the protocol, like but worse than the issue Kaminsky found with anonymity.
>Can you keep N redundant copies of your gold?
Have you ever found that your backups didn't contain what they were supposed to contain?
Gold gets stolen, but besides such things as costume jewellery, I think it doesn't often get lost.
Not exactly. In order to spend bitcoins (transfer them to other wallets) you need them not encrypted.
So if you are a bitcoin business making bitcoin payments, at moment you cannot avoid the risk of having the wallet stolen if someone gains unauthorized access to your operating computer.
"Is there a way to secure a bitcoin wallet on a machine someone else has root access to (either your datacenter's staff with physical access, or the people with hypervisor access to the hardware your vm is running on)?"
Probably not, _maybe_ trusted computing could help[1]. But it might secure enough if you had a special piece of hardware that stored the bitcoin-key and did all the signing operations. I believe things like these exists for Certifications and other signing-keys. I doubt VeriSign stores their root certificates on just-another-box :)
[1] In theory, TC could "anchor" your data to the hardware chip on the motherboard, and breaking it would require physical tampering.
You're thinking of a hardware security module (HSM). These are standard for e.g. certificate authorities. To the best of my knowledge, they have never been used by bitcoin outfits, but that is certainly possible, in principle.
Of course, sane people don't mix HSMs and VMs.
Still, the amount stolen in Bitcoins here is probably small potatoes next to the value of all the credit cards and personal details stored in other Linode instances. If someone can hack their admin panel and get root access to 8 accounts, they can get access to all the accounts. This time it happened to be done by someone who was going after the Bitcoins. But how can they claim anything in the way of security?
People who host on a VPS typically do so because they want to run complex applications that scale up, and do so without the cost or headaches of adding more physical hardware. Most people who use Linode probably do not do so to host static webpages, which can be done just fine on a shared server for a quarter the price. While Linode might be cheap, they do claim to be secure, and if they didn't it's doubtful people would host their apps there. And most apps do take some personal information; and a large number of them take financial details.
No one would use Linode to host their apps if they thought all accounts were rootable from a master login panel. And no one would use them to host static pages. So who would their customers be if people didn't expect their instances to be secure?
So, a customer service interface was compromised via stolen credentials and used to access various Linode instances. A couple questions that immediately come to mind:
1. Can this interface be accessed from anywhere on the Internet? If so, why? If not, does that mean other systems owned by Linode were compromised as well?
2. Why can customer service representatives access and update servers without the client being notified and with minimal logging?
Regarding #1, an update from Linode was just posted:
"Our investigation has revealed a customer support interface was used to access your account. The compromised credentials have been restricted and we are discussing policy changes to prevent this from recurring."
I'm a Linode fanboy, but we need maximum transparency on what occurred and what's being done. What support interface? How compromised? Who's credentials, etc.
Me too, I've been recommending them a lot and really like their service. I just checked our 2 boxes uptimes just in case.
Hopefully they're working on it, and will give a post mortem once they get it sorted out. I'm inclined to show patience and not demand they do anything other than ascertain the scale of the breach, alert those affected, and secure their systems at this point. Later, they can get into what happened and how they will avoid it in the future.
We can't wait for a full postmortem before Linode says anything.
Linode can't just leave us all wondering about our own security while pouring over over someone else's Pastebins.
Where are you reading this? The status page and the blog have no mention of the incident.
It's from his e-mail conversation with Linode support: http://pastebin.com/UW7iT5fj
So hardly "from linode"
More accurately "according to somebody at linode"
er no, either. ITYM "according to an alleged discussion with a Linode employee".
That update had already been released when I made my original comment (hence why I said "a customer service interface was compromised via stolen credentials"). It doesn't reveal how the credentials were compromised, nor how the attacker managed to use them to log in.
Reading the ticket slush posted it shows no password change logs, if linode was compromised either the whole infrastructure was compromised (unlikely) or a rouge admin or a admin comprimised account accessed the vps and stole the $, as per the bitcoin forums. Total stolen is roughly $16,000 USD
> if linode was compromised either the whole infrastructure was compromised (unlikely)
That's funny. I know from experience in the script kiddie part of the Internet that it was sometimes exceptionally easier to hack entire datacenters (even ones worth millions of dollars) just to get into a few of their customers, especially if those customers secured themselves.
Hosting companies have very sophisticated websites sometimes, meaning that they're almost always vulnerable to something.
I know of an SQL injection in an very large U.S. datacenter's administration panel which has been there for at least six years. Six years and it has not been fixed, and maybe a dozen people have independently discovered it. The deeper you delve, the more you realize that at least a handful of people also have access to important upstreams/backbones.
It's a lot bigger of a mess than anybody realizes. A bit of advice: if you say you're secure, you're either lying or colocating.
Linode will send you a confirmation email if you access the admin panel from a "new" IP. This guy must have had his email address compromised as well.
Looks like a class spear-attack.
So far there are 3 people who've reported their Linodes compromised. They all had popular Bitcoin services running on their Linode.
3 compromised emails? Very unlikely. They are all major contributors to Bitcoin, I think they know a little more than using the same password everywhere.
Linode will only send you a confirmation email if you enable the feature, otherwise tough luck. It's also been confirmed by the vice president of Linode to be a fault on their side.
Fair enough. I stand corrected.
More plausible to have broken web UI security than an entire bitcoin-community-wide targetting.
> Linode will send you a confirmation email if you access the admin panel from a "new" IP.
IF you have the IP whitelisting feature enabled on your account. It is not by default.
> Linode will send you a confirmation email if you access the admin panel from a "new" IP. This guy must have had his email address compromised as well.
The attack was not via the consumer facing admin panel. It was the internal Linode customer service interface.
The OP's tone clearly indicates that he expects some compensation, Linode's TOS are pretty clear: Therefore, subscriber agrees that Linode.com shall not be liable for any damages arising from such causes beyond the direct and exclusive control of Linode.com. Subscriber further acknowledges that Linode.com's liability for its own negligence may not in any event exceed an amount equivalent to charges payable by subscriber for services during the period damages occurred. In no event shall Linode.com be liable for any special or consequential damages, loss or injury.
This also provides an interesting dilemma when it comes to such events. In this case the damage is relatively easily quantifiable, he got X bitcoins stolen so the damage is X times the bitcoin value at that time. Still, it could have easily been user personal data or credit card information, which would have made an evaluation harder to make.
One of the risks of using such a platform I guess and something that anyone who does it should consider.
This is somewhat off topic, but just because the TOS says something doesn't necessarily make it true.
That is very interesting indeed. Holding Bitcoins in a wallet on your server is the equivalent of having actual cash on your server. When someone takes it, there is a definite value of the damages vs. ID theft where the damage could be drawn out over a period of time.
As I consider launching my own Bitcoin business, I have wondered about where to host the bitcoind that serves as my business bank account. This event certainly makes me reconsider just any VPS.
This especially seems hard to certify. Bitcoin aims to be quasi anonymous. Users could just log into their own systems and transfer coins to another anonymous wallet elsewhere and then try and claim robbery. Or be rooted (by say script kiddies) and then take advantage of that, transfer coins, and try and claim for them.
They end result is something like bitcoin would seem nearly impossible to insure for in any reasonable way.
Isn't this akin a bit to storing a bag with $12K in a storage locker in a public space and then asking to reimbursed after robbers broke into the locker and stole the bag.
I think the author is hoping for way too much. The world doesn't work that way, nor probably can or should it.
It's actually a bit worse than this -- basically this is analogous to putting the $12k in a storage locker and then posting on the internet the exact location of the locker.
Given that a bitcoin client was almost certainly running on this box, basically anyone could have connected to the bitcoin network and established this guy's IP address as a target.
If you run bitcoin software, you are advertising your IP address as a high value target.
This is why insurance exists.
I wonder if there are any insurance providers who'd be willing to provider coverage for this sort of event.
Exactly. The attitude of pushing more and more trust onto the provider is just going to cause bigger blowups when something inevitably goes wrong.
Insurance won't insure for what they don't understand and build a risk model for. I can assure they won't understand something like this for a very long time.
There are a couple of insurance companies who will give you a quote on literally anything. You may not like their quote, but they'll give you one.
I'm not sure they need to understand bitcoin specifically to build a risk model, wouldn't they just need general data on losses suffered by various internet hosts due to hacking attacks? Doesn't really matter exactly what is stolen as long as they have a corpus of data on the value of everything that is stolen in this manner. Bitcoin wallets probably fit somewhere in the payoff curve for that.
I've never heard of one, but there may very well be a market for that sort of thing. The problem with Bitcoins will be the legal jurisdictions involved, since these services are regulated nearly everywhere, right?
I write software for use by high risk insurance carrier. There is coverage for just about anything. One of the inland marine insurance types might cover this. (possibly Electronic Data Protection or Valuable Papers)
I am pretty sure this is OP's first experience with ISP contracts. I guarantee it is not Linode's first legal dispute over their TOS with a noob.
> from such causes beyond the direct and exclusive control of Linode.com.
But isn't access to the customer service portal under direct and exclusive control of Linode.com?
Please people (not corporations w/ staffs), do not run critical systems in the cloud.
Get a dedicated server (not cheapest you can find) and secure it with:
(install in this order)
APF - http://www.rfxn.com/projects/advanced-policy-firewall/
BFD - http://www.rfxn.com/projects/brute-force-detection/
rkhunter
Ideally, install rkhunter on fresh system, right after updates, APF, & BFD. Then update the binary check-sums with this command, if you know server is secure:
Update file properties: # rkhunter --propupd --sk
Run a system check to make sure it is known clean: # rkhunter --check --sk
Lastly, sign up for the security alert mailing list for your version of linux on your server.
If you want maximum security, be sure to password protect your boot loader and use an encrypted file system. This will make it very difficult for ISP to work on your server however!
And, I'd add "if any of this is news to you, you should _seriously_ question whether you're skilled/competent enough to be admin-ing publicly accessable servers with files (like bitcoin wallets) that can be valued in the thousands (or tens or hundreds of thousands or more)".
Shipping syslog offsite is a good step, too.
Hmm, for a customer of a cloud provider, this sort of thing will be very hard to defend against.
Maybe if the customer service system had had two-factor security, this might have been avoided (i.e., customer service can access your account only if you read them your hardware token's code).
Requiring SSL/SSH client certificates even for intranet accesses might have deterred this attack.
I hope other cloud providers take note of this incident. This is a very interesting incident.
> customer service can access your account only if you read them your hardware token's code
At the very least, I'd hope Linode implements two-factor authentication for their own logins. A customer-provided OTP would be great but you'd need a customer service reset tool for that when people forget, which would put you back where you started...
Not necessarily if the reset tool is manually driven and audited. The vulnerability we're worried about here is an automated attack against many customers of a single hosting provider.
There will always be ways to human-engineer your way into any single host. Having a hosting provider just increases the attack surface a little.
> The vulnerability we're worried about here is an automated attack against many customers of a single hosting provider.
This was an attack against Linode's customer service systems, which allow their support reps to reset root passwords. There's no reason for that system not to be protected by two factor authentication on top of heavy logging.
You obviously have never worked for a retail ISP.
Actually not. Just use a loopback cypto FS to store the sensitive stuff. The reason they had to reboot the machine is that they just had access to the HDD where they could change the password, as opposed to having live root access.
Indeed. Well-designed money storage systems aren't vulnerable no matter who has console access.
where do you keep the key to the crypto fs?
You memorize it.
how do you transmit it to the server in such a way that it can't be intercepted by someone that compromises the bits of your virtual that boot before the key is uploaded?
By using your public key.
http://www.debian-administration.org/articles/152
You can also require a password AND a cert.
Yes, but will that help you if the attacker trojans whatever it is that is doing the decryption?
I mean, I'm very clearly not a crypto expert, but I do believe that this would be quite a lot like what Bruce Schnier calls 'the evil maid' attack. Instead of having a bootloader, you have a minimal Linux install, then you get a key to that minimal linux install, and that minimal linux install uses that key to decrypt your encrypted disk.
I believe that if that minimal linux install that does the decrypting is compromised before you log in, in theory, the attacker could then insert something in the code that runs after the data is decrypted to, say, send some of that data elsewhere, or, say, open a back door for them to log in and examine the decrypted data.
I mean, certainly, you are making the attacker wait for you to log in (assuming that they've only compromised the admin interface and don't have full root on the dom0) and that's another step you are forcing the attacker to take, and you do at least have a chance then of detecting the compromise and /not/ sending the key, so I'm not saying that it's worthless.
Of course, this is all protecting against a compromise of the admin tools that does not lead to a compromise of the dom0. If the attacker compromises the dom0 without rebooting it or otherwise disturbing the guests? they have access to your ram. They can snapshot both the ram and the disk and take apart the system at their leisure.
I advise you seek the services of a professional sys admin to secure systems.
He's right though.
I don't think I've ever met a professional sysadmin who could defeat the evil maid attack. All the ones I've met would refuse to think about an attack vector if it implied they could not boot their servers.
Yep. Classically, we abdicate responsibility as soon as physical security is breached. With virtual machines the problem is mostly the same, but it moves from "true physical" to "virtual physical" -- once the host environment is breached, all bets are off.
Any sysadmin that claims to be able to protect against a physical access attack or its contextual equivalent is either lying or incompetent. In neither case should that sysadmin be considered "professional".
Use BIOS & boot loader passwords. Encrypt the file systems. This will not stop 3 letter agencies, foreign governments, or aliens.
It won't stop a private individual, either, not even one with a budget of approximately $0. Cold boot and similar attacks, even just yanking the memory or forcing a CMOS reset, are trivial. Please tell me you don't do this for a living.
We are talking about live servers. Monitoring should detect server going down, raising suspicion of a physical attack.
The evil maid attack would be useful against the system admin's workstation. Securing NOCs is beyond the scope of this discussion, but it is not difficult.
Written on a scrap of paper in your wallet. only the password and no other info should be on the scrap.
If you can memorize it, it is a bad password.
That's not strictly true. If you're careful and imaginative, with moderate effort you can commit a fair amount of highly random data to memory. You just can't expect to change it every month.
This may be oversimplified, but it's the correct horse battery staple.
My password was randomly generated, has 85 bits of entropy and, it took me only a couple of days to memorize.
Hmm, hope you don't take many vacations
Eh, it depends, a random fragment of a very long poem for instance can be quite easy to memorize but at 80-90 words/300-400 characters long is pretty damn secure, doesn't even need to be written down then.
https://www.xkcd.com/936/
"muscle memory" helps here. People only need to refer to the scrap of paper for the first couple of weeks of entering a password.
Then that scrap can be securely stored somewhere else. (eg: In your "in case of death" box if you want people to have access when you die.)
What the? If you write it down, it's a bad password. Make it a pass phrase, make it long. And possibly store it in an encrypted password database, protected by a master pass phrase.
I really doubt that "only HDD access" is a secure position in cases where your cloud provider CSR is pwned.
If that's even the case here. We don't have much indication that the attacker couldn't have taken a snapshot of a running system (or that that will be how it happens next time).
So, sure, use a loopback crypto filesystem. It just doesn't provide much in the way of a security guarantee (but it adds some measurable costs).
Yes. Tracing this back to one of their own employees, the company is clearly liable for any resulting loss. If they do not find themselves liable (and a court does not) this will be a big problem for people doing shared or cloud hosting, and traditional, physically secure datacenters will see a resurgence.
How many times does something like this have to happen before people learn to encrypt? Any serious business or financial data should be encrypted, period. Almost all of the major hacks we read about could have been minimized if not entirely avoided if the data was encrypted.
I just read the release from Bitcoinica where they explained that the server accessed contained _only_ Bitcoinica's "hot wallet", and that no code, services, customer data, or other wallets were stored on the server.
If this was the case, why couldn't every access to that wallet, which, assuming the above is true, necessarily occurs on other servers, run a decryption on the file first? Even if you keep the passphrase and/or secret key in plaintext on the machines that run the code, the separation should prevent this kind of rogue access as long as the intrusion is isolated as these people claim.
There is really no excuse just to have a plaintext wallet sitting around anywhere anymore (the official bitcoin client now supports symmetrical encryption). Like credit card numbers, when a wallet is accessed it should be decrypted in ethereal storage like RAM and promptly discarded; it should never hit disk as plaintext. At least the same practices used for PCI compliance and credit card data should be used for btc wallets; preferably better since there is no recourse if your btc wallet is compromised.
I do not believe implementing something like you describe is viable right now.
Although, AFAIK, bitcoin 0.5+ support keys encryption. I am not sure if this available only through GUI or through API as well, but even if it is not available through API, it might be good idea to implement.
>If this was the case, why couldn't every access to that wallet, which, assuming the above is true, necessarily occurs on other servers
From the sounds of it, this was that other server. All it did was operate on the wallet. And if they used other servers, then those would have been the target of the attack.
And, no matter how much damn encryption they have, they rooted the box that operates on the decrypted data, thats game over. The only attacker you would be able to thwart with more encryption would be the one who is able to root a linnode VPS, but unable to extract the key or decrypted wallet from from software running on that box. Sure, there is probably some number of attackers in that space, but security is a game of diminishing returns, and there are different security measures to take that are a much better investment of time than stopping that small slice of people.
My reading was that the Linode with the hot wallet did not contain the software that operated on it. Perhaps it did, in which case you are right.
The encryption wouldn't really help, because the software manipulating with the wallet is on the server anyway.
The server software was used for automating bitcoin spending, too, so you couldn't just save public parts of the wallet on the server.
The only thing that could be done would be obfuscating the encryption key to the data very hard in the code, but that's hardly a security, just obscurity.
I'm not really sure why people are trying to store bitcoins on a VPS in the first place. You can't process credit cards on a VPS and be PCI compliant (it's against the rules), but any moron can do what they want with bitcoins.
This strikes me as a really good point. The onus for proof of merit is on the decentralized currency.
Simply put, people trust Visa and MasterCard to safely manage transactions, for better or for worse. Regulation like PCI helps ensure that this trust is sound.
The Bitcoin community at large could really benefit from a set of published best practices for managing transactions. Anybody possessing an insecure wallet is ultimately a liability to the credibility of the currency.
But all that regulation is evil and it's the freedom of bitcoin that gives it the power*
*for hackers to get away with the entertaining virtual train robberies we've seen in the last year
To be fair, websites were all continuously compromised last years, many of which have nothing to do with bitcoin.
Even so, bitcoin seems to attract every shark in a one thousand mile radius.
It'd be worth it if more people used Bitcoins. No, we don't NEED regulation. We need competence and standards, which can come about without incompetent government intervention.
Nobody said anything about government intervention. PCI DSS is private regulation.
Show me your wallet with a good amount of cash and leave the room for a while.
Afterwards, let's talk about your comparison. Is 'can be stolen' really something that the state can protect you against? Let's discuss it over dinner. Depending on the contents of the wallet I'd pay.
On a more serious note: Your mockery, while amusing, is unrelated to the problem at hand. 'Stealing amounts of $currency from private persons' is not a new idea or something that bitcoin is supposed to change?
What state? PCI DSS is private regulation.
I believe some states have laws requiring parts of PCI DSS to be implemented.
Two problems.
1) I don't think PCI is relevant here. If you store bitcoins somewhere and they get stolen then this is, in my world, cash. It's your very own digital cash. Not a credit card. That's why I constructed a (probably poorly implemented) example of someone leaving a wallet full of shiny $currency notes out there.
2) 'What state?' WTH? Can I reply with 'What kind of question is that?' The state I'm coming from is called 'Northrhine-Westfalia' [1]. Now I'm living elsewhere and there are no 'states' here. I can offer the district 'Tel-Aviv'? The point is, 'what state' is invoking aggressive feelings towards your US-centered mindset.
1: https://en.wikipedia.org/wiki/Northrhine-Westfalia
Oh, FFS. I meant "what state?" as in, "why are you talking about the state?", since you said:
and since the PCI (which was what we were talking about) is private, it doesn't make sense to talk about the State.
US-centered mindset
The fuck? Firstly, I'm European. Secondly, I assumed you were talking about the State[1], not a particular state.
[1]: https://en.wikipedia.org/wiki/State_(polity)
First and foremost: I'm sorry. We clearly didn't talk about the same thing and I misunderstood what you wrote.
My take: Someone was mocking Bitcoins with "But all that regulation is evil and it's the freedom of bitcoin that gives it the power" and I tried to make a point saying that _no regulation is involved here_ (laws? certainly). This is a wallet, it got stolen. Your credit cards are protected, your cash is gone for good.
You invoked PCI and I was (and am) unable to make the connection, maybe again because of a misunderstanding? I'm talking cash. Bitcoins are cash in my world (or - at least their value is equivalent to cash, if you choose to sell them).
From there we went downhill and I overreacted. Yes, for me 'state' is exactly what you posted. Again, sorry for the lapse.
Well, I'm sorry for the confrontational reaction.
I invoked PCI because of the thread: the original post was from klodolph, who said:
(...) You can't process credit cards on a VPS and be PCI compliant (it's against the rules), but any moron can do what they want with bitcoins.
And to that gravitronic replied:
But all that regulation is evil and it's the freedom of bitcoin that gives it the power(...)
"All that regulation" only makes sense if gravitronic is talking about PCI, which was the only regulation cited by klodolph.
Are you sure? I think that you may be mistaken. The bar is just set higher in a "virtualized environment"...
"In a public cloud environment, additional controls must be implemented to compensate for the inherent risks and lack of visibility into the public cloud architecture. A public cloud environment could, for example, host hostile out-of-scope workloads on the same virtualization infrastructure as a cardholder data environment. More stringent preventive, detective, and corrective controls are required to offset the additional risk that a public cloud, or similar environment, could introduce to an entity’s CDE.
These challenges may make it impossible for some cloud-based services to operate in a PCI DSS compliant manner. Consequently, the burden for providing proof of PCI DSS compliance for a cloud-based service falls heavily on the cloud provider, and such proof should be accepted only based on rigorous evidence of adequate controls."
From: https://www.pcisecuritystandards.org/documents/Virtualizatio...
Amazon:
http://aws.amazon.com/security/
Amazon getting a PCI compliance pass was a big deal. The last time I looked, you needed to be able to ensure secure access to the facility, enumerate who has physical access to the hardware and when, and things of that effect. And you need to be able prove all that in the event you're ever compromised.
> The last time I looked, you needed to be able to ensure secure access to the facility, enumerate who has physical access to the hardware and when, and things of that effect. And you need to be able prove all that in the event you're ever compromised.
None of that should be particularly difficult for a VPS provider as large as Linode.
> you needed to be able to ensure secure access to the facility
This part isn't doable if you don't own the datacenter. Slicehost has a problem with this because they never owned the datacenters they used.
You don't need to own the datacenter, but you do need your own, secured space in a datacenter.
Securing physical access is a requirement for anyone wishing to obtain pci compliance.
I'd argue this isn't about bitcoins. A (popular) VPS provider, according to that article, had a security problem that allowed some idividuals to access the VPS management interface for any machine they cared for.
They could've defaced your site in high traffic times. They could've logged in and delete your projects on the VPS. Depending on your setup (they had root) they could've searched for your backups. They could've read your mail, if that machine is your infrastructure service - and continue from there (password reset, amazon, buy expensive stuff. password reset, twitter, damage your reputation).
In this case the bits modified were part of a virtual currency and had a more or less clear value. I'd say there could've been worse results of that security hole though and 'don't put anything on a vps' is not a solution.
It's not about "don't put anything on a VPS", it's about "don't put money on a VPS."
Or passwords, or medical records, or anything confidential or of value?
I bet people think twice before storing medical records on services like Linode. At least I _hope_ they do?
I'm sorry, but you just said the same thing again that I was arguing against. 'Money' is not as clear cut as you'd like it to be.
Bitcoins are no real recognized currency. So you can trade them for USD -> Don't store it?
What about this great project I'm working on? All my stuff on the VPS, because that's convenient and accessible from everywhere. I spent a double digit number of days on it. I have a daily rate for working as a programmer. Don't store it?
You totally ignored (so hard, that I think you didn't read it fully) my post about issues that are harder to value even. Access to your mail can be devastating. Even if you don't store 'money' on that VPS. Putting a dent into your online reputation by messing with your life on the net is hard to value, but certainly damaging. Again, no 'money' stored.
Bottom line: You ignored my point or didn't read my post at all. You picked a line out of context and refuted it with a pointer to the argument _I explicitly tried to prove wrong_.
Bad comparison in my opinion, BitCoins can be stolen (taken away and become unrecoverable) whilst your project you've been working on is recoverable.
Also the value something is worth is what someone else will pay for it. You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.
1) Recoverable: Only if I have backups (which I excluded in my comparison, and would be a fault on my side. But go with me here..). Otherwise I'd need to invest (see the word I used here?) time to create it from scratch. That's equivalent to an amount of money (the exact amount is hard to define, granted).
2) 'You can't value a project you're working on as your hourly rate * hours worked, it doesn't really work like that.' Right. But it's totally okay to value ~worthless~ stuff you have according to market rates, although you didn't sell them yet? Why are we talking about ~12k USD here? That's just a couple of bits and bytes on a disk. Yes, he _could've sold_ that at a specific time for a specific amount equaling ~12k USD. He didn't. Why do you assign this value to a highly fluctuating 'currency' on your disk, but don't like me assigning value to a 'yet to be successful' project on disk?
Point 1 isn't really debatable in my opinion, you make work, you back it up and if someone deletes it off your server you just move it back on. Bitcoins by design don't allow you to recover them.
Point 2, back in the day when currency was backed by gold reserves would you have said the coins and notes people had were also worthless? As they are not 'sold' yet into gold?
Also I don't say a project on your disk has no value, a project on your disk has worth, the amount it's worth is how much someone will pay for it. Because the project on your disk doesn't really abide by any fixed standards you will probably find it's quite difficult to sell it.
Then this conversation should be about the security breach and the BitCoin aspect should be a side note. The data lost is not what is important, just how the breach should be resolved and prevented in the future yes?
They were coins of a mining pool (slush pool - mining.bitcoin.cz - one of the largest three). And they were only the 'hot coins' left on-line for user withdrawals. The majority of the coins are kept offline/cold, as a common security measure for any bitcoin service.
Update: the Linode compromise first reported was that of the "slush" mining pool (mining.bitcoin.cz), reporting a loss of 3094 BTC. Second report was the donation-funded bitcoin faucet, reporting a loss of all of its 5 BTC.
Third report is the biggest, Bitcoinica.com which is arguably the second-largest exchange. Their main site is hosted at rackspace, but their 'hot wallet' was hosted at Linode, and contained 10,000 BTC which were stolen.[1]
1: https://www.bitcoinica.com/posts/warning-please-do-not-re-us...
EDIT: Those not following this incident on the bitcoin forums might be amused that the attacker used the stolen bitcoins to form a transaction with a size of 1337 bytes. That's probably not a coincidence, since the size of bitcoin transactions are usually under 1kb.
http://blockchain.info/tx-index/2893660/d9804de366aa4c2a0156...
> Although passwords are stored using SHA1 with a salt,
Where's the bcrypt/scrypt/whatever police in this comments thread?
I already asked in the comments of the original article how many rounds of sha1 are used. SHA-1 still isn't the best, since it yields to FPGA attacks, but a single round can brute-force all 8 character passwords in less than 2 days on a GPU. My guess is that 10k rounds of sha-1 would probably not be feasible for non-dictionary attacks without specialized hardware.
The article mentions salted SHA-1, which is much more resistant to attack.
Obviously, more rounds and unique salts per user would yield better results, regardless of the hashing scheme employed.
You can salt all you want, but an 8 character password with a single round is going to fall very, very fast. Salt, being public, has nothing to do with it.
It does however mean you have to spend two days per password, rather than two days for the entire user base, or ten minutes with a pre-existing lookup table.
No, consumer GPUs can do almost a billion SHA1 hashes per second now. We're talking seconds to minutes for "complex" passwords, not days.
http://codahale.com/how-to-safely-store-a-password/
GPUs that Bitcoin miners would certainly have around. Not saying that this is what happened, but it got me wondering.
Well using SHA1 with a salt is a lot closer to bcrypt than plaintext or encrypting the passwords.
In my opinion it is not. Here is the common use case for people: 66% of them use 8 character or less passwords. Forget about the common longer passwords like "letmeinplz" just doing a brute force would give them 66%. In terms of loss, even 1% of passwords being compromised is bad since many people use the same password for multiple things.
Now, how long would it take for people to take down bcrypt? in the scale of time bcrypt is all the way over there -> from SHA1 and cleartext. Bcrypt will protect you even as computers get faster. SHA1 a couple decades from now will be be compromised for 9 or maybe even 10 characters. No way with bcrypt.
Your second point makes no sense, bcrypt factors are manually increased and require a new write for every item in the store. If we time capsule the SHA1 hashed db of passwords and the Bcrypt db for 10 years you cannot prove that the SHA1 database will be more easily compromised than the Bcrypt database. There may be a cryptographic break in blowfish in the time.
They don't mention a work factor for their SHA1 hash storage, they could be using PBKDF2 for all we know.
If you're looking for something to pick on about Linode here, go after something else, this isn't the worst thing to come out today.
Trying to find some official information to figure out the status of our own 'nodes. :-(
official announcement: http://status.linode.com/2012/03/manager-security-incident.h...
How did the attackers know what they were looking for. I'm going to assume that it's a small minority of linode users who have bitcoins on their machines. How were just these users targeted so accurately? What tied together knowledge they used bitcoins to those VMs and their linode accounts?
Also, was the nature of the attack just that the were able to login to your linode admin panel and from their root the machines and then loot your wallets?
Simplest answer is probably the right one in this case:
Someone at Linode did it. Ran a script to see how many bitcoin files there were on all the machines (they probably do these types of queries for anti-virus/whatever anyways) and took a customer support password to log in and get the coins. If he did it right he still might be working there, as it is easy to get credentials from friends/coworkers (even though it should be really really hard).
Pretty sure there is a default port that accepts connections as part of bitcoind, so you can just portscan for it.
I'd be very surprised (and suspicious) if they were running any kind of diangostics over their customer's data without an explicit signed contract. The liability worry there alone is scary.
you do realize that in this theory, the same person then went and stole thousands of dollars in bitcoins, right? I don't think they were worrying about liability...
You misunderstand. Not the thief's liability, Linodes. If someone, say, engages in insider trading because of something they saw in Linode's own analysis system, Linode can be sued for failing to protect that information. If they have a policy of never reading customer data (and can prove it) that becomes much harder. The posited "anti virus checker" would throw that promise out the window.
How so? Google credibly claims (somehow) that they don't "read" customer email, even though they run the largest-scale automated email reading system on the planet.
Sorry, there is just no way that this is the case. Please don't throw such a serious allegation out there without any evidence. To even suggest that this is technically possible for an employee to do is a serious allegation, let alone suggesting that someone did it maliciously. This spreads all kinds of FUD.
I'll happily eat my words if that turns out to be what happened, but it is definitely not the simplest answer.
> To even suggest that this is technically possible for an employee to do is a serious allegation
It is technically possible for an employee to do it because it seems (from the linked pastebin above) that is how it was compromised, an elevated account for linode manage was compromised.
As for an employee being the one that did it, that is probably the least likely cause.
I believe he's referring to the part about employees (at least the ones that have access to the customer dashboards) being able to run a script to scan for bitcoins.
Inside job is usually the answer for targeted attacks against inside systems. Inside collusion at a minimum.
I wonder how anyone can trust their linode systems after an admin account being compromised.
It would likely ruin their business to re-install everything, but that is the only way to know root kits have not been installed.
Or just check if your instance was restarted and your root password was changed. If it wasn't, you were not exploited this way.
You can't secure against God. Of course it's technically possible for a Linode employee to do.
Up-vote to the original comment because it's not stupid or impossible, just unlikely.
We can only speculate at this point.
The simplest answer is probably that one of the staff was subject to a targeted hack and a 3rd party gained external access to the CSR tools.
Possibly for an extended period of time. <-- This is the concerning part.
It's relatively unlikely an internal staff member would do something this dumb (but, not impossible. we've had this happen _here_ where I work, with credit card numbers, but obviously the person responsible was caught almost immediately).
They could have looked at the ip's of bitcoin services and noted those running on a linode. Or collected ip's from bitcoin nodes that their own node has seen and looked up those allocated to linode.
>Also, was the nature of the attack just that the were able to login to your linode admin panel and from their root the machines and then loot your wallets?
The way I understand it the attackers were able to get access to the admin panel and invoked some kind of 'change root password' emergency stuff. The machines were rebooted it seems, which makes sense: The interface of Linode has probably/hopefully no access to the root password. Maybe this 'Reset my root' feature (now I'm guessing) reboots the machine in single user mode or passes init=/bin/sh to the kernel to reset the password once and reboots again afterwards.
Only THEN the attacker had access. But yes, he had root. The good (if you want to call it that) part of it is that this procedure rings every alarm possible. The real owner doesn't have the password anymore, as he'll soon figure out. It's everything but sneaky.
I DO wonder why root is allowed to log in at all, though..
"The real owner doesn't have the password anymore, as he'll soon figure out."
He won't figure it out until he tries to login though.
I disabled root login when I was setting up the server. Could my server be affected too?
Also admins that only log with ssh keys and don't use root won't be able to notice that, will they?
1) No idea, that's something Linode needs to answer. I only guessed what it takes to change a root password of a VPS system.
2) Very good point. In that case it might work undetected for quite a while..
Probably. You disabled root login how, via the sshd_config file? If so, you're still screwed.
Even if you fully disable root, that's not going to stop the init=/bin/sh script.
Even if you fix that (securing grub?) you're still screwed because it's a virtual machine, and they can just mount the partition to another VM, and pull all your data/reset root that way.
So, maybe if you have an encrypted partition, no root access, secure grub, and real hardware (it's probably possible to dump the VMs memory by snapshotting it, then pulling the key out that way), you would be secure against attacks like this.
With a VM? No, it's not nearly secure enough for very important things.
Well, having the whole disk dm-crypted is kind of secure I guess. At least I still have no idea how I get at my ssl certification keys from startssl, although I have a dd of that drive in question from the vps provider. I was just too clever thinking of a long passphrase and too stupid to keep at least a hint around somewhere..
Total dataloss for me. But i fyou _do_ remember your dm_crypt password, I think you're safe against these kind of attacks
I would not be surprised if the Linode "reset root password" function shuts down your VM, mounts the filesystem directly on the host, and edits /etc/shadow, and maybe the PAM configs if they're feeling real nice. Using something we can't mount (e.g. encrypted)? Not using /etc/shadow nor PAM? Sorry, we can't help you beyond advising you try and reboot in single user and ssh in to your VM console.
It sucks that money was lost but I can't help but to shake my head at someone keeping something like that on a cheap VPS. It's just stupid to think that was at all safe. That's something you should do on your personal computer where you can assure your security.
I'm not really sure if the author of the article expects to be compensated but if so, he's dreaming. Just read through their terms.
Next time he won't be so ignorant as to put something so sensitive on a server like this.
Bitcoinica just reported losing 10K BTC (worth $50K USD) in this same incident. - http://bitcointalk.org/index.php?topic=66961.msg778254#msg77...
Saw that too. In past discussions on HN, Zhoutong said he hosted it at Heroku, but apparently they keep the 'hot wallet' alone on Linode instead for some reason, and use that to enable instant withdrawals.
At least Bitcoinica is eating the loss, it's not client money that was directly stolen.
It's quite possible that the attacker has been using the support admin login details for much longer against Linode, without being noticed, until now.
What sort of defenses can developers put in place to protect against admin panels?
I've used these sorts of techniques in the past:
1) Separate username/password system compared to the regular website 2) IP whitelist of who may even access the admin panel 3) Failed login attempts send an e-mail alert with a log entry
Any other recommendations or suggestions?
We install a little script that runs at boot up to page us if /.expected-reboot isn't present (or removes it if it is). Then, to reboot systems, we run expected-reboot, which is a tiny script that touches /.expected-reboot before calling shutdown.
Wouldn't have prevented this but would likely have paged this unfortunate soul when his machine rebooted unexpectedly.
As a Linode customer, I'm really looking forward to hearing out their side on this issue
Forum thread regarding this: https://bitcointalk.org/index.php?topic=66916.40;all
The comments calling for 'tainting' of stolen bitcoins and blocking their exchange will be the end of bitcoin. The anonymity of bitcoin is only due to general laziness. What happens when the market figures that out? Bitcoin's byzantine agreement is novel, but its crypto is crap.
This is obviously an unacceptable incident. I don't understand how the author can write:
Especially upsetting is that I went to great pains to keep everything as secure as possible.
When that's plainly not true. Surely having a wallet stored on a VPS is a really bad idea, what with admins potentially having full access to hard drive contents? Wouldn't a PGP'd local copy be a better solution, or am I missing a trick?
This reminds me of a situation when I first signed up for linode... my password on my account inexplicably changed one day(I use lastpass so no I did not type the randomly generated password wrong). I contacted support and they fixed it, but I still remember questioning why or how...
I am off to store some cash in my car and put all my important docs in a bus locker. BRB.
Without passing too much judgement........ it's common sense that as your revenue goes up, the time and effort put into ensuring you are on an appropriate platform should go up as well.
Because sh*t happens...... whether we like it or not. Even if the technical requirements are light and it runs fine on a tiny linode, that might not be the right place from a security or integrity point of view, depending on the value of the app.
(for me, a digital wallet worth that much, I'd want at my home..... where I can control it)
The title reads like a title one would expect from the future.
Since my $1,000 worth of bitcoins dropped in value to $150 over a period of weeks, I've become significantly less interested in using it as a currency.
You mean less interested in using it as a way to profit from speculation. As a currency it is not as critical that the value only goes up.
A person or merchant receiving bitcoins can easily convert them out to USDs and still lose less in fees than the same transaction would cost compared to accepting a credit card or debit card payment. For example, BTC -> USD at most exchanges is around half a percent.
"As a currency it is not as critical that the value only goes up."
True, but to be a practical currency it is critical that the value remains relatively stable. A currency capable of dropping from $1000 USD to $150 USD in a very short time is clearly not stable.
This is unfortunately untrue. Bitcoin has been subject to huge 30% swings in a matter of minutes.
Well, no. Currencies as we know them are not only for exchange, but also are supposed to be reasonable stores of value. If a currency loses 85% of its worth in a period of weeks, it has failed at this fundamental task.
Now, for merchants this might not seem to matter, as they might be able to always convert bitcoins to a real currency immediately and never hold them long enough for depreciation to matter, but if the currency is excessively volatile there will never be a buyer at the exchange besides speculators, which prevents the currency from ever being really useful.
It sounds like Bitcoin is doing much better in the "Medium of Exchange" side of the money coin than the "Store of Value" side.
Which is not saying much, because if you try to avoid volatility by doing a USD->BTC->USD transaction you get hit with fees that are almost as much as credit cards, not to mention the complexity.
> As a currency it is not as critical that the value only goes up.
Not just not critical, highly undesirable.
You were gullible and invested at the peak of the bubble at $30/BTC (now worth $5/BTC). Any bubble would have crushed you, eg the dotcom stock market frenzy. Your fault.
Bitcoin is up 400% over the last year (from $1 to $5/BTC), which has made it an excellent investment for other (smarter) investors not swayed by a bubble.
Has anyone solved the liquidity mess? Say I want to buy a car and have to unload $20k of bitcoins. Can I do that? With a latency of less than 24 hours? Without getting my PayPal account frozen?
What does PayPal have to do with Bitcoins? Trying to buy or sell Bitcoins with PayPal is foolish.
Right. So no liquidity solution then. It's foolish, obviously, because it doesn't work: it looks like fraud. As it does to all the other banks out there. So bitcoins are illiquid. They're like oddball collections. "Valuable" to a few, but basically useless as an "investment" to people with liquidity requirements.
One corrolary you'll note is that because they are both illiquid and volatile, they even suck as a speculative investment: if you make a bad bet you can't get out of it.
Yes you can, but not in 24h. (Hopefully buying a $20k car is not an impulse buy you make in a day, ahem...)
Sell the BTC on MtGox and withdraw the USD via Dwolla directly to your bank account. No need to use Paypal!
MtGox's withdrawal limit can be raised to $10k per day if you provide a notarized government ID copy (IIRC). Dwolla's limit is $5k per transfer with as many txfer per day. So it would take 2 days for completing the withdrawal, plus a few days for your bank to actually post the transaction (thank the legacy financial system for these unexplainable delays).
Of course the very best way to do it is to actually buy a car in bitcoins... see the Bitcoin Market subforum and find a seller. I remember last year someone was happy to announce he was the first person ever to buy a used car with bitcoins.
thank the legacy financial system for these unexplainable delays
I understand your feelings on this. But the fact remains that the using the "legacy financial system" I can move my money between investments on my etrade account with a latency of minutes. I can buy that car on a credit card or with a personal check with zero latency. Bitcoins aren't remotely there yet.
There may be some privacy or social justice reasons behind pushing bitcoins. They may be fun (I'm sure they're more fun for a hacker to play with than mutual funds). But they're not a serious option for someone looking to "invest" their money, and claiming they are is doing the people you're trying to sell on the idea a disservice.
This latency is not a pb inherent to Bitcoin. It happens whenever you hold currency X and need currency Y, for any value of X != Y. You are going to waste time exchanging one for the other. This is one of the reason why I expect Bitcoin's adoption to take off for international trade where the 2 parties of a transaction use different currencies to begin with.
Also I doubt you can sell stocks on Etrade and withdraw dollars to your bank in minutes. When I do this with my stockbroker (TD ameritrade) it takes at least 2-3 days because the transfer is made by ACH which takes a while to clear.
This latency is perfectly acceptable for stock market investors, therefore I see no reason why it would not be acceptable for Bitcoin investors...
My understanding of when they move money around in minutes is that they're basically giving you a short-term, interest-free loan in exchange for using their service. In reality, it still takes a day or two for things to clear.
Right. Because it's a robust system with clear properties and sufficient risk tolerance to make that practical. The bitcoin economy lacks those properties, which is my whole point. And people who plus bitcoins as "investments" without awareness of these issues are hurting, not helping.
Could this have been a vulnerability in Lish, which can be run from a browser using Linode's AJAX console?
http://library.linode.com/troubleshooting/using-lish-the-lin...
I've completely ruined networking and disabled root logins on a Linode VPS, but could still access that same VPS as root using Lish.
Linode compromised! That is important news that concerns me. If the headline didn't mention the BitCoin scam that HN is always pumping would it have made it to the front page? Certainly haven't heard anything from Linode :-(
Sigh. Please give me a definition of 'scam' that fits with bitcoin and not e.g. Apple or Google shares.
Hint: In a scam, there's deceit. The bitcoin devs never deceived anyone. The whole system is transparent, so if there's anyone who bought without understanding the risks, they have no one to blame but themselves.
(Note: No, I don't own any bitcoins).
Perhaps I am just cynical. I did not mean to suggest the devs were scammers. But I still believe the ecosystem as a whole reeks of pyramids and other scams and I am sick of reading about it. I also think Scientology and MLM are scams but there are people who think they are not and we can't all agree. But then I also think casinos and lotteries are scams so I am kind of outside mainstream opinion on a few things I guess.
I was probably too harsh, but frankly, just as you're sick of reading about it, I'm sick of every single thread on bitcoin having that inaccuracy. What can I say, I'm literal minded - the misuse of words annoys me.
But I still believe the ecosystem as a whole reeks of pyramids and other scams
Oh, sure, that's kind of inevitable, it's a result of the lack of constraints and oversight. But personally, it's a part of why I like reading about it - it still has that feeling of a "wild west", populated by pioneers and thieves. Kinda like the Internet as a whole a few decades ago. Of course, it also means I wouldn't trust it with my money.
I am sick of reading about it.
Sorry, but then... why not just skip the link? There are a few topics I'm kinda sick of too, but I just ignore them.
But then I also think casinos and lotteries are scams so I am kind of outside mainstream opinion on a few things I guess.
Again, my literal mind jumps when I read that ; ) I can completely understand that you consider them immoral and/or predatory, but there's no need to call them a scam particularly - FSM knows there are plenty of other immoral acts.
not having your wallet separately encrypted means you're asking to be robbed.
The lesson repeatedly not being learned seems to be that it's not a good idea to keep wallet files on other people's servers, where you have no control over their security process.
For those of us late to the Bitcoin idea, how does one "steal" Bitcoins? Is it the equivalent of copying someones private key and then deleting all their copies of the key?
Generally the thief copies the wallet (private keys) and transfers the BTC to his own account. It's not necessary to delete anything.
Or just copying the key and "spending" it before anyone else can.
No, you use a stolen private key to transfer the Bitcoins to a new public key whose private key is only known to the thief.
I know that bit pin is supposed to be annonomous but is there any way to get these back? I mean is there some logs or if they were signed by his account or anything?
All Bitcoin transactions are publicly logged, but they aren't reversible. So you can see who stole your money but you can't do anything about it.
There is a bunch of violent people who would disagree.
Only if those violent people can locate the thief's physical location, which they can't.
Why the fuck are people putting their bitcoins on servers that they don't control? That's just stupid.
Update from linode: http://status.linode.com/
i think bitcoin could use another layer of authentication to verify the person is indeed the owner of bitcoins.
One of the features of the next release of the bitcoin protocol is to allow things like multi-factor authentication (e.g. require a signature from the private key on your computer and your mobile phone before the bitcoins can be spent)
In this case, there's no one to authenticate - it's an automated system that transfers bitcoins. If the application is able to send bitcoins, so is anyone with root over the machine running it.
The attitude that Linode should refund the loss is a fragilising attitude. The more trust you keep pushing onto the provider the bigger everything is going to blow up when something goes wrong.
LOL, bitcoin.
If this was sensitive data why was it not encrypted?
Replace "bitcoin wallet" with "medical history" or "credit card numbers".
It had to be decrypted to be used. It was in use. Ergo...
It had to be rebooted to reset the root password. I see no good reason not to have a decryption key held in memory and require you to log in and enter the key upon reboot for something this important.
The writeup of this is rather suspect. What happened is someone guessed slush's Linode account password, and used the root password reset feature from there.
What I don't understand is why does such a feature exist, why doesn't Linode require >16 character length passwords that are sufficiently random (or eschew password auth altogether), and why does slush (apparently from what I can tell) allow password auth for ssh AND allow root to login on ssh.
> why doesn't Linode require >16 character length passwords that are sufficiently random
Well, depending on how they got Marek's password, it might not matter. If someone went to his apartment and saw it written down on a post-it...
If they had guessed his password then their login would have shown up in the activity logs for his account, which he indicated was not the case.
The response from linode says that it was a "a customer support interface" that was used to access the account. This seems to indicate an error in their support system rather than someone guessing slush's password.
Yes, the writing is a little incoherent. Maybe that's the reason that caused you to miss that, in fact, someone used Linode's 'Customer Service Representative' interface to get access to his account.
Don't stop reading and comment with 'I call bullshit'.