There are a couple of things I wanted to say, and I can speak with some authority on the subject as I speak on behalf of DigitalOcean.
This was mentioned to me on twitter hours ago, prior to this post. The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible and I would be amiss to point that if anyone did ever find a software vulnerability filing it and waiting 24 hours for the appropriate response is preferred. - https://www.digitalocean.com/security
As far as I can tell here, there is no unexpected behavior that isn't documented or stressed. In both our API documentation, and our control panel, we note that you must either pass a flag or check the box to security delete the data. As far as I can tell, the flag is currently functionally correctly. so..
Is the complaint that customer data is being leaked from VMs? That the flag being passed via our API/Dash isn't actually working? Or, that our policy on not doing a secure delete by default isn't something you agree with?
j.
Any company that has ever stored data on DigitalOcean now needs to operate under the assumption that other DigitalOcean customers have accessed it.
Even if every staff member believes they checked the "Scrub Data" checkbox or used the API flag when destroying droplets, human memory is unreliable and people make mistakes.
This is a very serious security issue and it's appalling that anyone is making excuses for it, and it's even more appalling that the company responds by blaming customers.
Customers should not be able to access other customers' data under any circumstances. It shouldn't even need to be stated that providing access to other customers' data should not be the default.
"Or, that our policy on not doing a secure delete by default isn't something you agree with?"
This one. You have chosen a default that fails deadly. It's like designing a car that explodes when you turn it off. Oh, there's a button over here you can push to disable the explosion feature. That doesn't really make it better.
You've created an option that can and will deeply screw many of your users. The mere existence of the option is not wrong by itself. But the fact that it can and will so easily screw so many people means that the option needs to have lots of flashing warning lights around it and it needs to be on by default.
I just checked out the "Destroy" tab for my droplet. There is absolutely no, none, zero indication that failing to check this box will allow the next person to occupy my spot to read all of my data. Here is the exact text:
"This is irreversible. We will destroy your droplet and all associated backups."
"Scrub Data - This will strictly write 0s to your prior partition to ensure that all data is completely erased. Estimated Destroy Time: 11 minutes 22 seconds"
I would expect "destroy your droplet" to mean that the data gets destroyed. I would expect the "scrub" option to be for paranoid people worried about the FBI seizing your equipment and using electron microscopes to extract residual data. At no point does anything in here give me any expectations that the default is "hand over all of the data currently on the VM to the next random stranger who walks in the door".
Do you really speak on behalf of DigitalOcean? If so, you need to get your head straight fast, because this is not even remotely acceptable. You cannot defend the current practice, because it is not defensible. If you don't understand why that is, you need to sit down and think about it until you do.
Right now, as a customer of yours, my thought is this: if you think this isn't important and doesn't need to be called out, what else have I missed? What other crazy data leaks do you allow by default with the defense that I could turn them off if I cared? I hope and assume the answer is "none", but now I'm rather worried.
I can kind of sort of understand how one might end up building a system like this, thinking that it was a good idea at the time. But I cannot understand at all how someone could possibly defend it once it's pointed out that it's terrible.
Hey Mike,
I'm going to give you a call later this afternoon but I wanted to clarify. First, Moisey and I worked on this this morning, so worth a read: https://digitalocean.com/blog/
Second, the way this was approached was super super confusing, originally as it was in 140 characters, on the twitters. Maybe my short comings for not totally understanding the situation before I spoke, but information was fairly fragmented.
First: Yes, I do speak on behalf of DigitalOcean.
My original understanding was that this issue was with the secure delete flag not working when being passed. This promoted me to request, and continue to request, that if it was the case, security@ was notified with an outline of what is going on per http://digitalocean.com/security/ - If it's an expected behaviour, while still not good at all, it isn't my call nor was I prepared to call the company into the office at midnight on a Sunday knowing we would issue a software update in the morning. Had the flag not been respected, I would have immediately called the senior engineering team as well as Ben and Moisey, so fully understanding the situation was very important.
As a customer, I'd like you to know we do take security very very seriously, it's something we discuss going into everything, as we appreciate healthy conversations about the way our product works.
I spend 4 hours last night trying to figure out exactly what was going on, it felt very difficult to get a straight answer of "your policy fucking blows and you better change it tomorrow" - That's something I can take to the bank, but I'm sorry if I wasn't clear.
j.
Thanks. I think your initial reply here was way off base, but I see that you guys have done the right thing now, and that's commendable. There's nothing more I could want that doesn't require time travel.
Regarding the call, just e-mail if you want to get in touch directly for whatever reason. But I think we're square.
Appreciate it.
>The first thing I said is that most people these days understand the importance of a responsible disclosure, and that we take all security issues very seriously. Not following responsible disclosure with a company such as DigitalOcean is extremely irresponsible
Oh bullshit. Don't deflect the issue here by complaining that you don't like full disclosure policies that many security experts agree with. (Such as, I don't know, Bruce Schneier?) If you want to get into secondary levels of annoyance, how about the fact there have been multiple instances in the past with DO that were only resolved by open/full disclosure on forums?
As someone who isn't a DO customer, the thing that most dissuades me from this about becoming a customer is that this is a case of insecure by default. How many more of those are lurking around?
Here's another compound fuckup for you:
1. DigitalOcean users are unable to install their own kernel updates!
2. DigitalOcean have to bother making a new kernel image available via their admin interface; they haven't done this for over six months of Debian kernel updated in my experience.
3. Even if DigitalOcean did make a new kernel available, there's no notification to inform the customer that they have to log in to the admin interface and pick the new kernel from the list, then reboot their VM.
4. The list of kernels in the admin interface is sorted... bizarrely. I check it every so often and there is no sensible overall naming scheme; you are presented with a popup menu listing every single kernel for every single distribution; the latest kernel for Debian is in the middle of the list.
5. My attempt to resolve these issues with DigitalOcean support covinced me that the person I was corresponding with has no idea what a kernel even is, much less that DigitalOcean's list of available kernels is... lacking.
This situation, plus a couple the longstanding lack of progress towards IPv6 support; lack of ability to control kernel parameters; lack of a way to snapshot the filesystem for backups; makes me an unhappy DigitalOcean user who is going to jump ship for Bytemark at the earliest opportunity.
Funny, I've been able to update off the standard package repos
You may have updated the package, but did you really boot from it? Run 'uname -v' to check:
DigitalOcean systems do not boot from the kernel image installed within your VM; they are externally provided.
This reminds me of something I omitted from my original rant. I've actually had to pin the kernel image package that I've got installed on my VM to the version that DigitalOcean provide:
Because an unforseen ABI break in some netfilter module means that if I install the newest package, then reboot, one of the modules used by my iptables setup fails to load. ferm notices this and rolls back my firewall configuration--to the default state which allows all traffic. I noticed this, but I wonder how many other customers with similar setups did not, and hence have not noticed that their iptables rules are incorrect or absent.
Oh FFS. You can't whine about "responsible disclosure" while saying "As far as I can tell here, there is no unexpected behavior that isn't documented or stressed."
If it's documented, you've already disclosed it yourselves.
So you're saying, in the same response that 1) there was no "responsible disclosure" and 2) that this wasn't actually a security issue. You can't have it both ways.
"I never borrowed your saw, and anyway it was broken when you lent it to me."
As a DO customer I find this completely unexpected and unacceptable. Secure delete should be always done before transferring to another customer, if this takes time it should scrubbed as queued job, before the next customer gets it.
Agreed (also as a DO customer), DO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270
It's a win-win :)
> Or, that our policy on not doing a secure delete by default isn't something you agree with?
This one. Choosing insecure defaults for a virtualization API is a Bad Idea. As a rule of thumb (to put it bluntly), people are dumb. If you give them a loaded gun, they will shoot themselves with it. And they will blame you for it. At least put the safety on and make them take a conscious step before blowing their face off. Don't mean to tell you your business, but seriously, insecure defaults are a Bad Idea for a virt API.
While you're totally right, this doesn't even come down to "people are dumb". The documentation is simply lacking here. At no point that I can find do they discuss the ramifications of not using the scrub option, and even a smart person could reasonably expect that not using that option still doesn't leak your data, just in some other way (one less safe against certain attackers, presumably).
It doesn't seem like a security issue by DO in any way. It seems that the fog api (the project this 'issue' is filled for) doesn't allow a user to access the required flag to scrub the drive. Not a DO problem in any way I can tell assuming the scrub parameter is working correctly.
This is a well-documented situation that almost every provider of 'consumable infrastructure' before DigitalOcean came along has faced and solved.
It is disheartening to see the same mistakes being made.
Whilst I absolutely see their USP was always solid-state storage, and that has pitfalls in terms of how you can scrub data to avoid it being leaked, the platform should take every precaution to protect customers data.
There shouldn't be an option to 'scrub data' and it shouldn't be defaulted to off so they can save some hassle, and avoid spending a few dollars. It shouldn't be an option because it should be on all the time, anything else is surprising the customer mightily.
"What do you mean, my data leaked? Oh that's fine" -Nobody
"Why'd you pick a provider who doesn't take our companies information security seriously?" -Every boss anywhere
It is absolutely a DO problem in two ways: 1) they default to bad behavior and 2) their UI does not make it at all clear what the consequences of that bad default actually are.
I have to agree with the other posters, this response is disappointing. The initial concern the other posters have is obviously the lack of a secure default.
Now i've known about this for quite some time and I mostly use DO for development and testing Chef recipes so I don't have an issue with it being off by default. I love the transparent $5 pricing. But I also don't store sensitive information on DO, yet. I also assumed when I was a newbie any sensitive info would be deleted.
The easiest option would be to include a notice next to that checkbox that if there was any sensitive information you should select this option. I understand that SSD wiping would drive up your costs. Another option would be to switch to mechanical hard drives. I don't how easy that option is with your setup.
DO please follow the instructions of Aaron Friel: https://news.ycombinator.com/item?id=6983270
It's a win-win :)
Your attitude and response here is exactly why disclosing this was in fact "responsible".
I asked three questions so I could address the title of the tread, that our users data is being leaked.
I presume the audience felt a Chief Technology Evangelist had more control over the tone and intent of his messages on the Internet, and therefore actually meant to couch his three questions within pointy remarks about responsible disclosure of a problem which was announced as fixed by DigitalOcean and reported as such in the press over 6 months ago.
You either know or should know the answers to the questions you've asked in plainly bad faith, as the problem has been clearly described both here and in the linked github issue, which isn't even filed with DO because DO has already made it crystal clear you don't care about having secure defaults in this matter.
The issue was instead filed against fog, so that users of that library may be protected to the extent possible under the circumstances.
In other words: This isn't your issue anymore. You've already publicly dismissed it. Worse, you've gone back on an earlier promise about it.
It is now in the hands of the community to try and protect your customers, since you have refused to.
To be perfectly frank, I think the title of this post is totally disingenuous and started the wrong conversation, and that makes me sad because I know we do care about our customers, and their security. Had the title said "DigitalOcean doesn't care about it's customers security" I'd have been happy, because that would start a conversation we should probably be having about how data is deleted.
As it seems like this is actually an issue with people not liking how our product works, I've begun the internal conversation going about two things:
First, communicating again to our customers by way of a blog post that this is how the production functions, as well as highlighting any relevant tutorials.
Second, working with the product team and engineers to either reverse this functionality at best, at minimum draw greater attention to it.
j.
Not EVER presenting one customer's data to another customer is a basic part of any business involving multiple customers. In our case we store the UserId in every database table along with the other data, and validate that against the actual logged in user before returning it. It's why I was so horrified at a bug in Cyrus IMAPd replication which occasionally overwrote files belonging to other users.
And it's why you are wrong. Giving the same data back to the same user (holding their blocks) would be fine - but allowing customer data to be read by other customers, for any reason, is bad practice in any area of business.
In many sane jurisdictions, the practice when selling something is to factor the cost of eventual disposal or recycling into the initial purchase cost. This is required by law, for example deposits on drink bottles which can be redeemed by returning the bottle.
In your case, the honest thing to do would be to factor the eventual cleanup of data from the disk into the initial purchase cost of the service. So the cost to provision a VM would include the wipe cost.
Pointing out that you don't do that is a community service. Congratulations to the author of the post for noticing the issue and bringing it to everyone's attention. Now we can all make an informed decision.
"at minimum draw greater attention to it." No that is not sufficient. You have to fix this, and you should have said so even if it was early on Sunday morning.
In my opinion implementing the deletion of data in a way that can be forgiven by the user (and even worse API) is a bad idea. It's like burying unexpected clauses in the middle of a 50 page terms and agreements.
It's definitely not illegal or reprehensible, but anyone doing *aaS has exponentially more knowledge than it's users and knows it. Anyone who has once tried not to impose minimum password complexity knows what I mean.
To me it seems that data cleaning should be something you choose as you purchase the service. I understand that cost might be a factor in this particular case, but in then why not communicate about it and make a premium or discount system? It would probably come in good light to both users with sensitive data and those who don't care.