Anthropic's oss-scanner includes patches. We've received a batch and I'm working on some of it.
A good amount of the patches is straight garbage. It's not helped by the fact that a good amount (20~30%, maybe more) of the issues is also junk themselves.
I'll hopefully be able to share numbers (or data) after we're done with the batch.
I won't claim that my experience is representative, but what I've seen is closer to 50% being false positives, and then maybe 20% of the remainder being bugs but not vulnerabilities (aka lower priority bugfixes). The suggested patches have a much worse hit rate though, to the extent that I accepted only a small fraction after reviewing.
That said, an automated system generating only 30% true positives is probably still ahead on ROI as long as we're talking about tens of findings and not hundreds. The amount of human time spent investigating things that turn out to be false positives being the main difference for how over or underwhelming it is.
Yeah, I was being "optimistic" in my estimate. And of course with a batch, you start at the high severity end, so it gets worse as you progress through. (Well, I assume at least, even some of the "critical" was just plain "not even a security issue" in our case.)
70-80% not being garbage sounds like a good thing :)
I suppose it depends on:
- How many patches you receive
- How easy it is/long it takes to distinguish the good from the garbage
You're assuming the other 70-80% is actually good. It's not, there's a whole spectrum between "junk" and "good". A whole bunch of this stuff is "technically correct, technically even a security issue, but also just plainly not of actual importance".
You get free meals, 70-80% of them tasty on different degrees, and the rest burnt. That's OK because free meals, right? Except the small letters say you have to eat 300 meals a day.