I ran it against a few servers I'm responsible for. I'm not impressed.
I got a large number of reports for a Debian system running Apache. These were all old vulnerabilities where fixes were backported to Debian's packages, so these are false positives. Also got a warning for another server about the recent XMSS issue in OpenSSH, which is code that is disabled by default (and disabled on the scanned server).
It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability. It's not a "vulnerability scanner" in a sense that it actually tests for the presence of a vulnerability.
(Full disclosure: Just copied over my comment I posted on reddit earlier today)
> It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability
It sounds as though it's illustrative of a problem with either the index the scanner software is referring to or the update methodology the distro relies on. Debian don't really fix the software without incrementing the version number in some fashion, do they?
It's pretty typical for these "backported" packages to have version numbers that only differ from the original by a suffix, and that suffix often only appears in the package manager.
For example, the Debian stable Apache package is currently version 2.4.38-3+deb10u3, but Apache will typically report something less than all that information, like just 2.4.38.
If it's really just looking at exposed version numbers, there's plenty of other problems, like servers that are configured to not expose their version numbers.
This has been my experience with similar tools (mostly openvas / greenbone) and companies peddling vulnerability tests for compliance purposes. There doesn't seem to be a lot of "functional scanning" in this space, like you're talking about.
So I agree the false positives are a pain in networks with patched systems but this also seems to be standard behavior.
Yes it’s unfortunately standard for expensive scanners too.
Our goal was to make something easy to run and deploy with similar results to something like openvas or Nessus.
I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing, so they dial down the accuracy. This works using the same methods as a Nessus, banner grabbing and indexing issues.
We are going to keep working on this and look for ways to make it more accurate, but I’d say our results with this versus our expensive scanners was pretty much the same.
Yeah. FWIW those openvas reports really satisfied the C-level itch to have some kind of work product in the form of an official-looking report, and your looks identical. Overall you seem to have built an awesome toolkit to start with. Greenbone is a total pain to get working initially and openvas updates are broken out-of-the-box. If I were still doing this kind of stuff I'd be immediately trying out Flan Scan.
"I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing"
That might be a bit cynical, not sure I'd call openvas expensive.. I think it comes down to the fact that the damage of a single false negative can far outweigh the cost of the false positives. In the end, these scanners are mostly a tool to help document your security efforts, often for compliance reasons, making you think about and document the choices you make from a security perspective.
Definitely a little cynical =]. I think the same thing about a lot of 3rd party security consulting.
OpenVAS is free, but the big issue we've had with it is the complexity of setting it up and maintaining it. We are a security team, and would rather not spend our time managing servers, especially since we aren't the best people to do that at Cloudflare.
Are you in a position to share that session online or perhaps a blog post on your experience and approach ?
I've looked at it (openvas), got something working, but was never happy with it and ended up returning to a simpler/proven nmap base that I could manage better and add complexity if/when needed.
I was planning to do a blog post in about a month's time. The DC4420 meetings and/or talks are not recorded (luckily!), but I intend to polish the talk up for a future re-run.
On the other hand.. I do have something that might be enough to get you going. The setup we built is open: https://github.com/smarkets/vuln-scanner - go have a look.
The glue code has comments on some of the stranger bugs I had to work around. So does the readme. If something isn't clear, feel free to ask.
One of my challanges was understanding the the zoo of tests OpenVAS would run and trying to reliably select which ones to apply. Did you, or anyone here, ever spot a way of outputting all the tests (nmap scripts etc.) that a particular run would trigger (but without actually running them)
Yup.... I haven't encountered products yet that actually report useful information on systems that receive regular automated software updates. All you get are loads of false-positives.
Most vulnerability scanners with functional scanning capabilities are run without them. Enterprises are concerned about the effect of functional scanning upon their production systems, is it going to bring something down, and functional scanning often requires a level of system access which enterprises are usually not comfortable giving. Since they are rarely used vendors don't invest heavily in developing them.
My experience in corporate and government is the opposite. They require functional scanning. Always.
But they also required expensive functional scammer/scanner subscriptions. I see this basic tool plus some scraping of changelogs (included in suse and redhat binary rpms) for cve numbers should substantially reduce the false positives. For the price of another 50 lines of python.
Don't know what other distros or the BSDs have available.
We have had completely opposite experiences then. I have almost never never had a corporate or government entity require a functional scan in a production environment. Some have started off asking for one but in the end I have never had one actually do one. They always end up balking at the need to supply credentials and are uneasy with the risk to the stability of production system inherent in a functional scan. Instead they settle for the non-functional scan from the expensive subscription tools such as Qualys which they use.
The majority of time these reports are being produced to fulfill an auditing or FISMA requirement and not considered worth the risk to systems a functional scan could present.
But what else would you propose? Only thing you could do which would be more accurate is to try to exploit all the issues it found, which isn't really practical, because POCs are not always available and if they are, you would have to build some kind of database with all of them. Maybe it could be combined with Metasploit...
Tools like Metasploit can often be destructive, not sure I would want to run them against production by default.. A more reasonable approach would be to set up a lab when needed when one of the less invasive scanners flags a potential issue.
I think I'd prefer a tool that identifies points of interest for my team to investigate rather than one that actually tries to execute a hack against my servers, introducing possible instability.
Similar experience. Many scanners seem to be designed around showing 'value' to management vs real value for engineering and security teams. Another example is Blackduck. Its slow, noisy, has tons of FPs (which you cannot globally ignore), etc. It ends up being a tool users fight against instead of work with.
Another caveat -- scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. Its a mixed bag -- at scan time the system may be secure, but a future change may result in a vulnerable system. Unless you're religious about scanning after each build/config change; this could bite you as well. It would be nice to know these minutiae without digging into each CVE report.
> scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability.
This is a straight-up artifact from the fact that the scanners (which essentially have to operate in non-destructive mode) rely on banner grabbing and very basic service behaviour pattern matching. Hence for every service they find, they report - quoting your words here - the potential vulnerability.
Which basically is what you want. Trying to intelligently reduce noise would be really cool, but the moment the intelligence suppresses a finding that you WOULD have wanted to know about, you're screwed. It's bad either way, but the worst-case scenarios of the two are asymmetrical.
I personally find point-in-time scans a useless exercise. Findings from individual cans are one thing, the difference over time is what really matters.
Remember: This is a 50 line Python script that is a wrapper for nmap + vulners NSE script.
It's not a product, but some simple automation around existing tooling. Better than paying $$$ for a full fledged scanner? Maybe, but depends on your use case.
Flan Scan is a thin wrapper around Nmap that converts this popular open source tool into a vulnerability scanner with the added benefit of easy deployment.
Vulners.com isn't a free service, are there other cheaper alternatives (perhaps less comprehensive) that maintain database of vulnerabilities?
Interestingly, the very impressive infosec people behind Vulners are employed (?) by QIWI, a Russian payments company. That isn't an issue now that you publicly claim to use their service, but were there any reservations raised by legal or otherwise?
How much in relative terms were the cost savings when Cloudflare did switch to in-house audits via Flan Scan given the requirements / development / operational / maintenance effort expended + licensing service from Vulners?
Does Flan Scan also scan network equipment (like switches, routers etc)?
Given the complex heterogeneous nature of the global Cloudflare network, what did the deployment process look like? Will there a follow-up blog post on how that was automated/accomplished?
What are the other big cloud / CDN providers doing to scan for vulns / compliance at scale if you're privy to it? Have any of them shown interest in contributing to and/or using Flan Scan?
What does the short-term and long-term roadmap for Flan Scan look like?
That LaTeX report output is hot. It'd be cool if the bar could be raised for scanners like this, which mostly just try to get services to respond with a version number.
It is always funny seeing how less than a hundred lines of code can replace six or seven digits in licensing, negotiation, lawyering, and time.
Cool project. I'd be interested in using the scan data to automate workflows. Is Cloudflare hooking this into some security automation engine to make the data actionable?
The article essentially reads as "we payed lots of monies for a mediocre scanner, and then we discovered that the FOSS nmap did everything we needed. So we took nmap and added a little bit of extra, a web interface, and gave it the name Flan Scanner."
I ran it against a few servers I'm responsible for. I'm not impressed.
I got a large number of reports for a Debian system running Apache. These were all old vulnerabilities where fixes were backported to Debian's packages, so these are false positives. Also got a warning for another server about the recent XMSS issue in OpenSSH, which is code that is disabled by default (and disabled on the scanned server).
It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability. It's not a "vulnerability scanner" in a sense that it actually tests for the presence of a vulnerability.
(Full disclosure: Just copied over my comment I posted on reddit earlier today)
> It seems all this tool does is some kind of version matching (i.e. "a CVE has been reported for version x.y.z of software A, so a server running x.y.z is vulnerable"). This is a poor proxy for the actual existence of a vulnerability
It sounds as though it's illustrative of a problem with either the index the scanner software is referring to or the update methodology the distro relies on. Debian don't really fix the software without incrementing the version number in some fashion, do they?
It's pretty typical for these "backported" packages to have version numbers that only differ from the original by a suffix, and that suffix often only appears in the package manager.
For example, the Debian stable Apache package is currently version 2.4.38-3+deb10u3, but Apache will typically report something less than all that information, like just 2.4.38.
If it's really just looking at exposed version numbers, there's plenty of other problems, like servers that are configured to not expose their version numbers.
This has been my experience with similar tools (mostly openvas / greenbone) and companies peddling vulnerability tests for compliance purposes. There doesn't seem to be a lot of "functional scanning" in this space, like you're talking about.
So I agree the false positives are a pain in networks with patched systems but this also seems to be standard behavior.
Yes it’s unfortunately standard for expensive scanners too.
Our goal was to make something easy to run and deploy with similar results to something like openvas or Nessus.
I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing, so they dial down the accuracy. This works using the same methods as a Nessus, banner grabbing and indexing issues.
We are going to keep working on this and look for ways to make it more accurate, but I’d say our results with this versus our expensive scanners was pretty much the same.
Yeah. FWIW those openvas reports really satisfied the C-level itch to have some kind of work product in the form of an official-looking report, and your looks identical. Overall you seem to have built an awesome toolkit to start with. Greenbone is a total pain to get working initially and openvas updates are broken out-of-the-box. If I were still doing this kind of stuff I'd be immediately trying out Flan Scan.
"I personally think accuracy issues are because security vendors don’t want to show that their expensive software found nothing"
That might be a bit cynical, not sure I'd call openvas expensive.. I think it comes down to the fact that the damage of a single false negative can far outweigh the cost of the false positives. In the end, these scanners are mostly a tool to help document your security efforts, often for compliance reasons, making you think about and document the choices you make from a security perspective.
Definitely a little cynical =]. I think the same thing about a lot of 3rd party security consulting.
OpenVAS is free, but the big issue we've had with it is the complexity of setting it up and maintaining it. We are a security team, and would rather not spend our time managing servers, especially since we aren't the best people to do that at Cloudflare.
Yeah, OpenVAS is a nasty piece of work to set up and operate reliably. Especially headless.
Shameless plug: I got it to work. Will be doing a talk about the experience at week's DC4420 meeting.
Are you in a position to share that session online or perhaps a blog post on your experience and approach ?
I've looked at it (openvas), got something working, but was never happy with it and ended up returning to a simpler/proven nmap base that I could manage better and add complexity if/when needed.
I was planning to do a blog post in about a month's time. The DC4420 meetings and/or talks are not recorded (luckily!), but I intend to polish the talk up for a future re-run.
On the other hand.. I do have something that might be enough to get you going. The setup we built is open: https://github.com/smarkets/vuln-scanner - go have a look.
The glue code has comments on some of the stranger bugs I had to work around. So does the readme. If something isn't clear, feel free to ask.
Thanks, looks promising.
One of my challanges was understanding the the zoo of tests OpenVAS would run and trying to reliably select which ones to apply. Did you, or anyone here, ever spot a way of outputting all the tests (nmap scripts etc.) that a particular run would trigger (but without actually running them)
Sadly no. We tried to figure out a way to reliably and permanently disable a whole suit of test scripts, but that got surprisingly fiddly.
I think there might be a way to choose categories to include/exclude but haven't had the time to actually investigate.
This is accurate. Various types of scanners and sensors are standard requirements for even the most basic SOC2.
Yup.... I haven't encountered products yet that actually report useful information on systems that receive regular automated software updates. All you get are loads of false-positives.
In some industries you actually have to document and prove that. That's where these tools shine.
Most vulnerability scanners with functional scanning capabilities are run without them. Enterprises are concerned about the effect of functional scanning upon their production systems, is it going to bring something down, and functional scanning often requires a level of system access which enterprises are usually not comfortable giving. Since they are rarely used vendors don't invest heavily in developing them.
My experience in corporate and government is the opposite. They require functional scanning. Always.
But they also required expensive functional scammer/scanner subscriptions. I see this basic tool plus some scraping of changelogs (included in suse and redhat binary rpms) for cve numbers should substantially reduce the false positives. For the price of another 50 lines of python.
Don't know what other distros or the BSDs have available.
We have had completely opposite experiences then. I have almost never never had a corporate or government entity require a functional scan in a production environment. Some have started off asking for one but in the end I have never had one actually do one. They always end up balking at the need to supply credentials and are uneasy with the risk to the stability of production system inherent in a functional scan. Instead they settle for the non-functional scan from the expensive subscription tools such as Qualys which they use.
The majority of time these reports are being produced to fulfill an auditing or FISMA requirement and not considered worth the risk to systems a functional scan could present.
But what else would you propose? Only thing you could do which would be more accurate is to try to exploit all the issues it found, which isn't really practical, because POCs are not always available and if they are, you would have to build some kind of database with all of them. Maybe it could be combined with Metasploit...
Tools like Metasploit can often be destructive, not sure I would want to run them against production by default.. A more reasonable approach would be to set up a lab when needed when one of the less invasive scanners flags a potential issue.
I think I'd prefer a tool that identifies points of interest for my team to investigate rather than one that actually tries to execute a hack against my servers, introducing possible instability.
Similar experience. Many scanners seem to be designed around showing 'value' to management vs real value for engineering and security teams. Another example is Blackduck. Its slow, noisy, has tons of FPs (which you cannot globally ignore), etc. It ends up being a tool users fight against instead of work with.
Another caveat -- scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability. Its a mixed bag -- at scan time the system may be secure, but a future change may result in a vulnerable system. Unless you're religious about scanning after each build/config change; this could bite you as well. It would be nice to know these minutiae without digging into each CVE report.
> scanners usually do not account for the current v vulnerable configuration. So while they may flag a vulnerability, its often a potential vulnerability.
This is a straight-up artifact from the fact that the scanners (which essentially have to operate in non-destructive mode) rely on banner grabbing and very basic service behaviour pattern matching. Hence for every service they find, they report - quoting your words here - the potential vulnerability.
Which basically is what you want. Trying to intelligently reduce noise would be really cool, but the moment the intelligence suppresses a finding that you WOULD have wanted to know about, you're screwed. It's bad either way, but the worst-case scenarios of the two are asymmetrical.
I personally find point-in-time scans a useless exercise. Findings from individual cans are one thing, the difference over time is what really matters.
Remember: This is a 50 line Python script that is a wrapper for nmap + vulners NSE script.
It's not a product, but some simple automation around existing tooling. Better than paying $$$ for a full fledged scanner? Maybe, but depends on your use case.
Flan Scan is a thin wrapper around Nmap that converts this popular open source tool into a vulnerability scanner with the added benefit of easy deployment.
Feel free to ask any questions you have. We have quite a few folks from Cloudflare's security team here.
Vulners.com isn't a free service, are there other cheaper alternatives (perhaps less comprehensive) that maintain database of vulnerabilities?
Interestingly, the very impressive infosec people behind Vulners are employed (?) by QIWI, a Russian payments company. That isn't an issue now that you publicly claim to use their service, but were there any reservations raised by legal or otherwise?
How much in relative terms were the cost savings when Cloudflare did switch to in-house audits via Flan Scan given the requirements / development / operational / maintenance effort expended + licensing service from Vulners?
Does Flan Scan also scan network equipment (like switches, routers etc)?
Given the complex heterogeneous nature of the global Cloudflare network, what did the deployment process look like? Will there a follow-up blog post on how that was automated/accomplished?
What are the other big cloud / CDN providers doing to scan for vulns / compliance at scale if you're privy to it? Have any of them shown interest in contributing to and/or using Flan Scan?
What does the short-term and long-term roadmap for Flan Scan look like?
Why "Flan Scan"? :)
Thanks a lot.
Having had to deal with heavy duty, expensive, vendor-supplied scans in a previous life (Qualys, yuck) this seems a very nice breath of fresh air.
This is really our use case. A little bit of packaging around a pretty good vuln scanner you can set up in 10 minutes
I’ve managed Nessus in a past life and it was a nightmare.
That LaTeX report output is hot. It'd be cool if the bar could be raised for scanners like this, which mostly just try to get services to respond with a version number.
It is always funny seeing how less than a hundred lines of code can replace six or seven digits in licensing, negotiation, lawyering, and time.
Cool project. I'd be interested in using the scan data to automate workflows. Is Cloudflare hooking this into some security automation engine to make the data actionable?
TLDR; they use nmap + vulners (nmap script). Their wrapper outputs those low/medium/high-risk cve reports that no one likes to look at
Every good vuln scanner needs a custom typeface — excited to hear about Flan Scan Sans!
The article essentially reads as "we payed lots of monies for a mediocre scanner, and then we discovered that the FOSS nmap did everything we needed. So we took nmap and added a little bit of extra, a web interface, and gave it the name Flan Scanner."
. . .
The corporate world is so facepalm sometimes
> added a little bit of extra, a web interface
Sometimes that's the difference between people using it not using it. The classic rsync and a bunch of scripts vs. Dropbox HN comment.
https://news.ycombinator.com/item?id=9224
For people new to HN, this is the comment dewey is referring to, I think. 'dhouston' is Drew Houston, the founder of Dropbox.
And that's how people miss the opportunity to make a lot of money because they don't understand the needs of corporations.