Why bug bounty firms want to be penetration testing companies

A popular form of crowdsourcing might have a problem with the size of its crowd.

Most of the high-value digital security vulnerabilities reported to bug-bounty programs are found by just a fraction of the freelance researchers who participate in those contests, recent reports show, suggesting that there are not enough skilled bounty hunters to handle the available work.

The trend has big implications for an industry that has come to expect regular growth over the past half-decade. For the companies, it means their customers — corporations such as Fiat Chrysler, LinkedIn, Starbucks and others — are paying to hear about lots of low-severity bugs while more critical problems potentially remain undiscovered.

The latest numbers come from the 2019 Hacker Report by HackerOne, one of the leading bug bounty platforms along with Bugcrowd and Synack. Seventy-two percent of the hackers polled by HackerOne said they preferred to probe for vulnerabilities in websites. Compare that to the 3.5 percent who said they look for bugs in operating systems, which require more expertise to hack. Network hardware and memory safety, two other areas where qualified security pros are hard to find, are not explicitly named in the HackerOne report at all. (The company says that’s because researchers aren’t typically invited to hack those systems.)

But when the crowd goes to work, only a fraction of what it catches is really important. HackerOne data published in 2018 also showed that “25 percent of vulnerabilities reported and assigned severity are critical to high in severity,” spokesperson Lauren Koszarek said in an email. More than 300,000 hackers have signed up to HackerOne, roughly 10 percent have found something to report and just more than a quarter of those have received a bounty, the company told TechRepublic in January. By that math, some 97 percent of the hackers have never sold a bug, according to Katie Moussouris, founder of Luta Security and a former HackerOne employee.

And as the industry has matured, its limitations have become more clear, corporate security officers said.

“There are places where having more eyes is helpful, and places where [having] the right expertise matters more than the number of people looking at it,” says Chris Betz, chief security officer at telecommunications company CenturyLink. “The more specialized, nuanced or complex an area [is], the more important it becomes to have an expert there … I don’t trust crowdsourced areas as much as specialized.”

As the economics and perceptions shift, the bug bounty business model will need to change in order to survive, say industry practitioners and market research analysts.

These companies built their buzz around platforms that offered efficient and organized access to white-hat hackers, but classic bug bounty programs don’t appear to be a viable option for researchers hacking to earn their rent. Now, bounty brokers are figuring out how to work in closer tandem with clients to resolve deeper, higher-value issues. In short, they’re replicating another side of the hacker-for-hire industry: the penetration-testing strategies that rely on closer relationships with clients, rather than only the power of the crowd.

After the bug hits the windshield

The bug bounty companies certainly seem to realize a shift is underway. HackerOne announced last month its developing a crowdsourced penetration-testing model on top of its traditional bug bounty platform, in part because the “pen test” market now stands at roughly $1 billion compared to the bug bounty market’s $150 million, CEO Mårten Mickos told CyberScoop. Pen-testing can be defined as anything from a quick automated scan to a weeks-long process that involves computer hacking, phone calls and attempts to physically infiltrate a workplace.

HackerOne’s first order of business, Mickos said, will be to recruit experienced pen-testers. That could be a challenge, as the best pen-testers work in roles where they’re paid a steady salary with benefits. HackerOne hires bounty hunters as contractors, Mickos said.

The work is different, too. While bounty hunters seek out vulnerabilities and report them in exchange for a reward, pen-testers go through multi-step processes to uncover more complicated weaknesses, and often help clients understand how to fix them.

“I view pen testing as humans testing weaknesses in an organization across a variety of assets,” said Toby Bussa, a Gartner analyst specializing in security and privacy, adding that HackerOne’s shift to that business is “not a surprise.”

Various security companies have offered penetration tests for years. California-based Synack, founded in 2013, says its augments a roster of freelance hackers with software that automates vulnerability reports. “We’ve always sold tests, not bugs, and we’ve always stood between our customers and our hackers for their respective protection, taking on the full liability of the testing,” the company said in a statement.

Bugcrowd started offering its “Next Gen Pen Test” in November 2018 to address the “multibillion-dollar market opportunity,” said founder Casey Ellis.

A January blog post published by security research company Trail of Bits demonstrates how just a select few researchers report finding high quality bugs. Researchers Ryan Ellis, Keman Huang, Michael Siegel, James Houghton and Moussouris, of Luta Security, examined 61 HackerOne bounty programs including Twitter, Square, Slack from 2013 to 2015, and one Facebook program over 45 months.

They found over the “entirety of the HackerOne and Facebook data sets, the 7% of participants with 10 or more bugs were paid for 1,622 bounties, while the other 93% of the population earned 2,523.” The complete research is available in “New Solutions for Cybersecurity,” an MIT publication.

The top one percent of bounty hunters submitted bugs to nearly five different programs, researchers found. That’s an indication to Moussouris that companies are in danger of losing security support if those researchers shift their attention elsewhere.

“The actual number of human beings hasn’t changed much over the past five or six years,” said Moussouris, who also led the bug bounty program at Microsoft. “We all had high hopes that leaders in that space would effectively recruit more people and build it up. But we see a user base of spammers, and a smaller number of people submitting good reports.”

Bounties don’t replace security

While external vulnerability researchers provide companies with an extra check on their software, they aren’t a replacement for traditional security teams. When a company awards millions of dollars to researchers who uncover thousands of vulnerabilities in a short amount of time, for example, it suggests that company is leaning too hard on researchers who may not have the incentive to lend help again in the future. It also means corporate networks, along with the user data and business secrets contained inside, could be protected by a false sense of security.

Penetration testing may refer to a more collaborative process which involves hiring experts earlier in the software development cycle and trying to uncover any flaws before a product goes live.

“The vulnerability assessment part comes later, because that’s when you’re finding bugs,” said Betz, of CenturyLink. “If you have a product where you haven’t [conducted penetration testing] yet, you can chase a large number of vulnerabilities, but end up missing some of the larger, more crucial architecture issues…[W]ithout doing pentesting, you’re in a situation where you’re focused on a bunch of small bugs, rather than the overall security of the product.”

The two processes are not interchangeable. That might be a reality check after a marketing blitz helped pass a law requiring the Department of Homeland Security to create a bug bounty program for the federal government, said Moussouris.

But 93 percent of the companies on the Forbes Global 2000 list do not have a “a policy to receive, respond, and resolve critical bug reports submitted by the outside world,” according to HackerOne. The company says that makes for a less safe society, but Moussouris suggested many of the Forbes Global 2000 already has security protocols in place, and is not seeking the spam that comes with unsolicited vulnerability disclosures.

“There’s just no way bug bounties can grow past a certain point of effectiveness because that freelance labor is not there,” she said. “There’s spammers, and there’s the super-elite. But we’ve lost the bug bounty middle class.”

Update 4/12/19 10:35am ET: This story has been updated to state that HackerOne is expanding to include penetration testing services. The company will continue to offer bug bounty services.