Pentagon hackers-for-hire take just 4 hours to find critical vulnerability in sensitive system

This article first appeared on FedScoop.

The Pentagon’s cybersecurity swat team has hosted highly-publicized challenges to find flaws in department and military branch websites, but it also recently orchestrated a more secret, complex project.

The Defense Digital Service wanted to hire outside researchers to root out vulnerabilities in systems carrying sensitive department data — but without worrying about compromising the data, or getting the researchers in trouble. And in January, a group vetted by the company Synack descended on copies of two sensitive department systems to look for ways to break them.

The researchers found their first critical vulnerability in four hours, said Synack co-founder and Chief Technology Officer Mark Kuhr in an interview with FedScoop. Verifying it was accurate and valid, they turned it over to the government within the first 24 hours. 

During the Defense Department’s first bug bounty, Hack the Pentagon, white-hat hackers found 138 vulnerabilities that the department has since remediated. But in this new program, Synack and its researchers have to keep mum about specific vulnerabilities found, and even the total number of vulnerabilities discovered. Kuhr did say in a followup email, “we found more than we expected.”

“The Hack the Pentagon program, you know that was on some recruiting sites, or some public-facing sites,” Kuhr said. “So we were a little surprised because … they designed this system to prevent certain actions and then they tasked us with objectives to make sure that those actions could not be done. And then we were able to prove that those actions could be done in a very short period of time.”

Nobody spoke to reporters until after the private bug bounty was conducted, which began early January and finished on Feb. 7. Bloomberg first reported the story on Monday.

“This is the first time that the Pentagon actually leveraged a crowdsourced solution on such a sensitive a system, that they were really concerned about the discoveries being disclosed in the public and really, the discoveries falling into the hands of the adversary,” Kuhr said. “So there were a lot of skeptics when we started this, saying ‘hey, that’s not a good choice for this system to use a crowdsourced approach. We should really use a traditional model with the defense contractor base.”

During the first day of Defense Secretary James Mattis’ standup, Defense Digital Service Director Chris Lynch pulled Mattis’ deputy chief of staff and a senior adviser aside and told them he needed to give them a heads up on a project DDS was working on: the bug bounty.

“These are sometimes very scary things so I just wanted to make sure that they knew so that they weren’t surprised and they could help support as needed,” Lynch said in an interview with FedScoop.

Home on the range, where the researchers roam free

It wasn’t easy for the Defense Digital Service to figure out a way to hire researchers without security clearances for the job, Lynch said.

But by creating a “range” — what is basically a copy of the environment the group wanted tested without classified or sensitive data, the researchers were able to work on something that “looked and felt and acted like the real thing,” Lynch said. The team picked two systems to be tested, so two “completely different environments had to be set up that looked just like a real production environment.”

“There wouldn’t be any risk of exfiltration or there wouldn’t be any risk of people being able to get into areas that we really didn’t want them in,” he said. “But we still would get the value of sending researchers into a place that would help us identify potential vulnerabilities that we would care deeply about if they were discovered by our adversaries.”

Lynch said the Digital Service team knew it wanted to run a bug bounty on more sensitive systems “but we also knew that we didn’t have the mechanism or the means to pull it off, especially when no bug bounty had been done before.”

“So we sort of laid it out as an overall strategy that would start with: let’s go to something that was seemingly innocuous, like a public website, so Defense.gov was, of course, one of the initial targets,” he said. “We knew if we could prove it out there, and prove the building did not, in fact, melt down that we could go to a slightly deeper target after that.”

Coming out of Hack the Pentagon, the Defense Digital Service awarded two contracts: one to HackerOne for the public crowdsourced bug bounties, and then one to Synack to run more sensitive bounties, “but we hadn’t picked any targets,” Lynch said.

Getting the emulated environments in place took a lot of planning and hard work, Lynch said, joking that there were a couple of weeks where he thought it wouldn’t come together.

“People worked extremely hard to come together around this once it was clear that the train had left the station,” he said. 

Surprising finds? Or a lack thereof?

“I mean of course I knew that they were going to find things,” Lynch said. “The question is how severe it would be.”

He continued, “I think that we need to get far better at coming up with new ways to allow hackers and researchers to help us vet our assumptions. Because the current systems that we have in place have not worked. So we need to come up with new ways to think through how we test our assumptions about things.”

Lynch said the Pentagon must look at securing its systems the way it does with security clearances for people, taking a periodic look back to make sure they are still where they need to be. The big difference: Those checks should happen more often — and faster — when it comes to software, he said.

“There’s doubt that programs like this running bug bounties and running programs like Hack the Pentagon positively benefits us and helps us secure the systems and the mission of the Department of Defense, every day,” he said. “There is no doubt that that value is there.”

Protected researchers and a new incentive model

Part of Synack’s contract with the Pentagon was to handle the vetting process for the researchers, around which the Defense Department put “loose constraints,” as Kuhr described them, in terms of countries of origin. Synack put about 80 researchers on the challenge from across U.S. and Five Eye countries.

Synack doesn’t even disclose the identities of the researchers to the government to protect their privacy, Kuhr said. And researchers actually VPN into Synack’s platform to access the range, Lynch said.

“We run a very closed operation, and it’s honestly because of our NSA experience, we just don’t trust random people on the internet. And once you release this type of identity information you don’t know who’s going to get their hands on it. So it’s really important that we protect our researchers’ privacy, and get to know them really well through our vetting process that includes a very strong relationship-building phase,” he said. 

Lynch said the researchers were “extremely well-vetted.”

“They’re here to help us,” he said. “We should be far more concerned with the people who are our adversaries who want to be able to take advantage of these systems.”

Kuhr said Synack’s model fixes a broken government contracting process where people are paid by the hour for work, regardless of how many system flaws they find.

Synack researchers, according to Kuhr, are “incentivized for success, versus just time participation.”

The company also works with the Internal Revenue Service, as FedScoop has reported. 

Working with the Defense Digital Service on this project was “crazy fast,” Kuhr said.

“It was probably the smoothest contract vehicle I’ve ever seen from [Request for Proposals] to award,” he said. “And I think that’s sort of a testament to Defense Digital Service.”