The relentless ingenuity that drives cyber hacking is a global engine that knows no rest. Anyone with a laptop and run-of-the-mill computer smarts can buy or rent a phishing kit and start attacking – or it can be done by professionals with increasingly devious techniques. For cyber-crooks, it’s a numbers game: the one in 10,000 user fooled into opening an email and updating a password on a “deep fake” e-commerce site – or, say, a fake IRS site – is a hacker win.

Webroot, Broomfield, Colo., combines machine learning with supercomputing to take on phishing, malware and other cyber frauds, while compiling – and growing – a threat intelligence database in machine readable format containing tens of millions of clues signaling illegitimate websites.

According to Webroot CTO Hal Lonas, phishing is the most common cybersecurity threat, and the phishing attack landscape is becoming increasingly complex.

“It used to be that attack campaigns would get launched and last for days or weeks,” Lonas said in an interview with us earlier this month. “Now the bad guys set up a phishing campaign and run it for literally minutes, then they get some people to click on (a fake website)… Then they take it down so they can’t be caught, so the security vendors and authorities don’t catch up with them…, the bad guys take them down before they can be discovered.”

Along with faster phishing timeframes, hackers also have “upped the fidelity,” the apparent authenticity of phishing sites. “You can’t tell anymore if you’re not going to a legit site like eBay or Microsoft or Google…, you can’t tell the difference anymore.”

Lonas said Webroot for years has used machine learning to classify the web and to classify files for threat detection, typically using AWS- and on prem-based compute capabilities. Data is collected from tens of millions of end users whose companies purchase Webroot security solutions through the company’s 90 OEM partners. As the volume of phishing data has risen along with the accelerated spinning up-tearing down of phishing campaigns, the company found itself unable to keep up with the rapid pace, despite dedicating more compute resources to the problem.

“We found it was taking us days to turn around a new machine learning model to catch phishing attacks, and it was slower than we wanted to go.”

The goal was to update Webroot’s phishing models several times per day. Enter Comet, a 2.76 (peak) petaflops system at the San Diego Supercomputer Center. Comprised of 1,944 Intel Haswell Xeon nodes along with 36 Nvidia K80 GPUs, 36 P100 Nvidia GPUs and 634 TB of flash memory.

Webroot has used Comet for about two years, and Lonas said training cycles have been cut from what had been three to five days using conventional computing to three to five hours using Comet. That means that instead of running their models once or twice a week, they can now do it as often as several times a day. “When we’re most active, we’ll do a morning, noon and night-time run at the San Diego Supercomputer Center; that’s how fast we can turn it around,” said Lonas.

The complexity of the workload stems from the tens of millions of features – indicators of potential threats – a number that continually expands.

“The way we organize our (database) is that every instance of our product is not only benefiting from the threat intelligence we can provide to protect customers, it’s also acting as a threat telemetry sensor,” Lonas said. “So if you’ve got an instance of the Webroot endpoint agent installed on your computer and you browse a website we’ve never seen before, or you get downloaded to your computer certain information you’ve never seen before, that telemetry goes to our cloud. We continually learn about what’s happening on the internet from user behavior. We’re very careful to protect our users from a privacy standpoint, but the threat telemetry information goes to our clouds so we can retrain models.”

Lonas said traits of suspicious websites that go into the Webroot model include how long the site has existed (the shorter the more suspicious), IP addresses that are known to be bad, the type of server the site is on and whether the site registrar is the same as known bad sites. Other indicators include graphics, images and logos that look real but the internal links and the IP addresses don’t line up with known information from the legitimate site.

On the daily inferencing side of the equation, Lonas said Webroot does “several hundred million checks” against its database in search of phishing activity based on the web activity of 50 to 60 million end users globally. He said Webroot identifies between 2,000 and 6,000 phishing sites every day – sites that are fed back into the machine learning model for updated training.