In some ways, the bigger the computer, the more vulnerable it is to cryptomining as Clemson University discovered after cryptominers dug into its Palmetto supercomputer. When a number of nodes on Clemson University’s Palmetto supercomputer were at peak processing capacity even though no students, faculty or researchers were using them, IT systems architect Nitin Madhok knew something was amiss.

“They weren’t doing anything, and yet its CPU and memory usage was always at 100 percent,” said Madhok, who’s been with Clemson since graduating from the university in 2013. “It wasn’t going down even when they weren’t doing any work. That was kind of unusual for us.” What followed was an investigation and discovery that bad actors were tapping into Palmetto stealing compute cycles and driving up operation costs.

Clemson’s cautionary tale is the subject of an article posted on Edscoop last week detailing efforts required to identify the breach and root out cryptominers. As it turns out computers in academia are among the most often high-jacked for cryptomining as noted in the excerpt from the article:

“Cryptomining efforts are popular across higher education, according to a study published in March from Vectra. The California-based cybersecurity company revealed that 85 percent of cryptocurrency mining instances happened in higher education between August 2017 and January 2018, compared to just three percent in the technology sector. The most recent high-profile scheme hit universities worldwide in February through a vulnerability in a browser widget to mine the cryptocurrency Monero. It’s hard enough for universities to monitor cryptomining on student-owned laptops, but in Clemson’s case, a supercomputer presented unique challenges in network defense.”

Cryptomining isn’t generally illegal but it is usually prohibited by institutional policies and in Clemsen’s case it’s both “a violation of campus policy and the South Carolina State Ethics Commission — neither students or faculty are allowed to use a state resource for personal financial gain.”

As one of the five most powerful supercomputers at public universities nationwide, according to the article, Clemson’s Palmetto is an enticing target for people who mine for cryptocurrency. Currently, the Palmetto cluster has:

• 2021 compute nodes, totaling 23072 cores
• 386 nodes equipped with NVIDIA Tesla GPUs: 280 nodes with NVIDIA K20 GPUs (2 per node), 106 nodes with NVIDIA K40 GPUs (2 per node)
• 4 nodes with Intel Phi co-processors (2 per node)
• 6 large memory nodes (5 with 505GB, 1 with 2TB), 262 nodes with 128GB of memory
• 100GB of personal space (backed up daily for 42 days) for each user
• “unlimited” scratch storage for temporary files
• 10 Gbps Ethernet, 10 Gbps Myrinet and 56Gbps Infiniband networks
• maximum run time for a single task limited to 72 hours (Infiniband nodes) or 168 hours (Myrinet nodes)

Manually monitoring Palmetto would have been virtually impossible, Madhok said in the article. He and his team could have collected the CPU and memory data on what processes are running, and the data from network packet aggregators showing the increased traffic, but that only told part of the story, and they needed answers — why was it running so hard?

In 2017, Madhok and his team licensed Splunk for enterprise security across Palmetto and Clemson’s general network. The university’s other security services —Bro, an intrusion detection system, and Gigamon, a network monitoring software, fed data like network packets, the logs from the servers, metrics like CPU and memory storage performance into Splunk. That raw data held the answers Madhok was looking for — without Splunk, though, it was virtually impossible to understand.

According to the article, Madhok said, schools can’t absolutely prevent users from mining cryptocurrency — the malware will always be available to download and there will always be a workaround bad actors can use. Because of that the solution is to be proactive by establishing a security environment, like Clemson’s, that can give administrators visibility of their networks. “It’s not necessarily a question of whether you can prevent it,” he said. “It’s a question of how fast you can respond to it.”

Link to full article: https://edscoop.com/how-clemson-keeps-cryptominers-off-its-supercomputer/