In a report published by ESET, titled “A wild Kobalos appears: Tricksy Linux malware goes after HPCs” details of a new malware strain which has been seen to target high-performance computing (HPC) clusters. Typically, HPC are collections of servers, referred to as nodes, connected to each other via fast interconnect. Each node has a specific task to handle logins, data transfer, or advanced computational processes and is geared towards ensuring the high performance of the system when in use. HPCs are sometimes referred to as a “super computer” as they perform tasks that regular desktop computers can’t do or would take too long in performing.
The malware, called Kobalos, is a surprisingly small but complex piece of malware. It is perhaps for this reason that the malware has been named after a sprite from Greek mythology known for causing mischief among mortals. Those who play Dungeons and Dragons will be familiar with the Germanic associations of the mythological creature, called Kobolds. The malware has already been seen in the wild infecting HPCs based in Europe and has been seen targeting other Linux based servers on a global scale.
In North America, the malware was discovered in an Endpoint Security Vendor, several personal servers, and Government networks. In Europe, other than HPC clusters, the malware was also discovered in university networks, website hosting servers, as well as a marketing agency. In Asia, a large internet service provider was impacted by the malware. ESET managed to discover which networks and servers were impacted by scanning for and connecting to the SSH server using a specific TCP source port.
Kobalos is described by researchers as a generic backdoor as it contains broad commands that don’t reveal the intent of the attackers. In short, Kobalos grants remote access to the file system, provides the ability to spawn terminal sessions and allows proxying connections to other Kobalos-infected servers. Researchers further discovered that the operators of the malware could reach infected machines in several ways. The most popular way involved when Kobalos is embedded in the OpenSSH server executable (SSHD) and will trigger the backdoor code if the connection is coming from a specific TCP source port.
Other variants of the malware discovered would either connect to the attacker’s command and control server and then will act as a middleman or wait for an inbound connection on a given TCP port. One of the things that make Kobalos unique is that the code for communicating with command and control (C2) servers is found in the malware itself, this allows for any machine infected with Kobalos to turn into a C2 server through the issuing of a single command. As the C&C server, IP addresses, and ports are hardcoded into the executable, the operators can then generate new Kobalos samples that use this new C&C server.
The question many will ask is what exactly does the malware do? As ESET states in a supplementary article,
“In most systems compromised by Kobalos, the SSH client is compromised to steal credentials. This credential stealer is unlike any of the malicious OpenSSH clients we’ve seen before, and we’ve looked at tens of them in the past eight years. The sophistication of this component is not the same as Kobalos itself: there was no effort to obfuscate early variants of the credential stealer. For example, strings were left unencrypted and stolen usernames and passwords are simply written to a file on disk. However, we found newer variants that contain some obfuscation and the ability to exfiltrate credentials over the network… The presence of this credential stealer may partially answer how Kobalos propagates. Anyone using the SSH client of a compromised machine will have their credentials captured. Those credentials can then be used by the attackers to install Kobalos on the newly discovered server later.”
Analysing Kobalos has proved difficult as the malware is encapsulated in one function. Other Linux malware discovered by the security firm often consists of several distinct modules, this makes analyzing the malware easier as it can be done on a module by module basis, effectively piecing together the malware’s functionality and features. Kobalos is a single function that recursively calls its tasks within the single function meaning that the whole function needs to be deciphered as a whole before a picture of the malware even begins to surface. To make matters harder the malware encrypts all its strings making analysis even more of an uphill task. Researchers concluded,
“We were unable to determine the intentions of the operators of Kobalos. No other malware, except for the SSH credential stealer, was found by the system administrators of the compromised machines. We also didn’t have access to network traffic captures of the operators in action…The way Kobalos is tightly contained in a single function and the usage of an existing open port to reach Kobalos makes this threat harder to find. Hopefully the details we reveal today in our new publication will help raise awareness around this threat and put its activity under the microscope. This level of sophistication is only rarely seen in Linux malware. Given that it’s more advanced than the average and that it compromised rather large organizations, Kobalos may be running around for a little while.”
HPC Clusters in Hacker’s Crosshairs
Kobalos is not the only instance of HPC clusters been targeted by attackers recently. Towards the middle of 2020 reports began emerging that ARCHER, the UK's national supercomputing service, was working on restoring services following a cyber incident. Four days after the attack, all services had not been fully restored. At the time US authorities warned that state-sponsored attackers working on behalf of the Chinese government were targeting networks belonging to academic, pharmaceutical, and healthcare organizations involved in COVID-19 research.
ARCHER may have been impacted by these operations as it provides supercomputing services to academic researchers and industrial users who need to run large calculations and simulations such as those involved in modeling the COVID-19 outbreak. ARCHER services are available to researchers in the UK and other countries. For those interested in the technical specs, its core hardware comprises a Cray XC30 massively parallel supercomputer with 111,080 Intel Ivy Bridge processing cores.
The original Archer service launched in November 2013 and is currently being transitioned over to a new 28 petaflops Cray supercomputer with 748,544 AMD cores. The incident was first disclosed to the public on May 11, with the incident being described as involving an exploit on ARCHER login nodes and pulled the service offline.
In a European Grid Infrastructure (EGI) CSIRT advisory, officials discovered two separate instances of HPCs being targeted by hackers. Servers in Poland, Canada, and China were seen to be impacted. One incident involved hackers targeting academic data centers with the attacker managing to hop from one victim to another via compromised SSH credentials. Depending on who the victim was determined what operation the hijacked machines were used for. Tasks involved:
- XMR mining hosts (running a hidden XMR binary)
- XMR-proxy hosts. The attacker uses these hosts from the XMR mining hosts, to connect to other XMR-proxy hosts and eventually to the actual mining server.
- SOCKS proxy hosts (running a microSOCKS instance on a high port). The attacker connects to these hosts via SSH, often from Tor. MicroSOCKS is used from Tor as well.
- Tunnel hosts (SSH tunneling). The attacker connects via SSH (compromised account) and configures NAT PREROUTING (typically to access private IP spaces).
Targeting HPC clusters for the purposes of cryptocurrency mining and deploying crypto mining malware would be considered a holy grail target for many crypto-mining hackers. The sheer amount of CPU and GPU resources on offer to be subverted to crypto mining would be the main target for hackers. This could result in faster more efficient mining, although illegal if done over long periods of time and the hacker’s malware would need to remain undetected for that period.
Given that HPC clusters will run simulations for world-changing technology it is little wonder state-sponsored groups would target these machines for the wealth of information they contain. It would not just be state-sponsored hackers but those employed for corporate espionage as well.