CISA updates advisory on hackers exploiting RCE vulnerability in NetScaler ADC, Gateway devices

The Cybersecurity and Infrastructure Security Agency (CISA) published Thursday an update to its July cybersecurity advisory (CSA) warning critical infrastructure organizations about hackers exploiting an unauthenticated remote code execution (RCE) vulnerability affecting NetScaler (formerly Citrix) Application Delivery Controller (ADC) and NetScaler Gateway devices. The advisory, originally released to warn network defenders of critical infrastructure organizations about hackers exploiting the vulnerability, now contains victim information gathered in August. 

Since July 2023, the Joint Cyber Defense Collaborative (JCDC) has facilitated continuous, real-time threat information sharing with and between partners on the post-exploitation activity of CVE-2023-3519. JCDC consolidated and shared detection methods, cyber hacker tactics, techniques, and procedures (TTPs), and indicators of compromise (IOCs) received from industry and international partners. The updated CSA contains new TTPs as well as IOCs received from some of these partners and an additional victim.

The security agency ‘strongly urges’ all critical infrastructure organizations to review the updated advisory and follow the mitigation recommendations, including prioritizing patching known exploited vulnerabilities like Citrix CVE-2023-3519. “If potential compromise is detected, organizations should apply the incident response recommendations provided in this CSA. If no compromise is detected, organizations should immediately apply patches provided by Citrix.”

“In June 2023, threat actors exploited this vulnerability as a zero-day to drop a webshell on a critical infrastructure organization’s non-production environment NetScaler ADC appliance,” CISA said. “The webshell enabled the actors to perform discovery on the victim’s active directory (AD) and collect and exfiltrate AD data. The actors attempted to move laterally to a domain controller but network-segmentation controls for the appliance blocked movement. The victim organization identified the compromise and reported the activity to CISA and Citrix. Citrix released a patch for this vulnerability on July 18, 2023.”

CISA said that in August 2023, it received TTPs and IOCs from an additional victim and third parties. “The actors implanted a webshell, gained root-level access to the compromised system, and performed discovery against the Active Directory (AD).”

New details identified that hackers uploaded a PHP webshell *logouttm[dot]php*, likely as part of their initial exploit chain, to ‘*/netscaler/ns_gui/vpn/*.’ “Within an hour of installing the webshell, the actors implanted an Executable and Linkable Format (ELF) binary ‘pykeygen’ that set user unique identifier (UID) to root and executed /bin/sh via ‘setuid’ and ‘execve syscall.’” 

“With root-level access, the actors used hands-on-keyboard for discovery. They queried the AD via ‘ldapsearch’ for users, groups, and computers. They collected the data in gzipped text files renamed 1[dot]css and 2[dot]css and placed the files in /netscaler/ns_gui/vpn/ for exfiltration,” the advisory said. “After exfiltrating the files, the actors deleted them from the system as well as some access logs, error logs, and authentication logs. The victim organization detected the intrusion and mitigated the activity but did not identify signs of additional malicious activity.” 

CISA identified that for command and control (C2), “the actors appeared to use compromised pfSense devices; the victim observed communications with two pfSense IP addresses indicating the actor was using them for multi-hop proxying C2 traffic.”

The agency disclosed that according to trusted third-party reporting, hackers leveraged open source webshells and other publicly available tools. They also loaded loading[dot]phpto/vpn/vpn/. Loading[dot]php, a modified version of an open-source PHP webshell. The modified webshell had a custom protocol handler, update://, created via a stream_wrapper_register. The custom protocol enabled the webshell to send encoded commands to the AD via ‘ldapsearch.’

The hackers also loaded a modified version of open-source SSH inject tool named libnsd[dot]so to the /tmp directory. The file hooks SSH and captures credentials. The actors’ modifications to the tool wrote encoded captured credentials (20 random characters + reversed/base64) to a file created in a hard-coded path (/var/nslog/counters/prometheus/metrc_codes_client.log). The actors then exfiltrated the collected credentials and deleted /var/nslog/counters/prometheus/metrc_codes_client.log 3 minutes after exfiltration). 

They were also identified to have deployed webshells to victim networks at \vpn \themes. One third-party observed the hackers deploy the SECRETSAUCE webshell. SECRETSAUCE is a webshell written in PHP. It can receive PHP code from an HTTP POST request, decrypt it using an internally embedded RSA public key, and execute the code in memory; each request to the webshell returns an HTTP 500 response code. The actor has used the following file names for SECRETSAUCE: vpn[dot]php, logout[dot]php, log[dot]php, prod[dot]php. Additionally, a second third party observed the actors deploy an open-source webshell with filename ‘defaults[dot]php. 

The agency also identified that the hackers deployed a LIGOLO-NG tunneller with the filename ‘the’ in the ‘/tmp’ directory. LIGOLO-NG is a tunneller written in Go that provides encrypted reverse TCP/TLS connections to a remote host. 

The advisory said that the hackers deployed the NPS tunneller to victim networks to the ‘/tmp’ directory. “NPS is an open-source tunneller written in Go. It must be configured from the command line or via a configuration file. Features including support for most common protocols and extensions, including compression, encryption, and port reuse,” it added. 

Some of the detection methods prescribed in the advisory for organizations to run victim-created checks on the ADC shell interface to check for signs of compromise. They include checking for files newer than the last installation; modifying the ‘-newermt’ parameter with the date that corresponds to the last installation; checking ‘http’ error logs for abnormalities that may be from the initial exploit; and checking shell logs for unusual ‘post-ex’ commands. 

It also included reviewing network and firewall logs for subnet-wide scanning of HTTP/HTTPS/SMB (80/443/445) originating from the ADC.7; and reviewing DNS logs for unexpected spikes in internal network computer name lookup originating from the ADC. It also suggests reviewing network/firewall logs for unexpected spikes in AD/LDAP/LDAPS traffic originating from the ADC; evaluating the number of connections/sessions from NetScaler ADC per IP address for excessive connection attempts from a single IP and paying attention to larger outbound transfers from the ADC over a short period of session time as it can be indicative of data exfiltration.

The advisory also called upon critical infrastructure organizations to review AD logs for logon activities originating from the ADC IP with the account configured for AD connection. It added that if logon restriction is configured for the AD account, check event 4625 where the failure reason is ‘User not allowed to logon at this computer;’ review NetScaler ADC internal logs for traces of potential malicious activity; and assess NetScaler ADC internal access logs for 200 successful access of unknown web resources. 

CISA recommends that organizations install the relevant updated version of NetScaler ADC and NetScaler Gateway as soon as possible. The agency also suggests adopting best cybersecurity practices across production and enterprise environments, including mandating phishing-resistant multi-factor authentication (MFA) for all staff and for all services. 

It also pointed organizations to the agency’s cross-sector Cybersecurity Performance Goals (CPGs), developed by CISA and the National Institute of Standards and Technology (NIST), which cover a prioritized subset of IT and operational technology (OT) security practices that can meaningfully reduce the likelihood and impact of known cyber risks and common TTPs. 

As a longer-term effort, CISA recommends applying robust network-segmentation controls on NetScaler appliances, and other internet-facing devices.

On Thursday, U.S. agencies released a joint advisory to highlight the presence of IOCs at an aeronautical sector organization as early as January 2023. The document confirms nation-state advanced persistent threat (APT) actors exploited CVE-2022-47966 to gain unauthorized access to a public-facing application (Zoho ManageEngine ServiceDesk Plus), establish persistence, and move laterally through the network. The vulnerability allows for RCE on the ManageEngine application.

Anna Ribeiro

Industrial Cyber News Editor. Anna Ribeiro is a freelance journalist with over 14 years of experience in the areas of security, data storage, virtualization and IoT.