Description
A Lightning Hunt has been conducted by our Threat Hunters and tickets have been opened as required with Citrix NetScaler
| Threat Info | |
| Name of threat | Vulnerability: CVE-2023-3519 |
| Severity | 9.8 CRITICAL |
| Type | Unauthenticated remote code execution |
| Discovered | mid-July 2023 |
| Target Industries | Critical infrastructure organizations are a main target |
Overview
On July 18, Citrix released security bulletin CTX561482, which described vulnerabilities in Citrix Netscaler Application Delivery Controller (ADC) and Citrix Netscaler Gateway. One of the vulnerabilities, CVE-2023-3519, could allow an unauthenticated remote attacker to perform arbitrary code execution. This vulnerability was assigned a CVSS of 9.8. Citrix has stated that they have observed exploitation of this vulnerability in the wild.
Affected versions of NetScaler ADC and NetScaler Gateway:
- NetScaler ADC and NetScaler Gateway 13.1 before 13.1-49.13
- NetScaler ADC and NetScaler Gateway 13.0 before 13.0-91.13
- NetScaler ADC and NetScaler Gateway version 12.1, now end of life
- NetScaler ADC 13.1-FIPS before 13.1-37.159
- NetScaler ADC 12.1-FIPS before 12.1-55.297
- NetScaler ADC 12.1-NDcPP before 12.1-55.297
Tiance must be configured as a Gateway (VPN virtual server, ICA Proxy, CVPN, RDP Proxy) or authentication, authorization, and auditing (AAA) virtual server for exploitation.
| Tactic(s) | Technique | Sub-technique | ID | Description |
| Initial Access | Exploit Public – Facing Application | T1190 | Adversaries may attempt to exploit a weakness in an Internet – facing host or system to initially access a network. | |
| Exfiltration | Archive Collected Data | T1560 | An adversary may compress and / or encrypt data that is collected prior to exfiltration |
Technical Details
MITRE Framework
| Tactic(s) | Technique | Sub-technique | ID | Description |
| Execution | Server SoftwareComponent | T1505 | Adversaries may abuse legitimate extensible development features of servers to establish persistent access to systems. | |
| Privilege Escalation | Hijack Execution Flow | T1574 | Adversaries may execute their own malicious payloads by hijacking the way operating systems run programs. | |
| Discovery | Permission Groups Discovery Domain Groups | T1069.002 | Adversaries may attempt to find domain-level groups and permission settings | |
| Discovery | Remote System Discovery | T1018 | Adversaries may attempt to get a listing of other systems by IP address, hostname, or other logical identifier on a network that may be used for Lateral Movement from the current system. | |
| Discovery | Domain Trust Discovery | T1482 | Adversaries may attempt to gather information on domain trust relationships that may be used to identify lateral movement opportunities in Windows multi-domain/forest environments. | |
| Discovery | System Network Configuration | T1016 | Adversaries may look for details about the network configuration and settings, such as IP and/or MAC addresses, of systems they access or through information discovery of remote systems. | |
| Discovery | System Network Configuration Discovery Internet Connection Discovery | T1016.001 | Adversaries may check for Internet connectivity on compromised systems. | |
| Discovery | Network Service Discovery | T1046 | Adversaries may attempt to get a listing of services running on remote hosts and local network infrastructure devices, including those that may be vulnerable to remote software exploitation. | |
| Discovery | Account Discovery Domain Account | T1087.002 | Adversaries may attempt to get a listing of domain accounts. | |
| Defense Evasion Masquerade | Masquerade File Type | T1036.008 | Adversaries may masquerade malicious payloads as legitimate files through changes to the payload’s formatting, including the file’s signature, extension, and contents. | |
| Privilege Escalation | Abuse Elevation Control Mechanism Setuid and Setgid | T1548.001 | Adversaries may circumvent mechanisms designed to control elevate privileges to gain higherlevel permissions. | |
| Credential Access | Unsecured Credentials Credentials In Files | T1552.001 | Adversaries may search local file systems and remote file shares for files containing insecurely stored credentials. | |
| Credential Access | Unsecured Credentials Private Keys | T1552.004 | Adversaries may search for private key certificate files on compromised systems for insecurely stored credentials. | |
| Persistence Server Software | Component Web Shell | T1505.003 | Adversaries may backdoor web servers with web shells to establish persistent access to systems. | |
| Command and Control | Encrypted Channel | T1573 | Adversaries may employ a known encryption algorithm to conceal command and control traffic rather than relying on any inherent protections provided by a communication protocol. | |
| Command and Control | Proxy Internal Proxy | T1090.001 | Adversaries may use an internal proxy to direct command and control traffic between two or more systems in a compromised environment. | |
| Command and Control | Ingress Tool Transfer | T1105 | Adversaries may transfer tools or other files from an external system into a compromised environment. | |
| Persistence | Scheduled Task/Job | T1053 | Adversaries may abuse task scheduling functionality to facilitate initial or recurring execution of malicious code | |
| Collection | Archive Collected Data Archive via Utility | T1560.001 | Adversaries may use utilities to compress and/or encrypt collected data prior to exfiltration. | |
| Collection | Data from Local System | T1005 | Adversaries may search local system sources, such as file systems and configuration files or local databases, to find files of interest and sensitive data prior to Exfiltration. | |
| Collection | Data Staged | T1074 | Adversaries may stage collected data in a central location or directory prior to Exfiltration. |
Initial Access to NetScaler
The threat actors exploit CVE-2023-3519 to implant a webshell on the organization’s NetScaler ADC appliance.
Mitre: T1190 – Exploit Public Facing Application
Persistence
The threat actors implant a generic webshell on the organization’s NetScaler ADC appliance.
Mitre: T1505.003 – Server Software Component: Web Shell
Privilege Escalation
As part of their initial exploit chain upload a TGZ file containing a setuid binary on the ADC appliance.
Mitre: T1548.001 – Abuse Elevation Control Mechanism: Setuid and Setgid
Defense Evasion
The threat actors exfiltrate data by uploading it as an image file to a web-accessible path.
Mitre: T1036.008 – Masquerading: Masquerade File Type
Credential Access
The threat actors obtain encrypted passwords from NetScaler ADC configuration files, and the decryption key is stored on the ADC appliance.
Mitre: T1552.001 – Unsecured Credentials: Credentials In Files
The threat actors obtain decryption keys to decrypt the AD credential obtained from the NetScaler ADC configuration files.
Mitre: T1552.004 – Unsecured Credentials: Private Keys
Discovery
The threat actors query the AD for trusts.
Mitre: T1482 – Domain Trust Discovery
The threat actors query the AD for groups
Mitre: T1069.002 – Permission Groups Discovery: Domain Groups
The threat actors query the AD for computers. The threat actors attempted to execute a subnet-wide curl command, in order to identify what was accessible from within the network as well as potential lateral movement targets. Network-segmentation controls prevent this activity. Mitre: T1018 – Remote System Discovery
The actors use a webshell for AD enumeration.
Mitre: T1016 – System Network Configuration Discovery
The threat actors attempt to verify outbound network connectivity with a ping command and execute host commands for a subnet-wide DNS lookup. Network segmentation controls prevent this activity.
Example: ping -c 1 google[.]com
Mitre: T1016.001 – System Network Configuration Discovery: Internet Connection Discovery
The threat actors conduct SMB scanning on the organization’s subnet.
Mitre: T1046 – Network Service Discovery
The threat actors query the AD for users.
Mitre: T1087.002 – Account Discovery: Domain Account
Collection
The threat actors encrypt discovery data collected via openssl in “tar ball.”
Example: tar -czvf – /var/tmp/all.txt | openssl des3 -salt -k <> -out /var/tmp/test.tar.gz.
Mitre: T1560.001 – Archive Collected Data: Archive via Utility
The threat actors view NetScaler ADC configuration files flash/nsconfig/keys/updated/* and /nsconfig/ns.conf.
Mitre: T1005 – Data from Local System
The threat actors upload data as an image file to a web accessible path: cp /var/tmp/test.tar.gz /netscaler/ns_gui/vpn/medialogininit.png. Mitre: T1074 – Data Staged
Command & Control
The threat actors exploit CVE-2023-3519 to upload a TGZ file containing a generic webshell, discovery script, and setuid binary on the ADC appliance
Mitre: T1105 – Ingress Tool Transfer
The actors likely use a PHP shell with proxying capability to attempt proxying SMB traffic to the DC (the traffic was blocked by a firewall and account restrictions).
Mitre: T1090.001 – Proxy: internal proxy
Detection techniques with NetScaler
Run the following victim-created checks on the ADC shell interface to check for signs of compromise:
1.Check for files newer than the last installation.
2.Modify the -newermt parameter with the date that corresponds to your last installation:
- find /netscaler/ns_gui/ -type f -name *.php -newermt [YYYYMMDD] -exec ls -l {} \;
- find /var/vpn/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;
- find /var/netscaler/logon/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;- find /var/python/ -type f -newermt [YYYYMMDD] -exec ls -l {} \;
3.Check http error logs for abnormalities that may be from initial exploit:
- zgrep ‘.sh’ /var/log/httperror.log*
- zgrep ‘.php’ /var/log/httperror.log*
4.Check shell logs for unusual post-ex commands, for example: – grep ‘/flash/nsconfig/keys’ /var/log/sh.log*
5.Look for setuid binaries dropped:
- find /var -perm -4000 -user root -not -path “/var/nslog/*” -newermt [YYYYMMDD] -exec ls -l {} \;
6.Review network and firewall logs for subnet-wide scanning of HTTP/HTTPS/SMB (80/443/445) originating from the ADC.
7.Review DNS logs for unexpected spike in internal network computer name lookup originating from the ADC (this may indicate the threat actor resolving host post-AD enumeration of computer objects).
8.Review network/firewall logs for unexpected spikes in AD/LDAP/LDAPS traffic originating from the ADC (this may indicate AD/LDAP enumeration).
9.Review number of connections/sessions from NetScaler ADC per IP address for excessive connection attempts from a single IP (this may indicate the threat actor interacting with the webshell).
10.Pay attention to larger outbound transfers from the ADC over a short period of session time as it can be indicative of data exfiltration.
11.Review AD logs for logon activities originating from the ADC IP with the account configured for AD connection.
12.If logon restriction is configured for the AD account, check event 4625 where the failure reason is “User not allowed to logon at this computer.”
13.Review NetScaler ADC internal logs (sh.log, bash.log) for traces of potential malicious activity (some example keywords for grep are provided below):
- database.php
- nsgui/vpn – /flash/nsconfig/keys/updated – LDAPTLSREQCERT
- ldapsearch
- openssl + salt
14.Review NetScaler ADC internal access logs (httpaccess-vpn.log*) for 200 successful access of unknown web resources.
How to Hunt for Authentication Attempts on NetScaler
It recommends organizations use available logs and Endpoint Detection & Response (EDR) telemetry to hunt for authentication attempts sourced from Netscaler management addresses (NSIPs) to all endpoints in the environment. Mandiant observed authentication attempts by the threat actor sourced from NSIPs of impacted Netscalers both via Remote Desktop Protocol (RDP) logons and network logons to endpoints within the victim’s environment. Additional information recorded in these events may capture both hostnames and IP addresses belonging to attacker infrastructure to further pivot and hunt for in the environment. It is unexpected and suspicious to observe traffic to the internal network and miscellaneous (non-Citrix) Internet IP addresses from the NSIP of an appliance. Rotate credentials for any impacted/targeted accounts identified in these attempts.
Review relevant firewall logs for any network based indicators identified. Additionally, Mandiant observed the string pwd;pwd;pwd;pwd;pwd; used within the exploit POST requests which can aid hunting. Also, prior to upload of the initial web shell, Mandiant identified requests by a Headless Chrome User Agent (executed via CLI) included as follows:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/112.0.5615.121 Safari/537.36 Furthermore, Mandiant recommends review of HTTP error logs for potential crashes, which can be indicative of vulnerability exploitation.
Mandiant observed LDAP queries sourced from NSIPs of impacted Netscalers in an attempt to identify accounts vulnerable to Kerberoasting. A sample query can be seen as follows:
(&(servicePrincipalName=*)(UserAccountControl:1.2.840.113556.1.4.803:=512)(!(UserAccountControl
:1.2.840.113556.1.4.803
Mandiant recommends review of the following directories and subdirectories for the presence of web shells:
/var/vpn/
/var/netscaler/logon/
/var/python/
/netscaler/ns_gui/
In order to identify malicious ELF binaries, Mandiant recommends a review of the /tmp/ directory. Similarly, review of files with timestamps after the Netscaler was last patched is especially important.
In review of NSPPE core (Netscaler Packet Processing Engine) dumps, Mandiant identified commands executed by the threat actor to redirect the contents of ns.conf, F1.key, and F2.key to a renamed JavaScript file for exfiltration. Mandiant recommends reviewing relevant NSPPE core dumps in the /core/ directory in order to identify similar activity. Rotation of the keys is recommended if similar activity is observed in NSPPE core dumps.
Finally, Mandiant recommends a review of /var/crontabs/nobody for scheduled execution of suspicious binaries. Mandiant identified a crontab for the aforementioned ELF tunneler, the.
Mitigating the CVE-2023-3519 NetScaler ADC and Gateway Vulnerabilities
Install the relevant updated version of NetScaler ADC and NetScaler Gateway as soon as possible. See Citrix ADC and Citrix Gateway Security Bulletin for CVE-2023-3519, CVE 2023-3466, CVE-2023-3467 for patch information.
Follow best cybersecurity practices in your production and enterprise environments, including mandating phishing-resistant multifactor authentication (MFA) for all staff and for all services.
Best Practices for Securing NetScaler
Given the scope and sophistication of this threat actor, Mandiant recommends that organizations rebuild any appliances that have been exploited. The ADC upgrade process overwrites some, but not all, of the directories where threat actors may create web shells, potentially leaving the appliance in a compromised state.
Organizations should evaluate whether their ADC or Gateway appliance management ports require unrestricted Internet access. Limiting the Internet access to only necessary IP addresses (such as Citrix related addresses) would make post-exploitation activities of this and any future vulnerabilities more difficult.
Recommendations for Mitigating the Threat
Mandiant has observed the threat actor copying the ADC ns.conf file as well as keys stored on the file system that are used to encrypt secrets within the configuration file. Public tooling exists to decrypt the ns.conf secrets although Mandiant has not validated it works for the most recent appliance versions. Given these TTPs, Mandiant recommends that impacted organizations rotate all secrets stored in the configuration file as well as any private keys and certificates that may be used for TLS connections.
Mandiant recommends hardening susceptible accounts in the domain to reduce the likelihood of credential exposure via Kerberoasting and to limit a potential threat actor’s ability to obtain credentials for lateral movement throughout the environment.
If compromise is detected, organizations should:
1.Quarantine or take offline potentially affected hosts.
2.Reimage compromised hosts.
3.Provision new account credentials.
4.Collect and review artifacts such as running processes/services, unusual authentications, and recent network connections.
5.Report the compromise to CISA via CISA’s 24/7 Operations Center (report@cisa.gov or 888- 282-0870).
References
https://www.mandiant.com/resources/blog/citrix-zero-day-espionage https://www.cisa.gov/sites/default/files/2023-07/aa23-201acsathreatactorsexploitingcitrix-cve-2023-3519toimplantwebshells.pdf https://www.mandiant.com/resources/blog/citrix-adc-vulnerability-ioc-scanner https://github.com/telekom-security/cve-2023-3519-citrix-scanner https://mikecybersec.medium.com/hunting-for-potentially-vulnerable-citrix-servers-with-shodan-cve-2023-3519-977540cae5df
