Category: Forensics

Linux Investigation (Part 4)

Hal Pomeranz1 Comment

In this installment I’m going to tell you the sequence of steps I took to create the scenario data. If you want to read about my investigation before you look at “the answers”, then please read Part 1, Part 2, and Part 3.

Our scenario begins with the “worker” login at 23:22:19. Any artifacts before this time are my earlier attempts to get the scenario running. For those of you contemplating making your own forensic scenarios in the future, I suggest you carefully script the scenario and practice several times on a test system so that you don’t make big mistakes and leave confusing artifacts like I did. Apologies again to those of you who thought the earlier artifacts needed to be explained as part of the scenario.

At 23:22:19 I logged into the system as user “worker” using the password for the account, which was “worker“. The idea I had was to mimic investigations I’ve been involved in recently where the attackers accessed the system via stolen credentials. The frustrating thing about these incidents is that there are generally no artifacts showing how the attackers obtained the credentials unless you can find signs of an earlier exploit. Welcome to my world.

The “worker” had unlimited Sudo access and even had the “NOPASSWD” option set. Privilege escalation was trivial. Again, this mimics my real world experience in some cases.

One of the open questions from the investigation is how the /dev/shm/kit directory was staged, though we suspected the SSH session that occurred at 23:23:48. This was in fact me using scp to copy a file called k.tgz into /dev/shm. I then unpacked this file, creating /dev/shm/kit, and then removed the tar file.

The /dev/shm/kit directory contained three files:

  • rk.so – the Father rootkit library
  • xmrig – the XMRig cryptocurrency miner
  • config – an SSH client config file, containing the parameters for the outbound connection

I regret that my earlier failed attempts at the scenario updated the atimes on the scp and tar binaries. Because of the “relative atime” updates in Linux, the fact that the atimes on these programs had been updated only a few hours earlier meant that they didn’t get an update when I ran them during the scenario. If they had been updated during the scenario window, there would have been a much clearer answer to how this information got staged.

I had also hoped that we would be able to acquire the files in the directory via Volatility–particularly the “config” file–even though they had been deleted. Alas, this didn’t work for me. I’ll need to research a way to get them to hang around in memory for future versions.

Once /dev/shm/kit had been created from the tar file, I copied rk.so to /usr/lib/x86_64-linux-gnu/libymv.so.3. I then used “echo /usr/lib/x86_64-linux-gnu/libymv.so.3 >/etc/ld.so.preload” to create the ld.so.preload file. But I needed the SSH daemon to pick up the rootkit library, so I did “systemctl restart sshd“.

With the Father rootkit active in the SSH daemon, I went old school and connected to port 22 with “nc -p 48411 192.168.4.22 22“. I’d hard-coded “ymv” as the rootkit password when I compiled the rootkit. I then used the usual Python pty.spawn() trick to get a nice interactive shell– and create a clear artifact for the SOC to key in on. Once my backdoor shell was fully operational, I closed my original SSH session.

We found the command history for the hidden bash shell. You can see me renaming xmrig to top and setting up my search path so that I could execute top from the /dev/shm/kit directory. I meant to include an “export HISTFILE=/dev/null” command to give a clue about the lack of shell history on the system, but failed to do so.

The next step in the backdoor session was to fire up the outbound SSH session. For those of you who were wondering, here are the contents of the “config” file I used:

host ymv
Hostname 192.168.5.95
User pocmp
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
LocalForward 3333 127.0.0.1:3333
SessionType none
StdinNull yes
RequestTTY no
ForwardX11 no

Maybe in the future I’ll add the evidence from the outbound host to the scenario. That would feel more real-world.

After backgrounding the outbound SSH process, it was time to run our renamed XMRig. I hadn’t considered the fact that the “-B” option would fully daemonize this process so that it inherited PPID 1. Frankly this was a happy accident from my perspective since it meant that our hidden top process was not immediately evident in the linux.pstree output.

With the cryptominer running, I popped up one level to /dev/shm (“cd ..“) and removed the “kit” directory (“rm -rf kit“). I debated using “shred” just to be evil, but decided it was too mean. Also I rarely see actual attackers using shred.

At this point I left the backdoor session running because I wanted the command history to be an artifact in the scenario. I logged in again as user “worker” and pretended to be the SOC running UAC to collect data from the system. Did anybody notice that the memory dump provided was not actually created by UAC? When I looked at the UAC dump, the memory file could not be parsed for even simple plugins. So I went back and ran AVML manually. Thankfully, I got a decent memory capture on the second try.

Personally I learned a lot creating this scenario and later analyzing it. I’ve got lots of ideas and notes for creating more such scenarios in the future. But wow is it a lot of work, so don’t get too antsy while waiting for the next installment.

Linux Investigation (Part 3)

Hal Pomeranz2 Comments

Please read Part 1 and Part 2 for additional background

We’ve actually done quite well reconstructing the major points of our intrusion scenario, but there are definitely some lingering questions. Let’s see if we can dive deeper into some of these artifacts.

LD_PRELOAD Rootkit Or Not?

It would be nice to positively identify libymv.so.3 as an LD_PRELOAD rootkit, rather than just referring to it as the “suspicious library”. Fortunately we have a memory image and can use Volatility to extract the library. We can use linux.elfs to dump all the objects associated with a PID. We believe that the SSH daemon was explicitly restarted to pick up libymv.so.3, so we’ll dump that process. As we saw in Part 2, that PID is 937.

(venv) $ mkdir dump-elfs-pid-937
(venv) $ vol -q -f memory_dump/avml.lime -o dump-elfs-pid-937 linux.elfs --pid 937 --dump
Volatility 3 Framework 2.27.1

PID	Process	Start	End	File Path	File Output

937	sshd	0x55e3f52a9000	0x55e3f52b2000	/usr/sbin/sshd	pid.937.sshd.0x55e3f52a9000.dmp
937	sshd	0x7f7da1093000	0x7f7da1098000	/usr/lib/x86_64-linux-gnu/libzstd.so.1.5.7	pid.937.sshd.0x7f7da1093000.dmp
937	sshd	0x7f7da115d000	0x7f7da1160000	/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.14.0	pid.937.sshd.0x7f7da115d000.dmp
937	sshd	0x7f7da120c000	0x7f7da1234000	/usr/lib/x86_64-linux-gnu/libc.so.6	pid.937.sshd.0x7f7da120c000.dmp
937	sshd	0x7f7da1400000	0x7f7da14f7000	/usr/lib/x86_64-linux-gnu/libcrypto.so.3	pid.937.sshd.0x7f7da1400000.dmp
937	sshd	0x7f7da1a3c000	0x7f7da1a3f000	/usr/lib/x86_64-linux-gnu/libz.so.1.3.1	pid.937.sshd.0x7f7da1a3c000.dmp
937	sshd	0x7f7da1a5e000	0x7f7da1a65000	/usr/lib/x86_64-linux-gnu/libselinux.so.1	pid.937.sshd.0x7f7da1a5e000.dmp
937	sshd	0x7f7da1aa5000	0x7f7da1aa7000	/usr/lib/x86_64-linux-gnu/libymv.so.3	pid.937.sshd.0x7f7da1aa5000.dmp
937	sshd	0x7f7da1ab3000	0x7f7da1ab5000	[vdso]	pid.937.sshd.0x7f7da1ab3000.dmp
937	sshd	0x7f7da1ab5000	0x7f7da1ab6000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2	pid.937.sshd.0x7f7da1ab5000.dmp
937	sshd	0x7f7da1ade000	0x7f7da1ae9000	/usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2	pid.937.sshd.0x7f7da1ade000.dmp

libymv.so.3 was extracted to dump-elfs-pid-937/pid.937.sshd.0x7f7da1aa5000.dmp. We could go full tilt at this and load it up for static analysis. But in the middle of an incident, I’m much more likely to do a first pass using “strings” and an internet search engine. Here are some of the more interesting strings I see in this sample

lpe_drop_shell
timebomb
backconnect
Enjoy the shell!
/tmp/silly.txt

Running those terms through my favorite search engine, my first hit was this useful list of rootkit IOCs. Apparently these strings are indicators for the Father rootkit.

If you look at the documentation for Father, it allows process, file, and directory hiding via a hard-coded GID. In our case this is apparently GID 7823 as we saw in Part 2. Father also has an accept() hook backdoor which is activated by to connecting to any network service from a hard-coded port. Based on the linux.sockstat output, this appears to be port 48411 in our case.

Father also has a PAM hook that steals user passwords and saves them to /tmp/silly.txt. UAC captured this file for us:

(venv) $ cat \[root\]/tmp/silly.txt 
100:1:password:worker

“worker” is the password for the worker account. The rest of the values here are hard-coded in the Father source code and are meaningless.

So even without static analysis I’m willing to state with high confidence that libymv.so.3 is the Father LD_PRELOAD rootkit. And since I implanted it into the system, I can also state authoritatively that this is exactly what the library is.

Is top Really XMRig?

Let’s try a different approach to discover whether our suspicious “top” process is really XMRig as the command history we extracted in Part 2 suggests. linux.proc.Maps allows us to dump the memory space of a process.

(venv) $ mkdir dump-mem-pid-977
(venv) $ vol -q -f memory_dump/avml.lime -o dump-mem-pid-977 linux.proc.Maps --pid 977 --dump
Volatility 3 Framework 2.27.1

PID	Process	Start	End	Flags	PgOff	Major	Minor	Inode	File Path	File output

977	top	0x400000	0x401000	r--	0x0	0	26	7 /dev/shm/kit/top (deleted)	pid.977.vma.0x400000-0x401000.dmp
977	top	0x401000	0xa5f000	r-x	0x1000	0	26	7 /dev/shm/kit/top (deleted)	pid.977.vma.0x401000-0xa5f000.dmp
977	top	0xa5f000	0xc40000	r--	0x65f000	0	26	7	 /dev/shm/kit/top (deleted)	pid.977.vma.0xa5f000-0xc40000.dmp
[...]
(venv) $ grep -rlF xmrig dump-mem-pid-977/
dump-mem-pid-977/pid.977.vma.0xa5f000-0xc40000.dmp
(venv) $ strings -a dump-mem-pid-977/pid.977.vma.0xa5f000-0xc40000.dmp
[...]
Usage: xmrig [OPTIONS]
Network:
  -o, --url=URL                 URL of mining server
  -a, --algo=ALGO               mining algorithm https://xmrig.com/docs/algorithms
      --coin=COIN               specify coin instead of algorithm
[...]

Finding the help text for the xmrig program in memory is conclusive enough for me. This is not a terribly surprising result, but it is nice to have additional confirmation.

Note that I tried a similar approach with linux.proc.Maps to try and recover the “config” file from the outbound SSH process (PID 975), but was unsuccessful. But while searching the memory strings for “:3333“, you can see some of the chatter from the XMRig process down the SSH tunnel.

[...]
4a747a5e140b7eb34933347c5154cb3d671bedc2e83b3a5e16f1603a4b660000
127.0.0.1:3333
method":"submit","params":{"id":"c1dcc2a9","job_id":"77","nonce":"ef000000","result":"982e4063bea92330be26b808c7d21d3fc349bc06f0c232fd673b7a827ab30000","algo":"rx/0"}}
"cn/rto","cn/rwz","cn/zls","cn/double","cn/ccx","cn-lite/1","cn-heavy/0","cn-heavy/tube","cn-heavy/xhv","cn-pico","cn-pico/tlo","cn/upx2","rx/0","rx/wow","rx/arq","rx/graft","rx/sfx","rx/yada","argon2/chukwa","argon2/chukwav2","argon2/ninja","ghostrider"]}}
[...]

Timeline Analysis

I’m passionate about timeline analysis, and will often use it in the early stages of a case to find indicators. However, we’ve been very successful with the other evidence sources that UAC provided and timeline analysis hasn’t been a priority. But now we can use timeline analysis to fill in any details we might have missed.

First we need to create the timeline from the body file provided by UAC with the help of the Sleuthkit’s mactime tool (“mactime -d -y -b bodyfile/bodyfile.txt >bodyfile/timeline.csv“). Alternatively, you could use my ptt.sh script which creates a timeline that merges file system inform with security log information including user logins, Sudo commands, etc.

After loading the timeline into our CSV viewer of choice, we can jump to 2026-03-24 23:22:19– the time of the “worker” login for the session where the Father rootkit was implanted. As usual, there is a lot of noise in the timeline, but the timeline generally confirms events we have discovered already.

Recall from Part 1 that the logs showed us a brief SSH session at 23:23:48. This session was not logged in /var/log/wtmp, indicating that it most likely was a single command or scp session that was not allocated a PTY and did not spawn an interactive shell. The timeline shows that at 23:23:48 the last access time on the “/run/shm -> /dev/shm” symlink was updated. Does this mean that the SSH connection at 23:23:48 was how the /dev/shm/kit directory was staged? This is certainly a plausible explanation, but not conclusive.

We know that the Father rootkit exfiltrates passwords in the file /tmp/silly.txt. The timeline shows us that this file was created at 23:34:32. This is when the SOC logged in as user worker to collect system data with UAC. This is just further confirmation that the Father rootkit was operational on the system.

Wrapping Up

At this point, we have answers for the important questions for the scenario:

  1. Is the system compromised? Definitely yes!
  2. How did the attackers gain access to the system? They logged in as user “worker“, using the account password “worker“. This password was easily guessable, or it may have been disclosed or stolen. User “worker” had unlimited Sudo access, so privilege escalation was trivial.
  3. Why is there unencrypted traffic on port 22/tcp? The attacker installed a rootkit that creates a backdoor in any networked service on the system, giving any user connecting from source port 48411/tcp a root shell on the system.
  4. What is consuming CPU on the system? Process ID 975 is the XMRig cryptocurrency miner running under the executable name “top“. This process is connecting out through an SSH tunnel to 192.168.5.95. This system is now in scope and must be investigated.
  5. Why can’t the SOC see what is happening on the system? The rootkit installed by the attacker is hiding multiple processes from the attacker’s backdoor session, including the cryptocurrency miner.

Here is our final timeline of important events during the incident:

23:22:19    User "worker" logs in with password from jump host (192.168.4.35) port 48364 [logs]
23:23:34    User "worker" uses sudo to execute root shell [logs]
23:23:48    Command-only/scp as user "worker" from jump host port 55504 [logs]
23:23:48    atime update on /dev/shm symlink, possible rootkit staging [bodyfile]
23:24:51    Father rootkit installed as /usr/lib/x86_64-linux-gnu/libymv.so.3 [bodyfile]
23:25:09    /etc/ld.so.preload created, points to .../libymv.so.3 [chkrootkit]
23:25:19    SSH daemon restarted [logs]
23:26:07    Unencrypted connect from 192.168.4.35 port 48411 via Father rootkit hook [memory]
23:26:22    python3 execution to promote raw shell [memory]
23:26:22    Hidden bash process started from python pty.spawn() [memory]
23:27:16    User "worker" SSH session from jump host port 48364 ends [logs]
23:27:51    /dev/shm/kit/xmrig renamed to "top" [memory]
23:28:17    ssh to 192.168.5.95 with tunnel on 3333/tcp [memory]
23:29:09    "top" process (renamed xmrig) started, comms via SSH tunnel [memory]
23:29:19    /dev/shm/kit removed [memory]
23:34:32    SOC logs in to start collecting data with UAC [logs]
23:34:32    Father rootkit stores "worker" password in /tmp/silly.txt [bodyfile]

Those of you who conducted your own investigation may have been thrown off by earlier artifacts left behind by my aborted attempts to create the scenario data. For example, there are login failures as user “worker” that could be construed as a brute force attack against this account, system crashes, and errors with libymv.so.3. My intention was that the scenario started with the login at 23:22:19, but kudos to those of you who found the earlier artifacts and invented plausible explanations for them.

Some submissions noted the fact that the kernel taint warning was triggered and speculated about a possible kernel rootkit. But only one submission actually ran down the source of the taint warning and realized it was due to the VirtualBox assistant and not enemy action. This is serious investigative dedication! Bravo!

One last part of this series is yet to come. I will be walking through what I actually did to create the scenario activity so that you can compare your answers with what really happened. In the meantime, don’t forget to check out the reports from the contest winners.

Linux Investigation (Part 2)

Hal Pomeranz3 Comments

Please read the previous installment of this investigation for additional background.

Having identified a potential LD_PRELOAD rootkit, I’m very curious to discover what the memory dump from the system can tell us. Note that before you can begin analyzing the memory for yourself you will need to (a) install Volatility, and (b) install a Linux profile that will work for this memory dump in .../volatility3/symbols/linux.

Process Information

One approach for finding suspicious processes is to look at the process hierarchy with linux.pstree. The Linux process hierarchy is usually rather flat, so interactive sessions tend to stand out:

(venv) $ vol -q -f memory_dump/avml.lime linux.pstree
[...]
* 0x8c7cc67e1980	937	937	1	sshd
** 0x8c7cc8278000	939	939	937	sh
*** 0x8c7cc67e4c80	940	940	939	python3
**** 0x8c7cc6011980	941	941	940	bash
***** 0x8c7cc6014c80	975	975	941	ssh
** 0x8c7cc81b1980	1005	1005	937	sshd-session
*** 0x8c7cc08b8000	1047	1047	1005	sshd-session
**** 0x8c7cc8354c80	1064	1064	1047	bash
***** 0x8c7cc351b300	1076	1076	1064	sudo
****** 0x8c7cc3859980	1078	1078	1076	sudo
******* 0x8c7cc039cc80	1079	1079	1078	bash
******** 0x8c7da0131980	119319	119319	1079	avml
[...]

The second hierarchy starting with PID 1005 is what SSH sessions normally look like: two sshd-session processes (privilege separation), then the login shell. The double sudo is an interesting wrinkle, but apparently an artifact of this user doing “sudo -s” to start a root shell rather than just “sudo bash“. Then we have the bash shell itself, where the user is running avml to grab the memory dump we are currently reviewing. This is more or less as expected.

The process hierarchy starting with PID 939, however, is just plain weird. The shell process with PID 939 is spawned directly out of the master SSH daemon, PID 937. This suggests some sort of exploit against the SSH daemon itself. Next comes a Python process (PID 940). Could this be the Python command from the original SOC report? Certainly PID 940 spawned a bash shell (PID 941), which matches what the SOC told us. That bash shell ran “ssh“– the SSH client program. What is going on here?

We can get more detail on these processes from the linux.psaux plugin:

(venv) $ vol -q -f memory_dump/avml.lime linux.psaux
[...]
939	937	sh	     /bin/sh
940	939	python3	 python3 -c import pty; pty.spawn("/bin/bash")
941	940	bash	 /bin/bash
[...]
975	941	ssh	     ssh -F config ymv
[...]

That’s definitely the Python command line the SOC reported to us. Apparently whatever is happening in the PID 939 shell is not happening inside of an encrypted session. This also tends to point towards some sort of pre-encryption and therefore pre-authentication compromise of the master SSH daemon. Recall from our log analysis in the last installment that the /etc/ld.so.preload file containing the path of the suspected rootkit library was created immediately before the SSH daemon was restarted. Are we looking at rootkit functionality for the SSH daemon compromise?

It’s also worth considering that ssh command line (PID 975). “-F config” means read client options from a file called “config“, but where was that file dropped to disk? Also, the SOC and the system admins for the machine have no idea what the host “ymv” is. It does not resolve within this network. Best guess is that the hostname “ymv” refers to configuration in the “config” file, wherever that ended up.

Command History From Memory

UAC didn’t find command history on disk, but the attacker’s bash shell was still active at the time the memory image was taken. So that command history should be in the memory dump.

(venv) $ vol -q -f memory_dump/avml.lime linux.bash --pid 941
Volatility 3 Framework 2.27.1

PID	Process	CommandTime	Command

941	bash	2026-03-24 23:26:26.000000 UTC	rest
941	bash	2026-03-24 23:26:30.000000 UTC	reset
941	bash	2026-03-24 23:26:44.000000 UTC	echo $SHELL
941	bash	2026-03-24 23:26:54.000000 UTC	id
941	bash	2026-03-24 23:27:25.000000 UTC	cd /dev/shm/kit
941	bash	2026-03-24 23:27:26.000000 UTC	ls
941	bash	2026-03-24 23:27:51.000000 UTC	mv xmrig top
941	bash	2026-03-24 23:27:59.000000 UTC	export PATH=.:$PATH
941	bash	2026-03-24 23:28:17.000000 UTC	ssh -F config ymv
941	bash	2026-03-24 23:28:28.000000 UTC	bg
941	bash	2026-03-24 23:29:09.000000 UTC	top -o 127.0.0.1:3333 -B
941	bash	2026-03-24 23:29:15.000000 UTC	cd ..
941	bash	2026-03-24 23:29:19.000000 UTC	rm -rf kit
941	bash	2026-03-24 23:29:24.000000 UTC	ps -ef | grep top
941	bash	2026-03-24 23:29:36.000000 UTC	ls

This shell history is extremely useful. After what is apparently an initial typo, the user executes a “reset” command, which is part of the typical recipe for elevating a raw shell to fully interactive via the Python pty.spawn() method. Our user then checks which shell they are in (“echo $SHELL“) and what user they are running as (“id“). This is classic post-exploit behavior: a typical user in a normal interactive session would not need to run these commands.

The user then changes to a directory named /dev/shm/kit. This is not a typical directory on this system, so where did it come from? The fact that it was staged in /dev/shm–a memory-based file system–is suspicious. This would be a good place for an attacker to stage files that they did not want to write to the system’s local disk.

Inside this directory was apparently a file named “xmrig“. XMRig is a popular Monero cryptocurrency miner. The user renames this file to “top” and later executes this “top” program from the /dev/shm/kit directory. “export PATH=.:$PATH” adds the current working directory to the front of the search path, followed later by “top -o 127.0.0.1:3333 -B” to execute the program. Note that the command line arguments to this “top” program match typical xmrig arguments– “-o 127.0.0.1:3333” to specify the mining infrastructure to connect to, and “-B” to run in the background as a daemon.

After starting the renamed xmrig, we see the user remove /dev/shm/kit (“cd ..“, “rm -rf kit“). This will not impact the running processes that were started here, but will make it more difficult for inexperienced investigators to find these files. We also see the user checking that their “top” program is still running or perhaps that it is invisible (“ps -ef | grep top“), and making sure that /dev/shm/kit is gone (“ls“).

But what is happening on localhost 3333/tcp? In between the “mv” command to rename the binary and executing the renamed xmrig as “top” we see “ssh -F config ymv” (later set to run in the background with “bg“) which we also saw in the linux.psaux output. Note that this command history implies that the “config” file was in /dev/shm/kit, now deleted.

It seems reasonable to assume that 127.0.0.1:3333 is an SSH tunnel. But can we find artifacts in memory to prove that?

Network Details

The linux.sockstat plugin should give us some insight into the network behavior on the system:

(venv) $ vol -q -f memory_dump/avml.lime linux.sockstat | grep -F 3333 | sort -u
4026531840	libuv-worker	977	979	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	libuv-worker	977	980	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	libuv-worker	977	981	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	libuv-worker	977	982	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	ssh	975	975	4	0x8c7cc404a900	AF_INET6	STREAM	TCP	::1	3333	::	0	LISTEN	-
4026531840	ssh	975	975	5	0x8c7cc4043900	AF_INET	STREAM	TCP	127.0.0.1	3333	0.0.0.0	0	LISTEN	-
4026531840	ssh	975	975	6	0x8c7cc405e880	AF_INET	STREAM	TCP	127.0.0.1	3333	127.0.0.1	59182	ESTABLISHED	-
4026531840	top	977	977	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	top	977	978	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	top	977	987	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	top	977	988	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	top	977	989	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-
4026531840	top	977	990	17	0x8c7cc405c280	AF_INET	STREAM	TCP	127.0.0.1	59182	127.0.0.1	3333	ESTABLISHED	-

The suspicious SSH process that was started (PID 975) is definitely listening on 127.0.0.1:3333. There is no “-L” option showing on the command line, so we presume the tunnel configuration is in the “config” file mentioned on the command line (“-F config“).

We can also see the “top” process, which is the renamed xmrig binary, actively talking on this port. This is consistent with the “-o 127.0.0.1:3333” option we saw in the command history. XMRig uses libuv for thread management, which is where the libuv-worker processes come from. Note that the top and libuv-worker entries share the same PID (977 in the third column of output) but different thread IDs (fourth column of output above).

But where is the SSH process connecting to? Let’s check the linux.sockstat output again:

(venv) $ vol -q -f memory_dump/avml.lime linux.sockstat | awk '$3 == "975"'
4026531840	ssh	975	975	3	0x8c7cc405cc00	AF_INET	STREAM	TCP	192.168.4.22	33440	192.168.5.95	22	ESTABLISHED	-
4026531840	ssh	975	975	4	0x8c7cc404a900	AF_INET6	STREAM	TCP	::1	3333	::	0	LISTEN	-
4026531840	ssh	975	975	5	0x8c7cc4043900	AF_INET	STREAM	TCP	127.0.0.1	3333	0.0.0.0	0	LISTEN	-
4026531840	ssh	975	975	6	0x8c7cc405e880	AF_INET	STREAM	TCP	127.0.0.1	3333	127.0.0.1	59182	ESTABLISHED	-

The outbound connection is to 192.158.5.95. We understood from the initial SOC report that all access to the system we are investigating is supposed to go through a jump host at 192.168.4.35, so this is a new system we have not heard of. The IP address is internal, and we will need to investigate this system to find out whether it too has been compromised. The scope of our investigation is growing!

We can also use linux.sockstat to investigate where our mysterious unencrypted SSH traffic is originating from. Here I am focusing in on only the activity related to the initial “sh” process (PID 939) that was created:

(venv) $ vol -q -f memory_dump/avml.lime linux.sockstat | awk '$3 == "939"'
4026531840	sh	939	939	0	0x8c7cc4059300	AF_INET	STREAM	TCP	192.168.4.22	22	192.168.4.35	48411	ESTABLISHED	-
4026531840	sh	939	939	1	0x8c7cc4059300	AF_INET	STREAM	TCP	192.168.4.22	22	192.168.4.35	48411	ESTABLISHED	-
4026531840	sh	939	939	2	0x8c7cc4059300	AF_INET	STREAM	TCP	192.168.4.22	22	192.168.4.35	48411	ESTABLISHED	-
4026531840	sh	939	939	8	0x8c7cc4059300	AF_INET	STREAM	TCP	192.168.4.22	22	192.168.4.35	48411	ESTABLISHED	-

It appears that this connection originated from the jump host system, 192.168.4.35 (source port 48411).

Detailed Process Information

The linux.pslist plugin will give us process start times as well as UID and GID info for all of our suspicious processes. The linux.pslist output is quite busy, so allow me to summarize the important pieces of info:

Process   PID  PPID  UID  GID    Started (UTC)
sh        939  937   0    7823   2026-03-24 23:26:07
python3   940  939   0    7823   2026-03-24 23:26:22
bash      941  940   0    7823   2026-03-24 23:26:22
ssh       975  941   0    7823   2026-03-24 23:28:32
top       977  1     0    7823   2026-03-24 23:29:24

One item that jumps out is that all of the processes are running as root (UID zero), but with the GID 7823. No other processes in the output are using this GID and the GID does not appear in the /etc/group file captured by UAC ([root]/etc/group). Rootkits will often use a group ID value to mark processes, files, and directories that should be hidden by the rootkit. That may be what is happening here.

Also note that the PPID of the “top” process is 1, which is systemd. This is an artifact of the “-B” option that was used to invoke the program. This option tells the process to run in the background like any other daemon process. In doing so the “top” process disassociates itself from the parent shell that spawned it, inheriting PPID 1. This is also why the “top” process did not appear in linux.pstree process hierarchy while the “ssh” command did.

Note that the start times for the “ssh” and “top” processes reported by linux.pslist differ from the timestamps from the command history. The linux.pslist times are approximately 15 seconds later than the corresponding command history timestamps. I have no explanation for this discrepancy, but will stick with the timestamps from the command history for consistency.

Status Check And Next Steps

Our timeline is filling in nicely. I’ve added a note at the end of each item to remind us where the information comes from:

23:22:19    User "worker" logs in with password from jump host (192.168.4.35) port 48364 [logs]
23:23:34    User "worker" uses sudo to execute root shell [logs]
23:23:48    Command-only/scp as user "worker" from jump host port 55504 [logs]
23:24:51    /usr/lib/x86_64-linux-gnu/libymv.so.3 created [bodyfile]
23:25:09    /etc/ld.so.preload created, points to .../libymv.so.3 [chkrootkit]
23:25:19    SSH daemon restarted [logs]
23:26:07    Unencrypted connect from 192.168.4.35 port 48411 spawns hidden sh (PID 939) [memory]
23:26:22    python3 execution to promote raw shell [memory]
23:26:22    Hidden bash process started from python pty.spawn() [memory]
23:27:16    User "worker" SSH session from jump host port 48364 ends [logs]
23:27:51    /dev/shm/kit/xmrig renamed to "top" [memory]
23:28:17    ssh to 192.168.5.95 with tunnel on 3333/tcp [memory]
23:29:09    "top" process (renamed xmrig) started, comms via SSH tunnel [memory]
23:29:19    /dev/shm/kit removed [memory]

From the timeline it appears that the suspicious library was planted during the original “worker” login session that started at 23:22:19. But once the hidden session was established and promoted via pty.spawn(), the legitimate login closed down. The hidden session was responsible for starting the SSH tunnel to 192.168.5.95 and running the renamed XMRig.

But there are still so many questions. How did the /dev/shm/kit directory and the suspicious library get onto the system in the first place and when? Can we recover the any of the suspect files– “libymv.so.3“, “top“, and the “config” file used for the SSH connection? Is “libymv.so.3” really an LD_PRELOAD rootkit as suspected? What else did the attacker accomplish on the system? More investigation in the next installment!

Linux Investigation (Part 1)

Hal Pomeranz4 Comments

I recently posted a Linux scenario that I had mocked up and asked for people to submit write-ups for judging. The winners have now been announced. Congratulations to all who participated!

I also wanted to provide an analysis of my own. Obviously, I know exactly what happened because I did everything. But I’m going to approach this investigation as if I were coming in cold, without any prior knowledge.

The scenario starts with an alert from the SOC containing two important pieces of information:

  • Unencrypted traffic on port 22/tcp, specifically the string “python3 -c 'import pty; pty.spawn("/bin/bash")'”
  • Heavy CPU usage but no process can be seen consuming the CPU

We have a UAC collection from the machine, including a full memory dump.

Initial Triage

“Heavy CPU usage but no process can be seen consuming the CPU” sounds a lot like a rootkit hiding processes to me, but let’s just confirm the SOC finding first. UAC collects the output from “top -b -n1” (live_response/process/top_-b_-n1.txt), and here’s the first part of that file:

top - 19:38:21 up 15 min,  1 user,  load average: 4.38, 3.44, 1.81
Tasks: 132 total,   1 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s): 97.7 us,  2.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   7947.4 total,   5206.4 free,   2780.6 used,    194.6 buff/cache     
MiB Swap:   1101.0 total,   1097.7 free,      3.3 used.   5166.7 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
      1 root      20   0   23812  14824  10756 S   0.0   0.2   0:00.70 systemd
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.00 kthreadd
      3 root      20   0       0      0      0 S   0.0   0.0   0:00.00 pool_wo+
[...]

Looking at line #3, the CPU is indeed maxed out. The process listing below the header information should be sorted by CPU utilization, but we see nothing that remotely accounts for the amount of CPU being consumed. Seems like the SOC made a good read here.

UAC has some modules that look for common rootkit indicators, so we look in the “chkrootkit” directory in the UAC output and hit pay dirt (for those of you playing along at home, the bodystat.sh script is available from my GitHub):

$ ls chkrootkit/
etc_ld_so_preload.txt  stat_etc_ld_so_preload.txt
$ cat chkrootkit/etc_ld_so_preload.txt 
/lib/x86_64-linux-gnu/libymv.so.3
$ cat chkrootkit/stat_etc_ld_so_preload.txt 
0|/etc/ld.so.preload|837578|-rw-r--r--|0|0|34|1774394719|1774394709|1774394709|1774394709
$ cat chkrootkit/stat_etc_ld_so_preload.txt | bodystat.sh 
  File: /etc/ld.so.preload
  Size: 34	UID: 0	GID: 0	Inode: 837578
Access: 2026-03-24 23:25:19
Modify: 2026-03-24 23:25:09
Change: 2026-03-24 23:25:09
 Birth: 2026-03-24 23:25:09

$ grep -F lib/x86_64-linux-gnu/libymv.so.3 bodyfile/bodyfile.txt | bodystat.sh 
  File: /usr/lib/x86_64-linux-gnu/libymv.so.3
  Size: 24024	UID: 0	GID: 0	Inode: 715378
Access: 2026-03-24 23:25:19
Modify: 2026-03-24 23:24:51
Change: 2026-03-24 23:24:51
 Birth: 2026-03-24 23:24:51

We have a suspicious library in /etc/ld.so.preload containing the library path /usr/lib/x86_64-linux-gnu/libymv.so.3. This libymv.so.3 file was created at 2026-03-24 23:24:51 UTC and /etc/ld.so.preload at 23:25:09.

There are many directions our investigation could go from this point, but I got curious to see if we could tie those file creation times to a particular login session on the machine. A quick look at live_response/system/last_-i.txt in the UAC data shows the following:

worker   pts/0        192.168.4.35     Tue Mar 24 19:34 - still logged in
worker   pts/0        192.168.4.35     Tue Mar 24 19:22 - 19:27  (00:04)
reboot   system boot  6.12.74+deb13+1- Tue Mar 24 19:21 - still running
[...]

There’s a time discrepancy here. If we assume that the session starting at 19:34 is where the UAC collection was made, this predates the arrival of the suspicious library by four hours. I suspect a local time zone issue.

$ ls -l \[root\]/etc/localtime
lrwxrwxrwx 1 hal hal 36 Mar 24 15:48 '[root]/etc/localtime' -> /usr/share/zoneinfo/America/New_York

The “last” output will be in the default time zone for the machine, which is apparently US/Eastern time. That’s four hours earlier than UTC at this time of year.

With that in mind, the creation times on our suspicious files line up nicely with the four minute session by user “worker” from 23:22 – 23:27 UTC (represented in the output above as 19:22 – 19:27). Unfortunately, UAC did not capture a .bash_history file for either the “worker” user or the “root” account. Anti-forensics is a possibility worth noting, but for now we don’t have any useful command history.

Looking at the security logs should provide more details about that login session. The only logs available are the Systemd journal, so it’s time to work some magic with journalctl.

$ journalctl -D \[root\]/var/log/journal/ -q -o short-iso --facility=auth,authpriv | grep -vE '(pam_unix|systemd-logind)'
[...]
2026-03-24T23:22:19+0000 vbox sshd-session[834]: Accepted password for worker from 192.168.4.35 port 48364 ssh2
2026-03-24T23:23:34+0000 vbox sudo[910]:   worker : TTY=pts/0 ; PWD=/home/worker ; USER=root ; COMMAND=/bin/bash
2026-03-24T23:23:48+0000 vbox sshd-session[914]: Accepted password for worker from 192.168.4.35 port 55504 ssh2
2026-03-24T23:23:48+0000 vbox sshd-session[923]: Received disconnect from 192.168.4.35 port 55504:11: disconnected by user
2026-03-24T23:23:48+0000 vbox sshd-session[923]: Disconnected from user worker 192.168.4.35 port 55504
2026-03-24T23:25:19+0000 vbox sshd[801]: Received signal 15; terminating.
2026-03-24T23:25:19+0000 vbox sshd[937]: Server listening on 0.0.0.0 port 22.
2026-03-24T23:25:19+0000 vbox sshd[937]: Server listening on :: port 22.
2026-03-24T23:27:16+0000 vbox sshd-session[834]: syslogin_perform_logout: logout() returned an error
2026-03-24T23:27:16+0000 vbox sshd-session[873]: Received disconnect from 192.168.4.35 port 48364:11: disconnected by user
2026-03-24T23:27:16+0000 vbox sshd-session[873]: Disconnected from user worker 192.168.4.35 port 48364
[...]

The “worker” user logs in (with a password) at 23:22:19, and uses sudo to get a root shell at 23:23:34.

But at 23:23:48, we see a second SSH session for “worker” that did not appear in the “last” output. Typically this means that the session did not spawn an interactive shell and was not allocated a PTY. That would indicate that the session only ran a single command specified by the remote user on their command line, or was possibly an scp. This idea is supported by the fact that the session connected and disconnected in the same second.

At 23:25:19 we see the SSH server restarted. Curiouser and curiouser.

What We Know So Far

At this point we’re starting to build a timeline of the incident:

23:22:19    User "worker" logs in with password from jump host (192.168.4.35) port 48364
23:23:34    User "worker" uses sudo to execute /bin/bash (root shell)
23:23:48    Command-only/scp as user "worker" from jump host (port 55504)
23:24:51    /usr/lib/x86_64-linux-gnu/libymv.so.3 created
23:25:09    /etc/ld.so.preload created (points to .../libymv.so.3)
23:25:19    SSH daemon restarted
23:27:16    User "worker" SSH session from jump host ends (source port 48364)

With everything laid out like this, it becomes obvious that the /etc/ld.so.preload file was created immediately before the SSH daemon was restarted. Since the only person operating on the system at the time was our suspected attacker, it is reasonable to assume that the SSH daemon was restarted so that it would pick up the suspicious libymv.so.3 library.

libymv.so.3 certainly has all the indications of being an LD_PRELOAD type rootkit. Memory analysis is a good path for further investigation, so we will pick up there in our next installment.

Fun With volshell

Hal Pomeranz1 Comment

When triaging a collection of memory images, I often find myself running multiple Volatility plugins on each image. Typically I do this by shell script, calling each plugin individually and saving the output in files. The problem with this approach is that Volatility has to re-parse the memory image each time my script calls a new plugin. This adds a lot of overhead and time. I started wondering if I could leverage volshell to run multiple plugins so that I wouldn’t have to pay the startup cost each time.

volshell Basics

volshell is an interactive shell environment for exploring a memory image. Explaining all the features of volshell would fill a book, so we’re just going to focus on the basics of starting up volshell and running plugins.

Starting volshell is straightforward. Specify a memory image with “-f” and the OS it comes from with “-w“, “-m“, or “-l” (Windows, MacOS, Linux, respectively). If we don’t want the progress meter as it’s ingesting the memory image, we can add “-q” (“quiet” mode).

$ volshell -f avml.lime -l -q
Volshell (Volatility 3 Framework) 2.27.1
Readline imported successfully

Call help() to see available functions

Volshell mode        : Linux
Current Layer        : layer_name
Current Symbol Table : symbol_table_name1
Current Kernel Name  : kernel

(layer_name) >>>

As the startup text suggests, help is available at any time by running the help() function:

(layer_name) >>> help()

Methods:
...
* dpo, display_plugin_output
    Displays the output for a particular plugin (with keyword arguments)
...

volshell methods generally have both long and abbreviated forms. For example, we’ll be using the display_plugin_output() method to run plugins. But rather than type that long string each time, we can just use dpo() instead.

For a simple example, let’s run the linux.ip.Addr plugin via volshell:

(layer_name) >>> from volatility3.plugins.linux import ip
(layer_name) >>> dpo(ip.Addr, kernel = self.config['kernel'])

NetNS	Index	Interface	MAC	Promiscuous	IP	Prefix	Scope Type	State

4026531840	1	lo	00:00:00:00:00:00	False	127.0.0.1	8	host	UNKNOWN
4026531840	1	lo	00:00:00:00:00:00	False	::1	128	host	UNKNOWN
4026531840	2	enp0s3	08:00:27:3a:05:32	False	192.168.4.22	22	global	UP
4026531840	2	enp0s3	08:00:27:3a:05:32	False	fdb0:fa27:86c5:1:19cf:7bad:b8a6:c5d7	64	global	UP
4026531840	2	enp0s3	08:00:27:3a:05:32	False	fdb0:fa27:86c5:1:a00:27ff:fe3a:532	64	global	UP
4026531840	2	enp0s3	08:00:27:3a:05:32	False	fe80::a00:27ff:fe3a:532	64	link	UP
4026532287	1	lo	00:00:00:00:00:00	False	127.0.0.1	8	host	UNKNOWN
4026532287	1	lo	00:00:00:00:00:00	False	::1	128	host	UNKNOWN
4026532345	1	lo	00:00:00:00:00:00	False	127.0.0.1	8	host	UNKNOWN
4026532345	1	lo	00:00:00:00:00:00	False	::1	128	host	UNKNOWN
4026532403	1	lo	00:00:00:00:00:00	False	127.0.0.1	8	host	UNKNOWN
4026532403	1	lo	00:00:00:00:00:00	False	::1	128	host	UNKNOWN
(layer_name) >>>

First we need to import the Volatility class that contains the plugin we want to invoke. The basic syntax here is “from volatility3.plugins.<os> import <class>“. “<os>” will be “windows“, “mac“, or “linux“. Since we’re running a Linux plugin, “<class>” will be the word that appears after “linux.” in the plugin name. For example, if were trying to run “linux.elfs.Elfs“, then “<class>” is “elfs“.

We use the dpo() method to actually run the plugin and get the output. If we were invoking the plugin on the command line, we would specify “linux.ip.Addr” as the plugin name. But here in Linux volshell, we can leave off the “linux.“. After the plugin name always specify “kernel = self.config['kernel']” to satisfy the dpo() method’s syntax.

If you want to run another plugin, just repeat the pattern. Import the appropriate Volatility class and run dpo() as before:

(layer_name) >>> from volatility3.plugins.linux import pstree
(layer_name) >>> dpo(pstree.PsTree, kernel = self.config['kernel'])

OFFSET (V)	PID	TID	PPID	COMM

0x8c7cc0281980	1	1	0	systemd
* 0x8c7cc0d79980	310	310	1	systemd-journal
* 0x8c7cc8268000	357	357	1	systemd-timesyn
* 0x8c7cc81a6600	365	365	1	systemd-udevd
* 0x8c7cc67e0000	687	687	1	avahi-daemon
** 0x8c7cc9986600	718	718	687	avahi-daemon
...

What’s great about this approach is that the memory image was already parsed when volshell started up. So each plugin runs very quickly.

Changing Output Modes

In many cases, it’s better for my workflow to get the plugin output in JSON format rather than the standard text output. I’d be embarrassed to admit how long the following little recipe took me to figure out, so let’s just get to the code:

(layer_name) >>> from volatility3.cli import text_renderer
(layer_name) >>> from volatility3.plugins.linux import psaux
(layer_name) >>> treegrid = gt(psaux.PsAux, kernel = self.config['kernel'])
(layer_name) >>> treegrid.populate()
(layer_name) >>> rt(treegrid,text_renderer.JsonLinesRenderer())

{"ARGS": "/sbin/init", "COMM": "systemd", "PID": 1, "PPID": 0, "__children": []}
{"ARGS": "[kthreadd]", "COMM": "kthreadd", "PID": 2, "PPID": 0, "__children": []}
{"ARGS": "[pool_workqueue_]", "COMM": "pool_workqueue_", "PID": 3, "PPID": 2, "__children": []}
{"ARGS": "[kworker/R-kvfre]", "COMM": "kworker/R-kvfre", "PID": 4, "PPID": 2, "__children": []}
{"ARGS": "[kworker/R-rcu_g]", "COMM": "kworker/R-rcu_g", "PID": 5, "PPID": 2, "__children": []}
...

First we’re importing the text_renderer class from volatility3.cli. This class contains methods for outputting various text formats, like JsonLinesRenderer() for single-line JSON format. Other options include JsonRenderer() for “pretty-printed” JSON output, or CSVRenderer() for comma-separated values formatting.

Next we import the class for Volatility plugin we want to invoke, just as before. But rather than calling dpo(), we create a new treegrid object with generate_treegrid() (abbreviated “gt()“). The arguments to gt() are the same as those for dpo().

gt() merely creates the treegrid object. We still have to call the treegrid.populate() method to load data into the object. Once we have populated the treegrid with data, we can invoke render_treegrid() (“rt()“) to output the data with our chosen text renderer.

Stumbling Towards Automation

Clearly this approach requires a lot of redundant typing. Automating the task of running multiple plugins through volshell is clearly the next step. My ptt.sh script has an initial attempt at this. At some point, I’d like to turn this idea into a standalone script outside of ptt.sh.

Linux Forensic Scenario

Hal Pomeranz1 Comment

Let’s try something a little different for today’s blog post!

I’ve been working on ideas for a major update on my Linux forensics class, including new lab scenarios. I recently threw together a rough draft of one of my scenario ideas: built a machine, planted some malware on it, and then used UAC to capture forensic data from the system. I was pleased with the results, and thought I would share them with the larger community.

And then I thought, why not turn it into a bit of a contest? For the moment I haven’t decided on any prizes other than bragging rights, but you never know. I have decided that the deadline for submissions for judging will be April 15th– tax day here in the USA.

The Scenario

You received an escalation from your SOC. They received an alert from their NMS about suspicious traffic to one of the Linux workers in the development group’s CI/CD pipeline. The alert was for unencrypted traffic on port 22/tcp, specifically the string “python3 -c 'import pty; pty.spawn("/bin/bash")'” which triggered the alert for “reverse shell promotion” in the NMS. They note that the system is showing signs of heavy CPU usage but that they don’t see any process(es) that account for this. Following their SOP, they acquired data from the system using UAC and have escalated to you as on-call for the internal IR/Threat team.

Other information about the system:

  • There is a single shared account on the system called “worker“. It has full Sudo privileges with the NOPASSWD option set.
  • All network access to the box is through a jump host at IP 192.168.4.35.
  • The UAC collection is uac-vbox-linux-20260324234043.tar.gz

Additional Comments

I threw this scenario together in a matter of hours, so when you look at the timeline of the system you will see that it got built and then compromised very quickly. For the final scenario I will doubtless do a more complete job running fake workloads for some time before the “attack” actually happens.

Similarly, you’ll probably discover that there is no significant network infrastructure around the compromised system. The “jump host” is really just another host in my lab environment that I was operating from.

But I still think there’s plenty of interesting artifacts to find in this scenario. I’m leaving things deliberately open-ended because I want to see what people come up with. But the goal would be to at least account for the issues raised by the SOC: why is there unencrypted traffic on 22/tcp, why is the system burning CPU, and why can’t the SOC see what is going on? Is the system compromised? When and how did that happen?

Submissions

Submissions for judging must be received no later than 23:59 UTC on 2026-04-15. I will accept submissions in .docx, PDF, or text. You may email your submissions to hrpomeranz@gmail.com. Please try to put something like “Linux Forensic Scenario Submission” in the Subject: line to make my life easier.

Depending on the number of submissions I get, I may need more folks to help with the judging. If you’re not planning to compete but would like to help judge, please drop me a line at the email address above. I’ll let you know if I need the help once I count the number (and length) of the submissions.

Happy forensicating! Have fun!

A Little More on LKM Persistence

Hal Pomeranz2 Comments

In my previous blog post I demonstrated a method for persisting a Linux LKM rootkit across reboots by leveraging systemd-modules-load. For this method to work, we needed to add the evil module into the /usr/lib/modules/$(uname -r) directory and then run depmod. As I pointed out in the article, while the LKM could hide the module object itself, the modprobe command invoked by systemd-modules-load requires the module name to be listed in the modules.dep and modules.dep.bin files created by depmod.

But a few days later it occurred to me that the module name actually only has to appear in the modules.dep.bin file in order to be loaded. modules.dep is an intermediate file that modules.dep.bin is built from. The modprobe command invoked by systemd-modules-load only looks at the (trie structured) modules.dep.bin file. So once modules.dep.bin is created, the attacker could go back and remove their evil module name from modules.dep.

I tested this on my lab system, installing the LKM per my previous blog post and then editing the evil module name out of modules.dep. When I rebooted my lab system, I verfied that the evil module was loaded by looking for the files that are hidden by the rootkit:

# ls /usr/lib/modules-load.d/
fwupd-msr.conf  open-vm-tools-desktop.conf
# ls /usr/lib/modules/$(uname -r)/kernel/drivers/block
aoe     drbd       loop.ko   nbd.ko       pktcdvd.ko  rsxx    sx8.ko   virtio_blk.ko  xen-blkfront.ko
brd.ko  floppy.ko  mtip32xx  null_blk.ko  rbd.ko      skd.ko  umem.ko  xen-blkback    zram

If the rootkit was not operating, we’d see the zaq123edcx* file in each of these directories.

I thought about writing some code to unpack the format of modules.dep.bin. This format is well documented in the comments of the source code for depmod.c. But then I realized that there was a much easier way to find the evil module name hiding in modules.dep.bin.

depmod works by walking the directory structure under /usr/lib/modules/$(uname -r) and creating modules.dep based on what it finds there. If we run depmod while the LKM is active, then depmod will not see the evil kernel object and will build a new modules.dep and modules.dep.bin file without the LKM object listed:

# cd /usr/lib/modules/$(uname -r)
# cp modules.dep modules.dep.orig
# cp modules.dep.bin modules.dep.bin.orig
# depmod
# diff modules.dep modules.dep.orig
# diff modules.dep.bin modules.dep.bin.orig
Binary files modules.dep.bin and modules.dep.bin.orig differ

The old and new modules.dep files are the same, since I had previously removed the evil module name by hand. But the *.bin* files differ because the evil module name is still lurking in modules.dep.bin.orig.

And I don’t need to write code to dump the contents of modules.dep.bin.orig— I’ll just use strings and diff:

# diff <(strings -a modules.dep.bin.orig) <(strings -a modules.dep.bin)
1c1
< ?=4_cs
---
> 4_cs
5610,5611d5609
< 123edcx_diamorphine
< <kernel/drivers/block/zaq123edcx-diamorphine.ko:
5616c5614
< 4ndemod
---
> demod
5619c5617
< enhua
---
> 5jenhua
5622a5621
> 7@53
5627d5625
< 8Z2c
5630c5628
< a2326
---
> 9Ja2326
5635c5633
< alloc
---
> <valloc

The output would be prettier with some custom tooling, but you can clearly see the name of the hidden object in the diff output.

From an investigative perspective, I really wish depmod had an option to write modules.dep.bin to an alternate directory. That would make it easier to perform these steps without modifying the state of the system under investigation. I suppose we could use overlayfs hacks to make this happen.

But honestly using modprobe to load your LKM rootkit is probably not the best approach. insmod allows you to specify the path to your evil module. Create a script that uses insmod to load the rootkit, and then drop the script into /etc/cron.hourly with a file name that will be hidden once the rootkit is loaded. Easy!

Linux LKM Persistence

Hal Pomeranz1 Comment

Back in August, Ruben Groenewoud posted two detailed articles on Linux persistence mechanisms and then followed that up with a testing/simulation tool called PANIX that implements many of these persistence mechanisms. Ruben’s work was, in turn, influenced by a series of articles by Pepe Berba and work by Eder Ignacio. Eder’s February article on persistence with udev rules seems particularly prescient after Stroz/AON reported in August on a long-running campaign using udev rules for persistence. I highly recommend all of this work, and frankly I’m including these links so I personally have an easy place to go find them whenever I need them.

In general, all of this work focuses on using persistence mechanisms for running programs in user space. For example, PANIX sets up a simple reverse shell by default (though the actual payload can be customized) and the “sedexp” campaign described by Stroz/AON used udev rules to trigger a custom malware executable.

Reading all of this material got my evil mind working, and got me thinking about how I might handle persistence if I was working with a Linux loadable kernel module (LKM) type rootkit. Certainly I could use any of the user space persistence mechanisms in PANIX that run with root privilege (or at least have CAP_SYS_MODULE capability) to call modprobe or insmod to load my evil kernel module. But what about other Linux mechanisms for specifically loading kernel modules at boot time?

Hiks Gerganov has written a useful article summarizing how to load Linux modules at boot time. If you want to be traditional, you can always put the name of the module you want to load into /etc/modules. But that seems a little too obvious, so instead we are going to use the more flexible systemd-modules-load service to get our evil kernel module installed.

systemd-modules-load looks in multiple directories for configuration files specifying modules to load, including /etc/modules-load.d, /usr/lib/modules-load.d, and /usr/local/lib/modules-load.d. systemd-modules-load also looks in /run/modules-load.d, but /run is typically a tmpfs style file system that does not persist across reboots. Configuration file names must end with “.conf” and simply contain the names of the modules to load, one name per line.

For my examples, I’m going to use the Diamorphine LKM rootkit. Diamorphine started out as a proof of concept rootkit, but a Diamorphine variant has recently been found in the wild. Diamorphine allows you to choose a “magic string” at compile time– any file or directory name that starts with the magic string will automatically be hidden by the rootkit once the rootkit is loaded into the kernel. In my examples I am using the magic string “zaq123edcx“.

First we need to copy the Diamorphine kernel module, typically compiled as diamorphine.ko, into a directory under /usr/lib/modules where it can be found by the modprobe command invoked by systemd-modules-load:

# cp diamorphine.ko /usr/lib/modules/$(uname -r)/kernel/drivers/block/zaq123edcx-diamorphine.ko
# depmod

Note that the directory under /usr/lib/modules is kernel version specific. You can put your evil module anywhere under /usr/lib/modules/*/kernel that you like. Notice that by using the magic string in the file name, we are relying on the rootkit itself to hide the module. Of course, if the victim machine receives a kernel update then your Diamorphine module in the older kernel directory will no longer be loaded and your evil plots could end up being exposed.

The depmod step is necessary to update the /usr/lib/modules/*/modules.dep and /usr/lib/modules/*/modules.dep.bin files. Until these files are updated, modprobe will be unable to locate your kernel module. Unfortunately, depmod puts the path name of your evil module into both of the modules.dep* files. So you will probably want to choose a less obvious name (and magic string) than the one I am using here.

The only other step needed is to create a configuration file for systemd-modules-load:

# echo zaq123edcx-diamorphine >/usr/lib/modules-load.d/zaq123edcx-evil.conf

The configuration file is just a single line– whatever name you copied the evil module to under /usr/lib/modules, but without the “.ko” extension. Here again we name the configuration file with the Diamorphine magic string so the file will be hidden once the rootkit is loaded.

That’s all the configuration you need to do. Load the rootkit manually by running “modprobe zaq123edcx-diamorphine” and rest easy in the knowledge that the rootkit will load automatically whenever the system reboots.

Finding the Evil

What artifacts are created by these changes? The mtime on the /usr/lib/modules-load.d directory and the directory where you installed the rootkit module will be updated. Aside from putting the name of your evil module into the modules.dep* files, the depmod command updates the mtime on several other files under /usr/lib/modules/*:

/usr/lib/modules/.../modules.alias
/usr/lib/modules/.../modules.alias.bin
/usr/lib/modules/.../modules.builtin.alias.bin
/usr/lib/modules/.../modules.builtin.bin
/usr/lib/modules/.../modules.dep
/usr/lib/modules/.../modules.dep.bin
/usr/lib/modules/.../modules.devname
/usr/lib/modules/.../modules.softdep
/usr/lib/modules/.../modules.symbols
/usr/lib/modules/.../modules.symbols.bin

Timestomping these files and directories could make things more difficult for hunters.

But loading the rootkit is also likely to “taint” the kernel. You can try looking at the dmesg output for taint warnings:

# dmesg | grep taint
[    8.390098] diamorphine: loading out-of-tree module taints kernel.
[    8.390112] diamorphine: module verification failed: signature and/or required key missing - tainting kernel

However, these log messages can be removed by the attacker or simply disappear due to the system’s normal log rotation (if the machine has been running long enough). So you should also look at /proc/sys/kernel/tainted:

# cat /proc/sys/kernel/tainted
12288

Any non-zero value means that the kernel is tainted. To interpret the value, here is a trick based on an idea in the kernel.org document I referenced above:

# taintval=$(cat /proc/sys/kernel/tainted)
# for i in {0..18}; do [[ $(($taintval>>$i & 1)) -eq 1 ]] && echo $i; done
12
13

Referring to the kernel.org document, bit 12 being set means an “out of tree” (externally built) module was loaded. Bit 13 means the module was unsigned. Notice that these flags correspond to the log messages found in the dmesg output above.

While this is a useful bit of command-line kung fu, I thought it might be useful to have in a more portable format and with more verbose output. So I present to you chktaint.sh:

$ chktaint.sh
externally-built (“out-of-tree”) module was loaded
unsigned module was loaded

By default chktaint.sh reads the value from /proc/sys/kernel/tainted on the live system. But in many cases you may be looking at captured evidence offline. So chktaint.sh also allows you to specify an alternate file path (“chktaint.sh /path/to/evidence/file“) or simply a raw numeric value from /proc/sys/kernel/tainted (“chktaint.sh 12288“).

The persistence mechanism(s) deployed by the attacker are often the best way to detect whether or not a system is compromised. If the attacker is using an LKM rootkit, checking /proc/sys/kernel/tainted is often a good first step in determining if you have a problem. This can be combined with tools like chkproc (find hidden processes) and chkdirs (find hidden directories) from the chkrootkit project.

More on EXT4 Timestamps and Timestomping

Hal Pomeranz1 Comment

Many years ago I did a breakdown of the EXT4 file system. I devoted an entire blog article to the new timestamp system used by EXT4. The trick is that EXT4 added 32-bit nanosecond resolution fractional seconds fields in its extended inode. But you only need 30 bits to represent nanoseconds, so EXT4 uses the lower two bits of the fractional seconds fields to extend the standard Unix “epoch time” timestamps. This allows EXT4 to get past the Y2K-like problem that normal 32-bit epoch timestamps face in the year 2038.

At the time I wrote, “With the extra two bits, the largest value that can be represented is 0x03FFFFFFFF, which is 17179869183 decimal. This yields a GMT date of 2514-05-30 01:53:03…” But it turns out that I misunderstood something critical about the way EXT4 handles timestamps. The actual largest date that can be represented in an EXT4 file system is 2446-05-10 22:38:55. Curious about why? Read on for a breakdown of how EXT4 timestamps are encoded, or skip ahead to “Practical Applications” to understand why this knowledge is useful.

Traditional Unix File System Timestamps

Traditionally, file system times in Unix/Linux are represented as “the number of seconds since 00:00:00 Jan 1, 1970 UTC”– typically referred to as Unix epoch time. But what if you wanted to represent times before 1970? You just use negative seconds to go backwards.

So Unix file system times are represented as signed 32-bit integers. This gives you a time range from 1901-12-13 20:45:52 (-2**31 or 0x80000000 or -2147483648 seconds) to 2038-01-19 03:14:07 (2**31 - 1 or 0x7fffffff or 2147483647 seconds). When January 19th, 2038 rolls around, unpatched 32-bit Unix and Linux systems are going to be having a very bad day. Don’t try to tell me there won’t be critical applications running on these systems in 2038– I’m pretty much basing my retirement planning on consulting in this area.

What Did EXT4 Do?

EXT4 had those two extra bits from the fractional seconds fields to play around with, so the developers used them to extend the seconds portion of the timestamp. When I wrote my infamous “timestamps good into year 2514” comment in the original article, I was thinking of the timestamp as an unsigned 34-bit integer 0x03ffffffff. But that’s not right.

EXT4 still has to support the original timestamp range from 1901 – 2038 and the epoch still has to be based around the January 1, 1970 or else chaos will ensue. So the meaning of the original epoch time values hasn’t changed. This field still counts seconds from -2147483648 to 2147483647.

So what about the extra two bits? With two bits you can enumerate values from 0-3. EXT4 treats these as multiples of 2*32 or 4294967296 seconds. So, for example, if the “extra” bits value was 2 you would start with 2 * 4294967296 = 8589934592 seconds and then add whatever value is in the standard epoch seconds field. And if that epoch seconds value was negative, you end up adding a negative number, which is how mathematicians think of subtraction.

This insanity allows EXT4 to cover a range from (0 * 4294967296 - 2147483648) aka -2147483648 seconds (the traditional 1901 time value) all the way up to (3 * 4294967296 + 2147483647) = 15032385535 seconds. That timestamp is 2446-05-10 22:38:55, the maximum EXT4 timestamp. If you’re still around in the year 2446 (and people are still using money) then maybe you can pick up some extra consulting dollars fixing legacy systems.

At this point you may be wondering why the developers chose this encoding. Why not just use the extra bits to make a 34-bit signed integer? A 34-bit signed integer would have a range from -2**33 = -8589934592 seconds to 2**33 - 1 = 8589934591 seconds. That would give you a range of timestamps from 1697-10-17 11:03:28 to 2242-03-16 12:56:31. Being able to set file timestamps back to 1697 is useful to pretty much nobody. Whereas the system the EXT4 developers chose gives another 200 years of future dates over the basic signed 34-bit date scheme.

Practical Applications

Why I am looking so closely at EXT4 timestamps? This whole business started out because I was frustrated after reading yet another person claiming (incorrectly) that you cannot set ctime and btime in Linux file systems. Yes, the touch command only lets you set atime and mtime, but touch is not the only game in town.

For EXT file systems, the debugfs command allows writing to inode fields directly with the set_inode_field command (abbreviated sif). This works even on actively mounted file systems:

root@LAB:~# touch MYFILE
root@LAB:~# ls -i MYFILE
654442 MYFILE
root@LAB:~# df -h .
Filesystem              Size  Used Avail Use% Mounted on
/dev/mapper/LabVM-root   28G   17G  9.7G  63% /
root@LAB:~# debugfs -w -R 'stat <654442>' /dev/mapper/LabVM-root | grep time:
 ctime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 atime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 mtime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
crtime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
root@LAB:~# debugfs -w -R 'sif <654442> crtime @-86400' /dev/mapper/LabVM-root
root@LAB:~# debugfs -w -R 'stat <654442>' /dev/mapper/LabVM-root | grep time:
 ctime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 atime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 mtime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
crtime: 0xfffeae80:8eb89330 -- Tue Dec 30 19:00:00 1969

The set_inode_field command needs the inode number, the name of the field you want to set (you can get a list of field names with set_inode_field -l), and the value you want to set the field to. In the example above, I’m setting the crtime field (which is how debugfs refers to btime). debugfs wants you to provide the value as an epoch time value– either in hex starting with “0x” or in decimal preceded by “@“.

What often trips people up when they try this is caching. Watch what happens when I use the standard Linux stat command to dump the file timestamps:

root@LAB:~# stat MYFILE
  File: MYFILE
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fe00h/65024d    Inode: 654442      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-09-02 09:58:13.598615244 -0400
Modify: 2024-09-02 09:58:13.598615244 -0400
Change: 2024-09-02 09:58:13.598615244 -0400
 Birth: 2024-09-02 09:58:13.598615244 -0400

The btime appears to be unchanged! The inode value has changed, but the operating system hasn’t caught up to the new reality yet. Once I force Linux to drop it’s out-of-date cached info, everything looks as it should:

root@LAB:~# echo 3 > /proc/sys/vm/drop_caches
root@LAB:~# stat MYFILE
  File: MYFILE
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: fe00h/65024d    Inode: 654442      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-09-02 09:58:13.598615244 -0400
Modify: 2024-09-02 09:58:13.598615244 -0400
Change: 2024-09-02 09:58:13.598615244 -0400
 Birth: 1969-12-30 19:00:00.598615244 -0500

If I wanted to set the fractional seconds field, that would be crtime_extra. Remember, however, that the low bits of this field are used to set dates far into the future:

root@LAB:~# debugfs -w -R 'sif <654442> crtime_extra 2' /dev/mapper/LabVM-root
root@LAB:~# debugfs -w -R 'stat <654442>' /dev/mapper/LabVM-root | grep time:
debugfs 1.46.2 (28-Feb-2021)
 ctime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 atime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
 mtime: 0x66d5c475:8eb89330 -- Mon Sep  2 09:58:13 2024
crtime: 0xfffeae80:00000002 -- Tue Mar 15 08:56:32 2242

For the *_extra fields, debugfs just wants a raw number either in hex or decimal (hex values should still start with “0x“).

Making This Easier

Human beings would like to use readable timestamps rather than epoch time values. The good news is that GNU date can convert a variety of different timestamp formats into epoch time values:

root@LAB:~# date -d '2345-01-01 12:34:56' '+%s'
11833925696

Specify whatever time string you want to convert after the -d option. The %s format means give epoch time output.

Now for the bad news. The value that date outputs must be converted into the peculiar encoding that EXT4 uses. And that’s why I spent so much time fully understanding the EXT4 timestamp format. That understanding leads to some crazy shell math:

# Calculate a random nanoseconds value
# Mask it down to only 30 bits, shift right two bits
nanosec="0x$(head /dev/urandom | tr -d -c 0-9a-f | cut -c1-8)"
nanosec=$(( ($nanosec & 0x3fffffff) << 2) ))

# Get an epoch time value from the date command
# Adjust the time value to a range of all positive values
# Calculate the number for the standard seconds field
# Calculate the bits needed in the *_extra field
epoch_time=$(date -d '2345-01-01 12:34:56' '+%s)
adjusted_time=$(( $epoch_time + 2147483648 ))
time_lowbits=$(( ($adjusted_time % 4294967296) - 2147483648 ))
time_highbits=$(( $adjusted_time / 4294967296 ))

# The *_extra field value combines extra bits with nanoseconds
extra_field=$(( $nanosec + $time_highbits ))

Clearly nobody wants to do this manually every time. You just want to do some timestomping, right? Don’t worry, I’ve written a script to set timestamps in EXT:

root@LAB:~# extstomp -v -macb -T '2123-04-05 1:23:45' MYFILE
===== MYFILE
 ctime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
 atime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
 mtime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
crtime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
root@LAB:~# extstomp -v -cb -T '2345-04-05 2:34:56' MYFILE
===== MYFILE
 ctime: 0xc1d6b290:61966bab -- Thu Apr  5 02:34:56 2345
 atime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
 mtime: 0x2044c7e1:ec154f7d -- Mon Apr  5 01:23:45 2123
crtime: 0xc1d6b290:61966bab -- Thu Apr  5 02:34:56 2345

Use the -macb options to specify the timestamps you want to set and -T to specify your time string. You can use -e to specify nanoseconds if you want, otherwise the script just generates a random nanoseconds value. The script is usually silent but -v causes the script to output the file timestamps when it’s done. The script even drops the file system caches automatically for you (unless you use -C to keep the old cached info).

And because you often want to blend in with other files in the operating system, I’ve included an option to copy the timestamps from another file:

root@LAB:~# extstomp -v -macb -S /etc/passwd MYFILE
===== MYFILE
 ctime: 0x66b37fc9:9d8e0ba8 -- Wed Aug  7 10:08:09 2024
 atime: 0x66d5c3cb:33dd3b30 -- Mon Sep  2 09:55:23 2024
 mtime: 0x66b37fc9:9d8e0ba8 -- Wed Aug  7 10:08:09 2024
crtime: 0x66b37fc9:9d8e0ba8 -- Wed Aug  7 10:08:09 2024
root@LAB:~# stat /etc/passwd
  File: /etc/passwd
  Size: 2356            Blocks: 8          IO Block: 4096   regular file
Device: fe00h/65024d    Inode: 1368805     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-09-02 09:55:23.217534156 -0400
Modify: 2024-08-07 10:08:09.660833002 -0400
Change: 2024-08-07 10:08:09.660833002 -0400
 Birth: 2024-08-07 10:08:09.660833002 -0400

You’re welcome!

What About Other File Systems?

It’s particularly easy to do this timestomping on EXT because debugfs allows us to operate on live file systems. In “expert mode” (xfs_db -x), the xfs_db tool has a write command that allows you to set inode fields. Unfortunately, by default xfs_db does not allow writing to mounted file systems. Of course, an enterprising individual could modify the xfs_db source code and bypass these safety checks.

And that’s really the bottom line. Use the debugging tool for whatever file system you are dealing with to set the timestamp values appropriately for that file system. It may be necessary to modify the code to allow operation on live file systems, but tweaking timestamp fields in an inode while the file system is running is generally not too dangerous.

Systemd Journal and journalctl

Hal Pomeranz

While I haven’t been happy about Systemd’s continued encroachment into the Linux operating system, I will say that the Systemd journal is generally an upgrade over traditional Syslog. We’ve reached the point where some newer distributions are starting to forgo Syslog and traditional Syslog-style logs altogether. The challenge for DFIR professionals is that the Systemd journals are in a binary format and require a command-line tool, journalctl, for searching and text output.

The main advantage that Systemd journals have over traditional Syslog-style logs is that Systemd journals carry considerably more metadata related to log messages, and this metadata is broken down into multiple searchable fields. A traditional Syslog log message might look like:

Jul 21 11:22:02 LAB sshd[1304]: Accepted password for lab from 192.168.10.1 port 56280 ssh2

The Systemd journal entry for the same message is:

{
        "_EXE" : "/usr/sbin/sshd",
        "_SYSTEMD_CGROUP" : "/system.slice/ssh.service",
        "_SELINUX_CONTEXT" : "unconfined\n",
        "SYSLOG_FACILITY" : "4",
        "_SYSTEMD_UNIT" : "ssh.service",
        "_UID" : "0",
        "SYSLOG_TIMESTAMP" : "Jul 21 07:22:02 ",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_TRANSPORT" : "syslog",
        "_SYSTEMD_SLICE" : "system.slice",
        "PRIORITY" : "6",
        "SYSLOG_IDENTIFIER" : "sshd",
        "_PID" : "1304",
        "_HOSTNAME" : "LAB",
        "__REALTIME_TIMESTAMP" : "1721560922218814",
        "_SYSTEMD_INVOCATION_ID" : "70a0b99512864d22a8f8b10752ad6537",
        "SYSLOG_PID" : "1304",
        "__MONOTONIC_TIMESTAMP" : "265429588",
        "_GID" : "0",
        "__CURSOR" : "s=743db8433dcc46ca9b9cecd7a4272061;i=1d6f;b=5c57e83c3abd457c95d0695807667c9e;m=fd22254;t=61dc0233a613e;x=31ff9c313be9c36f",
        "_CMDLINE" : "sshd: lab [priv]",
        "_MACHINE_ID" : "47b59f088dc74eb0b8544be4c3276463",
        "_COMM" : "sshd",
        "_BOOT_ID" : "5c57e83c3abd457c95d0695807667c9e",
        "MESSAGE" : "Accepted password for lab from 192.168.10.1 port 56280 ssh2",
        "_SOURCE_REALTIME_TIMESTAMP" : "1721560922218786"
}

Any of these fields is individually searchable. The journalctl command provides multiple pre-defined output formats, and custom output of specific fields is also supported.

Systemd uses a simple serialized text protocol over HTTP or HTTPS for sending journal entries to remote log collectors. This protocol uses port 19532 by default. The URL of the remote server is normally found in the /etc/systemd/journal-upload.conf file. On the receiver, configuration for handling the incoming messages is defined in /etc/systemd/journal-remote.conf. A “pull” mode for requesting journal entries from remote systems is also supported, using port 19531 by default.

Journal Time Formats

As you can see in the JSON output above, the Systemd journal supports multiple time formats. The primary format is a Unix epoch style UTC time with an extra six digits for microsecond precision. This is the format for the _SOURCE_REALTIME_TIMESTAMP and __REALTIME_TIMESTAMP fields. Archived journal file names (see below) use a hexadecimal form of this UTC time value.

Note that _SOURCE_REALTIME_TIMESTAMP is the time when systemd-journald first received the message on the system where the message was originally generated. If the message was later relayed to another system using the systemd-journal-remote service, __REALTIME_TIMESTAMP will reflect the time the message was received by the remote system. In the journal on the originating system, _SOURCE_REALTIME_TIMESTAMP and __REALTIME_TIMESTAMP are usually the same value.

I have created shell functions for converting both the decimal and hexadecimal representations of this time format into human-readable time strings:

function jtime { usec=$(echo $1 | cut -c11-); date -d @$(echo $1 | cut -c 1-10) "+%F %T.$usec %z"; }
function jhextime { usec=$(echo $((0x$1)) | cut -c11-); date -d @$(echo $((0x$1)) | cut -c 1-10) "+%F %T.$usec %z"; }

Journal entries also contain a __MONOTONIC_TIMESTAMP field. This field represents the number of microseconds since the system booted. This is the same timestamp typically seen in dmesg output.

Journal entries will usually contain a SYSLOG_TIMESTAMP field. This text field is the traditional Syslog-style timestamp format. This time is in the default local time zone for the originating machine.

Journal Files Location and Naming

Systemd journal files are typically found under /var/log/journal/MACHINE_ID. The MACHINE_ID is a random 128-bit value assigned to each system during the first boot. You can find the MACHINE_ID in the file /etc/machine-id file.

Under the /var/log/journal/MACHINE_ID directory, you will typically find multiple files:

root@LAB:~# ls /var/log/journal/47b59f088dc74eb0b8544be4c3276463/
system@00061db4ac78a3a2-03652b1534b78cc1.journal~
system@7166038d7a284f0f9f3c1aa7fab3f251-0000000000000001-0005f3e6144d8b0c.journal
system@7166038d7a284f0f9f3c1aa7fab3f251-0000000000001d59-0005f848158ef146.journal
system@7166038d7a284f0f9f3c1aa7fab3f251-00000000000061be-0005fbf9b35341f9.journal
system.journal
user-1000@00061db4b9047993-8eabd101686c1832.journal~
user-1000@a9d71fa481e0447a88f62416b6815868-0000000000001d7c-0005f8481f464fe7.journal
user-1000@c946abc41d224a5692053aa4e03ae012-00000000000007fe-0005f3e61585d7c2.journal
user-1000@e933964a572642cdb863a8803485cf10-0000000000006411-0005fbf9b50e0a5a.journal
user-1000.journal

The system.journal file is where logs from operating system services are currently being written. The other system@*.journal files are older, archived journal files. The systemd-journald process takes care of rotating the current journal and purging older files based on parameters configured in /etc/systemd/journald.conf.

The naming convention for these archived files is system@fileid-seqnum-time.journal. fileid is a random 128-bit file ID number. seqnum is the sequence number of the first message in the journal file. Sequence numbers are started at one and simply increase monotonically with each new message. time is the hexadecimal form of the standard journal UTC Unix epoch timestamp (see above). This time matches the __REALTIME_TIMESTAMP value of the first message in the journal– the time that message was received on the local system.

File names that end in a tilde– like the system@00061db4ac78a3a2-03652b1534b78cc1.journal~ file– are files that systemd-journald either detected as corrupted or which were ended by an unclean shutdown of the operating system. The first field after the “@” is a hex timestamp value corresponding to when the file was renamed as an archive. This is often when the system reboots, if the operating system crashed. I have been unable to determine how the second hex string is calculated.

In addition to the system*.journal files, the journal directory may also contain one or more user-UID*.journal files. These are user-specific logs where the UID corresponds to each user’s UID value in the third field of /etc/passwd. The naming convention on the user-UID*.journal files is the same as for the system*.journal files.

The journalctl Command

Because the journalctl command has a large number of options for searching, output formats, and more, I have created a quick one-page cheat sheet for the journalctl command. You may want to refer to the cheat sheet as you read through this section.

My preference is to “export SYSTEMD_PAGER=” before operating the journalctl command. Setting this value to null means that long lines in the journalctl output will wrap onto the next line of your terminal rather than creating a situation where you need to scroll the lines to the right to see the full message. If you want to look at the output one screenful at a time, you can simply pipe the output into less or more.

SELECTING FILES

By default the journalctl command operates on the local journal files in /var/log/journal/MACHINE_ID. If you wish to use a different set of files, you can specify an alternate directory with “-D“, e.g. “journalctl -D /path/to/evidence ...“. You can specify an individual file with “--file=” or use multiple “--file” arguments on a single command line. The “--file” option also accepts normal shell wildcards, so you could use “journalctl --file=system@\*” to operate just on archived system journal files in the current working directory. Note the extra backslash (“\“) to prevent the wildcard from being interpreted by the shell.

“journalctl --header” provides information about the contents of one or more journal files:

# journalctl --header --file=system@00061db4ac78a3a2-03652b1534b78cc1.journal~
File path: system@00061db4ac78a3a2-03652b1534b78cc1.journal~
File ID: a98f5eb8aff543a8abdee01518dd91f0
Machine ID: 47b59f088dc74eb0b8544be4c3276463
Boot ID: 9ac272cac6c040a7b9ad021ba32c2574
Sequential number ID: 7166038d7a284f0f9f3c1aa7fab3f251
State: OFFLINE
Compatible flags:
Incompatible flags: COMPRESSED-LZ4
Header size: 256
Arena size: 33554176
Data hash table size: 211313
Field hash table size: 333
Rotate suggested: no
Head sequential number: 27899 (6cfb)
Tail sequential number: 54724 (d5c4)
Head realtime timestamp: Sun 2024-07-14 16:18:40 EDT (61d3ad1816828)
Tail realtime timestamp: Sat 2024-07-20 08:55:01 EDT (61dad51f51a90)
Tail monotonic timestamp: 2d 1min 22.016s (284092380c)
Objects: 116042
Entry objects: 26326
Data objects: 64813
Data hash table fill: 30.7%
Field objects: 114
Field hash table fill: 34.2%
Tag objects: 0
Entry array objects: 24787
Deepest field hash chain: 2
Deepest data hash chain: 4
Disk usage: 32.0M

The most useful information here is the first (“Head“) and last (“Tail“) timestamps in the file along with the object counts.

OUTPUT MODES

The default output mode for journalctl is very similar to a typical Syslog-style log:

# journalctl -t sudo -r
-- Journal begins at Sat 2023-02-04 15:59:52 EST, ends at Sun 2024-07-21 13:00:01 EDT. --
Jul 21 07:34:01 LAB sudo[1491]: pam_unix(sudo:session): session opened for user root(uid=0) by lab(uid=1000)
Jul 21 07:34:01 LAB sudo[1491]:      lab : TTY=pts/1 ; PWD=/home/lab ; USER=root ; COMMAND=/bin/bash
Jul 21 07:22:09 LAB sudo[1432]: pam_unix(sudo:session): session opened for user root(uid=0) by lab(uid=1000)
Jul 21 07:22:09 LAB sudo[1432]:      lab : TTY=pts/0 ; PWD=/home/lab ; USER=root ; COMMAND=/bin/bash
-- Boot 93616c3bb5794e0099520b2bf974d1bc --
Jul 21 07:17:11 LAB sudo[1571]: pam_unix(sudo:session): session closed for user root
Jul 21 07:17:11 LAB sudo[1512]: pam_unix(sudo:session): session closed for user root
[...]

Note that here I am using the “-r” flag so that the most recent entries are shown first rather than the normal ordering of oldest to newest as you would normally read them in a log file.

The main differences between the default journalctl output and default Syslog-style output is the “Journal begins at...” header line and the markers that show which boot session the log messages were generated in. Like normal Syslog logs, the timestamps are shown in the default time zone for the machine where you are running the journalctl command.

If you want to hide the initial header, specify “-q” (“quiet“). If you want to force UTC timestamps, the option is “--utc“. You can hide the boot session information by choosing any one of several output modes with “-o“. Here is a single log message formatted with some of the different output choices:

-o short
Jul 21 07:33:36 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2

-o short-full
Sun 2024-07-21 07:33:36 EDT LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2

-o short-iso
2024-07-21T07:33:36-0400 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2

-o short-iso-precise
2024-07-21T07:33:36.610329-0400 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2

My personal preference is “-q --utc -o short-iso“. If you have a particular preferred output style, you might consider making it an alias so you’re not constantly having to retype the options. In my case the command would be “alias journalctl='journalctl -q --utc -o short-iso'“.

The “-o” option also supports several different JSON output formats. If you are looking to consume journalctl output with a script, you probably want “-o json” which formats all fields in each journal entry as a single long line of minified JSON. “-o json-pretty” is a multi-line output mode that I find useful when I’m trying to figure out which fields to construct my queries with. The JSON output at the top of this article was created with “-o json-pretty“.

In JSON output modes, you can output a custom list of fields with the “--output-fields=” option:

# journalctl -o json-pretty --output-fields=_EXE,_PID,MESSAGE
{
        "_EXE" : "/usr/sbin/sshd",
        "__REALTIME_TIMESTAMP" : "1721561616611216",
        "MESSAGE" : "Accepted password for lab from 192.168.10.1 port 56282 ssh2",
        "_BOOT_ID" : "5c57e83c3abd457c95d0695807667c9e",
        "__CURSOR" : "s=743db8433dcc46ca9b9cecd7a4272061;i=1e19;b=5c57e83c3abd457c95d0695807667c9e;m=3935b8a5;t=61dc04c9df790;x=d031b64e57796135",
        "_PID" : "1478",
        "__MONOTONIC_TIMESTAMP" : "959821989"
}
[...]

Notice that the __CURSOR, __REALTIME_TIMESTAMP, __MONOTONIC_TIMESTAMP, and _BOOT_ID fields are always printed even though we did not specifically select them.

“-o verbose --output-fields=...” gives only the requested fields plus __CURSOR but does so without the JSON formatting. “-o cat --output-fields=...” gives just the field values with no field names and no extra fields.

MATCHING MESSAGES

In general you can select messages you want to see by matching with “FIELD=value“, e.g. “_UID=1000“. You can specify multiple selectors on the same command line and the journalctl command assumes you want to logically “AND” the selections together (intersection). If you want logical “OR”, use a “+” between field selections, e.g. “_UID=0 + _UID=1000“.

Earlier I mentioned using “-o json-pretty” to help view fields that you might want to match on. “journalctl -N” lists the names of all fields found in the journal file(s), while “journalctl -F FIELD” lists all values found for a particular field:

# journalctl -N | sort
[...]
_SYSTEMD_UNIT
_SYSTEMD_USER_SLICE
_SYSTEMD_USER_UNIT
THREAD_ID
TID
TIMESTAMP_BOOTTIME
TIMESTAMP_MONOTONIC
_TRANSPORT
_UDEV_DEVNODE
_UDEV_SYSNAME
_UID
UNIT
UNIT_RESULT
USER_ID
USER_INVOCATION_ID
USERSPACE_USEC
USER_UNIT
# journalctl -F _UID | sort -n
0
101
104
105
107
110
114
117
1000
62803

Piping the output of “-F” and “-N” into sort is highly recommended.

Commonly matched fields have shortcut options:

--facility=   Matches on Syslog facility name or number
    journalctl -q -o short --facility=authpriv
    (Gives output just like typical /var/log/auth.log files)

-t            Matches SYSLOG_IDENTIFIER field
    journalctl -q -o short -t sudo
    (When you just want to see messages from Sudo)

-u            Matches _SYSTEMD_UNIT field
    journalctl -q -o short -u ssh.service
    (Messages from sshd, the ".service" is optional)

You can also do pattern matching against the log message text using the “-g” (“grep“) option. This option uses PCRE2 regular expression syntax. You might find this regular expression tester useful.

Options can be combined in any order:

# journalctl -q -r --utc -o short-iso -u ssh -g Accepted
2024-07-21T11:33:36+0000 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2
2024-07-21T11:22:02+0000 LAB sshd[1304]: Accepted password for lab from 192.168.10.1 port 56280 ssh2
2024-07-20T21:55:45+0000 LAB sshd[1559]: Accepted password for lab from 192.168.10.1 port 56278 ssh2
2024-07-20T21:44:55+0000 LAB sshd[1386]: Accepted password for lab from 192.168.10.1 port 56376 ssh2
[...]

TIME-BASED SELECTIONS

Specify time ranges with the “-S” (or “--since“) and “-U” (“--until“) options. The syntax for specifying dates and times is ridiculously flexible and is defined in the systemd.time(7) manual page. Here are some examples:

-S 2024-08-07 09:30:00
-S 2024-07-24
-U yesterday
-U “15 minutes ago”
-S -1hr
-S 2024-07-24 -U yesterday

The Systemd journal also keeps track of when the system reboots and allows you to select messages that happened during a particular operating sessions of the machine. “--list-boots” gives a list of all of the reboots found in the current journal files and “-b” allows you to select one or more sessions:

# journalctl --list-boots
 -3 f366a96b2f0a402a94e02eb57e10d431 Sun 2024-07-14 16:18:40 EDT—Thu 2024-07-18 08:53:12 EDT
 -2 9ac272cac6c040a7b9ad021ba32c2574 Thu 2024-07-18 08:53:45 EDT—Sat 2024-07-20 08:55:50 EDT
 -1 93616c3bb5794e0099520b2bf974d1bc Sat 2024-07-20 17:41:24 EDT—Sun 2024-07-21 07:17:12 EDT
  0 5c57e83c3abd457c95d0695807667c9e Sun 2024-07-21 07:17:40 EDT—Sun 2024-07-21 14:17:52 EDT
# journalctl -q -r --utc -o short-iso -u ssh -g Accepted -b -1
2024-07-20T21:55:45+0000 LAB sshd[1559]: Accepted password for lab from 192.168.10.1 port 56278 ssh2
2024-07-20T21:44:55+0000 LAB sshd[1386]: Accepted password for lab from 192.168.10.1 port 56376 ssh2

TAIL-LIKE BEHAVIORS

When using journalctl on a live system, “journalctl -f” allows you to watch messages coming into the logs in real time. This is similar to using “tail -f” on a traditional Syslog-style log. You may still use all of the normal selectors to filter messages you want to watch for, as well as specify the usual output formats.

“journalctl -n” displays the last ten entries in the journal, similar to piping the output into tail. You may optionally specify a numeric argument after “-n” if you want to see more or less than ten lines.

However, the “-n” and “-g” (pattern matching) operators have a strange interaction. The pattern match is only applied to the lines selected by “-n” along with your other selectors. For example, we can extract the last ten lines associated with the SSH service:

# journalctl -q --utc -o short-iso -u ssh -n
2024-07-21T11:17:41+0000 LAB systemd[1]: Starting OpenBSD Secure Shell server...
2024-07-21T11:17:42+0000 LAB sshd[723]: Server listening on 0.0.0.0 port 22.
2024-07-21T11:17:42+0000 LAB sshd[723]: Server listening on :: port 22.
2024-07-21T11:17:42+0000 LAB systemd[1]: Started OpenBSD Secure Shell server.
2024-07-21T11:22:02+0000 LAB sshd[1304]: Accepted password for lab from 192.168.10.1 port 56280 ssh2
2024-07-21T11:22:02+0000 LAB sshd[1304]: pam_unix(sshd:session): session opened for user lab(uid=1000) by (uid=0)
2024-07-21T11:33:36+0000 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2
2024-07-21T11:33:36+0000 LAB sshd[1478]: pam_unix(sshd:session): session opened for user lab(uid=1000) by (uid=0)
2024-07-21T19:56:09+0000 LAB sshd[4013]: Accepted password for lab from 192.168.10.1 port 56284 ssh2
2024-07-21T19:56:09+0000 LAB sshd[4013]: pam_unix(sshd:session): session opened for user lab(uid=1000) by (uid=0)

But matching the lines containing the “Accepted” keyword only matches against the ten lines shown above:

# journalctl -q --utc -o short-iso -u ssh -g Accepted -n
2024-07-21T11:22:02+0000 LAB sshd[1304]: Accepted password for lab from 192.168.10.1 port 56280 ssh2
2024-07-21T11:33:36+0000 LAB sshd[1478]: Accepted password for lab from 192.168.10.1 port 56282 ssh2
2024-07-21T19:56:09+0000 LAB sshd[4013]: Accepted password for lab from 192.168.10.1 port 56284 ssh2

From an efficiency perspective I understand this choice. It’s costly to seek backwards through the journal doing pattern matches until you find ten lines that match your regular expression. But it’s certainly surprising behavior, especially when your pattern match returns zero matching lines because it doesn’t happen to get a hit in the last ten lines you selected.

Frankly, I just forget about the “-n” option and just pipe my journalctl output into tail.

Further Reading

I’ve attempted to summarize the information most important to DFIR professionals, but there is always more to know. For further information, start with the journalctl(1) manual page. Keep your journalctl cheat sheet handy and good luck out there!