You know what Linux needs? Another task scheduling system!
said nobody ever
Important Artifacts
Command output:
- systemctl list-timers --all
- systemctl status *.timers
File locations:
- /usr/lib/systemd/system/*.{timer,service}
- /etc/systemd/system
- $HOME/.config/systemd
- [/var]/run/systemd/transient/*.{timer,service}
- [/var]/run/user/*/systemd/transient/*.{timer,service}
Also Syslog logs sent to LOG_CRON facility.
The Basics
If you’ve been busy trying to get actual work done on your Linux systems, you may have missed the fact that Systemd continues its ongoing scope creep and has added timers. Systemd timers are a new task scheduling system that provide similar functionality to the existing cron (Vixie cron and anacron) and atd systems in Linux. And so this creates another mechanism that attackers can leverage for malware activation and persistence.
Systemd timers provide both ongoing scheduled tasks similar to cron jobs (what the Systemd documentation calls realtime timers) as well as one-shot scheduled tasks (monotomic timers) that are similar to atd style jobs. Standard Systemd timers are configured via two files: a *.timer file and a *.service file. These files must live in standard Systemd configuration directories like /usr/lib/systemd/system or /etc/systemd/system.
The *.timer file generally contains information about when and how the scheduled task will run. Here’s an example from the logrotate.timer file on my local Debian system:
[Unit]
Description=Daily rotation of log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
[Timer]
OnCalendar=daily
AccuracySec=1h
Persistent=true
[Install]
WantedBy=timers.targetThis timer is configured to run daily within a one hour (AccuracySec=1h) random time window that Systemd deems is the most efficient time. Persistent=true means that the last time the timer ran will be tracked on disk. If the system sleeps during a period when this timer should have been activated, then the timer will run immediately on system wakeup. This is similar functionality to the traditional Anacron system in Linux.
The *.timer file may include a Unit= directive that specifies a *.service file to execute when the timer must run. However, as in this case, most *.timer files leave out the Unit= directive, which means that the timer will activate the corresponding logrotate.service file. The *.service file configures the program(s) the timer executes and other job parameters and security settings. Here’s the logrotate.service file from my Debian machine:
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
RequiresMountsFor=/var/log
ConditionACPower=true
[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
# hardening options
# details: https://www.freedesktop.org/software/systemd/man/systemd.exec.html
# no ProtectHome for userdir logs
# no PrivateNetwork for mail deliviery
# no NoNewPrivileges for third party rotate scripts
# no RestrictSUIDSGID for creating setgid directories
LockPersonality=true
MemoryDenyWriteExecute=true
PrivateDevices=true
PrivateTmp=true
ProtectClock=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectSystem=full
RestrictNamespaces=true
RestrictRealtime=trueTimers can be activated using the typical systemctl command line interface:
# systemctl enable logrotate.timer
Created symlink /etc/systemd/system/timers.target.wants/logrotate.timer → /lib/systemd/system/logrotate.timer.
# systemctl start logrotate.timersystemctl enable ensures the timer will be reactivated across reboots while systemctl start ensures that the timer is started in the current OS session.
You can get a list of all timers configured on the system (active or not) with systemctl list-timers --all:
# systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2024-05-05 18:33:46 UTC 1h 1min left Sun 2024-05-05 17:30:29 UTC 1min 58s ago anacron.timer anacron.service
Sun 2024-05-05 19:50:01 UTC 2h 17min left Sun 2024-05-05 16:37:36 UTC 54min ago apt-daily.timer apt-daily.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 54min ago exim4-base.timer exim4-base.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 54min ago logrotate.timer logrotate.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 54min ago man-db.timer man-db.service
Mon 2024-05-06 00:17:20 UTC 6h left Sun 2024-05-05 16:37:36 UTC 54min ago fwupd-refresh.timer fwupd-refresh.service
Mon 2024-05-06 01:13:52 UTC 7h left Sun 2024-05-05 16:37:36 UTC 54min ago fstrim.timer fstrim.service
Mon 2024-05-06 03:23:47 UTC 9h left Fri 2024-04-19 00:59:27 UTC 2 weeks 2 days ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2024-05-06 06:02:06 UTC 12h left Sun 2024-05-05 16:37:36 UTC 54min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2024-05-12 03:10:20 UTC 6 days left Sun 2024-05-05 16:37:36 UTC 54min ago e2scrub_all.timer e2scrub_all.service
10 timers listed.
systemctl status can give you details about a specific timer, including the full path to where the *.timer file lives and any related log output:
# systemctl status logrotate.timer
● logrotate.timer - Daily rotation of log files
Loaded: loaded (/lib/systemd/system/logrotate.timer; enabled; vendor preset: enabled)
Active: active (waiting) since Thu 2024-04-18 00:44:25 UTC; 2 weeks 3 days ago
Trigger: Mon 2024-05-06 00:00:00 UTC; 6h left
Triggers: ● logrotate.service
Docs: man:logrotate(8)
man:logrotate.conf(5)
Apr 18 00:44:25 LAB systemd[1]: Started Daily rotation of log files.Note that systemctl status *.timer will give this output for all timers on the system. This would be appropriate if you are quickly trying to gather this information for later triage.
If the command triggered by your timer produces output, look for that output with systemctl status <yourtimer>.service. For example:
# systemctl status anacron.service
● anacron.service - Run anacron jobs
Loaded: loaded (/lib/systemd/system/anacron.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2024-05-05 18:34:45 UTC; 28min ago
TriggeredBy: ● anacron.timer
Docs: man:anacron
man:anacrontab
Process: 1563 ExecStart=/usr/sbin/anacron -d -q $ANACRON_ARGS (code=exited, status=0/SUCCESS)
Main PID: 1563 (code=exited, status=0/SUCCESS)
CPU: 2ms
May 05 18:34:45 LAB systemd[1]: Started Run anacron jobs.
May 05 18:34:45 LAB systemd[1]: anacron.service: Succeeded.Systemd timer executions and command output are also logged to the LOG_CRON Syslog facility.
Transient Timers
Timers can be created on-the-fly without explicit *.timer and *.service files using the systemd-run command:
# systemd-run --on-calendar='*-*-* *:*:15' /tmp/.evil-lair/myStartUp.sh
Running timer as unit: run-rb68ce3d3c11a4ec79b508036776d2cb1.timer
Will run service as unit: run-rb68ce3d3c11a4ec79b508036776d2cb1.serviceIn this case, we are creating a timer that will run every minute of every hour of every day at 15 seconds into the minute. The timer will execute /tmp/.evil-lair/myStartUp.sh. Note that the systemd-run command requires that /tmp/.evil-lair/myStartUp.sh exist and be executable.
The run-*.timer and run-*.service files end up in [/var]/run/systemd/transient:
# cd /run/systemd/transient/
# ls
run-rb68ce3d3c11a4ec79b508036776d2cb1.service run-rb68ce3d3c11a4ec79b508036776d2cb1.timer session-2.scope session-71.scope session-c1.scope
# cat run-rb68ce3d3c11a4ec79b508036776d2cb1.timer
# This is a transient unit file, created programmatically via the systemd API. Do not edit.
[Unit]
Description=/tmp/.evil-lair/myStartUp.sh
[Timer]
OnCalendar=*-*-* *:*:15
RemainAfterElapse=no
# cat run-rb68ce3d3c11a4ec79b508036776d2cb1.service
# This is a transient unit file, created programmatically via the systemd API. Do not edit.
[Unit]
Description=/tmp/.evil-lair/myStartUp.sh
[Service]
ExecStart="/tmp/.evil-lair/myStartUp.sh"These transient timers can be monitored with systemctl list-timers and systemctl status just like any other timer:
# systemctl list-timers --all -l
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2024-05-05 17:59:15 UTC 17s left Sun 2024-05-05 17:58:19 UTC 37s ago run-rb68ce3d3c11a4ec79b508036776d2cb1.timer run-rb68ce3d3c11a4ec79b508036776d2cb1.service
Sun 2024-05-05 18:33:46 UTC 34min left Sun 2024-05-05 17:30:29 UTC 28min ago anacron.timer anacron.service
Sun 2024-05-05 19:50:01 UTC 1h 51min left Sun 2024-05-05 16:37:36 UTC 1h 21min ago apt-daily.timer apt-daily.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago exim4-base.timer exim4-base.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago logrotate.timer logrotate.service
Mon 2024-05-06 00:00:00 UTC 6h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago man-db.timer man-db.service
Mon 2024-05-06 00:17:20 UTC 6h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago fwupd-refresh.timer fwupd-refresh.service
Mon 2024-05-06 01:13:52 UTC 7h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago fstrim.timer fstrim.service
Mon 2024-05-06 03:23:47 UTC 9h left Fri 2024-04-19 00:59:27 UTC 2 weeks 2 days ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2024-05-06 06:02:06 UTC 12h left Sun 2024-05-05 16:37:36 UTC 1h 21min ago apt-daily-upgrade.timer apt-daily-upgrade.service
Sun 2024-05-12 03:10:20 UTC 6 days left Sun 2024-05-05 16:37:36 UTC 1h 21min ago e2scrub_all.timer e2scrub_all.service
11 timers listed.
# systemctl status run-rb68ce3d3c11a4ec79b508036776d2cb1.timer
● run-rb68ce3d3c11a4ec79b508036776d2cb1.timer - /tmp/.evil-lair/myStartUp.sh
Loaded: loaded (/run/systemd/transient/run-rb68ce3d3c11a4ec79b508036776d2cb1.timer; transient)
Transient: yes
Active: active (waiting) since Sun 2024-05-05 17:48:25 UTC; 10min ago
Trigger: Sun 2024-05-05 17:59:15 UTC; 3s left
Triggers: ● run-rb68ce3d3c11a4ec79b508036776d2cb1.service
May 05 17:48:25 LAB systemd[1]: Started /tmp/.evil-lair/myStartUp.sh.Note that these timers are transient because the /var/run directory is an in-memory file system. These timers, like the file system itself, will disappear on system shutdown or reboot.
Per-User Timers
Systemd also allows unprivileged users to create timers. The command line interface we’ve seen so far stays essentially the same except that regular users must add the --user flag to all commands. User *.timer and *.service files must be placed in $HOME/.config/systemd/user. Or the user could create a transient timer without explicit *.timer and *.service files:
$ systemd-run --user --on-calendar='*-*-* *:*:30' /tmp/.dropper/.src.sh
Running timer as unit: run-rdad04b63554a4ebeb12bc5ca42baaa31.timer
Will run service as unit: run-rdad04b63554a4ebeb12bc5ca42baaa31.serviceThe user can use systemctl list-timers and systemctl status to check on their timers:
$ systemctl list-timers --user --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Sun 2024-05-05 18:20:30 UTC 40s left n/a n/a run-rdad04b63554a4ebeb12bc5ca42baaa31.timer run-rdad04b63554a4ebeb12bc5ca42baaa31.service
1 timers listed.
$ systemctl status --user run-rdad04b63554a4ebeb12bc5ca42baaa31.timer
● run-rdad04b63554a4ebeb12bc5ca42baaa31.timer - /tmp/.dropper/.src.sh
Loaded: loaded (/run/user/1000/systemd/transient/run-rdad04b63554a4ebeb12bc5ca42baaa31.timer; transient)
Transient: yes
Active: active (waiting) since Sun 2024-05-05 18:19:35 UTC; 32s ago
Trigger: Sun 2024-05-05 18:20:30 UTC; 22s left
Triggers: ● run-rdad04b63554a4ebeb12bc5ca42baaa31.service
May 05 18:19:35 LAB systemd[1293]: Started /tmp/.dropper/.src.sh.As you can see in the above output, transient per-user run-*.timer and run-*.service files end up under [/var]/run/user/<UID>/systemd/transient.
Unfortunately, there does not seem to be a way for the administrator to conveniently query timers for regular users on the system. You’re left with consulting the system logs and grabbing whatever on-disk artifacts you can, like the user’s $HOME/.config/systemd directory.
Righteous IT 