- Created by Baptiste Grenier, last modified on 2023 Jun 23
This document describes a best-effort approach for preserving and analyzing compromised Linux installations. In many situations, errors in forensics can lead to destruction of evidences, so please take precaution and contact experts if you are unsure. This document proposes commands to run to obtain or analyse some data, but the responsibility of running them is yours: if you do not know them, read the man-page and/or contact experts.
You will find useful references or explanation in the section Forensics resources resources at the bottom of the page.
Preparation before incidents
The difficulty of any forensic varies on the effort attackers took to hide themselves. However, preparing yourself and your servers can ease forensics by a large amount
Prepare yourself
Check your procedures
Some questions should be answered before an incident happens, to make sure they don't hinder you while working on forensics:
- Who are you supposed to contact during incidents?
- Local security contacts?
- External CSIRT you depend on?
- What is your policy with regard to law enforcement? Will you pursue legal action?
- Do you legally have to report security breaches to your users?
- Who can decide to take systems and services down?
- Who will coordinate actions? (like: Who will answer any press contact?)
As a EGI resource centre the EGI CSIRT Security Incident Handling Procedure should be used to complement your local Incident Response Procedure.
Prepare some hardware
During forensics, you will most likely need:
- A USB key containing:
- A Linux distribution of your choice, with auto-mount disabled
- A static copy of forensics tools. We recommend in particular:
- Testdisk (For data recovery and carving)
- The Sleuth Kit (For offline file-system analysis and timeline)
- extundelete (For data recovery on ext3/ext4)
- Few large capacity USB hard drive to collect disk images and evidences
- [Optional] A SATA->USB converter to image disks offline (potentially with a write-blocking functionality) or a disk mirroring device.
Prepare your systems
- Collect all system and audit logs on a remote system with strict access control.
- Try to improve your file-system evidences:
- Disable 'prelink' (It alters binary metadata)
- Avoid cronjobs that read all files (Removes any meaning in the 'last access' metadata on file)
- Avoid mounting your system with 'noatime', use 'relatime' only if necessary
- Make sure to have basic tools installed on your systems that will help you collecting forensics data (you can't install them once you think you have an incident):
- netstat
- lsof: list of open files
- pstree (from psmisc): tree of processes
- gcore (from gdb): generate core images of processes
Remarks on live analysis
Data collection order
Data should be collected in a specific order, to avoid missing or destroying evidences.
Observation changes the observed object
On a live system:
- Every command you run will alter (at least) file system metadata (atime on that binary)
- Every data you write on disk can override data from files which were deleted before
Forensic data is volatile
On a live system, most of the data is volatile and usually survive only for a short time:
- Processor states, caches and RAM: Nano-seconds
- Network states: Milli-seconds
- Running processes: Seconds
- Disks: minutes
(List borrowed from “Forensic Discovery”, Farmer & Venema, Addison-Wesley 2005)
Please note that except for the last one, all these data are lost when the system is rebooted!
Danger from live system
Compromised tools
When performing live analysis, one has to remember that the system might have been modified to lie to you:
- The kernel might have been modified
- Binaries might have modified
- Libraries might have been modified or are injected
As a result, always doubt what you see and check if it makes sense:
- Compare output of different tools (e.g. ps VS pstree VS ls /proc/pid)
- Compare with external behaviour (e.g. network activity)
- Compare with one own's knowledge (e.g. missing files/activity)
Are you alone?
If the system has been compromised, malicious actors might still be present. If you are unlucky, they might still be around, detect your actions and start deleting evidences!
As a result, as soon as the breach is confirmed and basic network/process forensics evidences have been collected, the system should be isolated from other systems.
Live (Quick & Dirty) forensics
When you are not sure you have a case and/or for physical systems (forensics on virtual machines should concentrate on snapshots (including memory snapshots)), here is a guide to collect initial data from the live system.
All these actions MUST be performed before switching off the system, as some of the evidences would be destroyed otherwise!
Before you start
- Log every action you take, in order, with a time-stamp.
- Use paper and pen.
- Consider using script (check "man script") to log all your commands executed.
- If the case is complex, large and/or sensitive (e.g. suspicious of illegal data), consider imposing a 4 eyes policy
- If you connect to the system remotely, e.g. ssh, avoid using credentials that can be reused on other systems and consider them lost: change them as soon a possible
- Don't store data on the harddrive:
- Set HISTFILE to /dev/null
Store temporary files in a filesystem backed by a tmpfs (in RAM), remotely or on a USB drive. For example:
# pick one in this list, we consider /dev/shm here mount | grep tmpfs # verify that it is big enough df -h /dev/shm mkdir /dev/shm/work && cd /dev/shm/work
Collect live data
In your temporary location, start by collecting live data from the system:
netstat -apn | tee netstat_apn.txt
ip -4 neigh show | tee ip4_neigh_show.txt ip -4 route show | tee ip4_route_list.txt ip -4 link show | tee ip4_link_show.txt ip -6 neigh show | tee ip6_neigh_show.txt ip -6 route show | tee ip6_route_list.txt ip -6 link show | tee ip6_link_show.txt
w > w.txt last | tee last.txt lastlog | tee lastlog.txt
ps -auxwwwe | tee ps_auxwwwe.txt pstree -lap | tee pstree_lap.txt
lsof -b -l -P -X -n -o -R -U | tee lsof_blPXnoRU.txt lsof -b -l -P -X -n -o -R | tee lsof_blPXnoR.txt
cat /proc/mounts | tee proc_mounts.txt
cat /proc/modules | tee proc_modules ls /sys/modules |tee sys_modules
Collect malicious process data
Review rapidly the data you have collected and look for oddities, for example:
- Processes listening on a raw socket (not related to dhcp)
- Processes listening on ports that you don't know about, can't find on the internet
- Processes spawned by a process that should not do it usually
- Processes with network activity that should not do it usually
For each of such process:
Keep the PID number in a separate variable and folder:
export PID=12345 # <- INSERT PROCESS-ID (PID) HERE mkdir $PID cd $PID
Stop the process (can't be caught):
kill -STOP ${PID}
Copy the executable:
cp /proc/${PID}/exe ${PID}.exe
Dump a core of the process:
gcore ${PID}
If you don't have gcore, you need to do it manually:
gdb -p ${PID} # Type "gcore" # Type "detach" # Type "exit"
- Copy all interesting open files:
List files opened by the process:
lsof -np ${PID}
Copy 'deleted' files:
# FDNUM and FILENAME should be taken from the output of lsof cp /proc/${PID}/fd/${FDNUM} ${FILENAME}
Copy files stored in /dev/shm:
# FILENAME should be taken from the output of lsof cp /dev/shm/${FILENAME} ${FILENAME}
Copy all files from the proc folder:
tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan}
Get out of your temporary folder
cd ..
Collect filesystem metadata
For each locally mounted filesystem, you should collect their file metadata (access, modification and change times).
The list of local filesystem can usually be found with:
grep '^/dev/' /proc/mounts
For each of them:
Create a mount point:
mkdir mountpoint
Bind mount the existing mounted point there, e.g. for '/':
mount --bind / mountpoint
Remount it read-only (this cannot be done in a single step!):
mount -o remount,ro mountpoint
Get into the mount point:
cd mountpoint
- Get all metadata:
# replace 'root' with the actual filesystem you mounted find . -print0 \ | xargs -0 stat -c "%Y %X %Z %A %U %G %s %n" -- \ | tee ../root.files
Get out of the mountpoint and unmount it:
cd .. umount mountpoint
Once you have collected all this metadata and extracted it to another location, the timeline of each filesystem can be rebuilt using this script:
#! /usr/bin/python from __future__ import print_function from datetime import datetime import sys try: import pytz except: pass if len(sys.argv) > 1: try: timezone = pytz.timezone(sys.argv[1]) except: print("Impossible to use this timezone") sys.exit(-1) else: timezone = None def print_line(flags, t, mode, user, group, size, name): when = datetime.utcfromtimestamp(float(t)) if timezone is not None: try: when = pytz.utc.localize(when).astimezone(timezone) except: print("Timezone issue!") sys.exit(-1) print(' '.join([t, when.isoformat(), flags, mode, user, group, size, name])) for line in sys.stdin: line = line[:-1] (m, a, c, mode, user, group, size, name) = line.split(" ", 7) if m == a: if m == c: print_line("mac", m, mode, user, group, size, name) else: print_line("ma-", m, mode, user, group, size, name) print_line("--c", c, mode, user, group, size, name) else: if m == c: print_line("m-c", m, mode, user, group, size, name) print_line("-a-", a, mode, user, group, size, name) else: print_line("m--", m, mode, user, group, size, name) print_line("-a-", a, mode, user, group, size, name) print_line("--c", c, mode, user, group, size, name)
[Optional] Automated tests
You can run some automated tools to try to identify malicious activity/files, but you first need:
- To have these tools installed (e.g. chkrootkit, rkhunter, ossec-rootcheck)
To first remount all your local filesystems as read only. A first approach for this remount can use the following code, but you should rather review the list and remount them manually.
for mountpoint in $(grep '^/dev/' /proc/mounts | cut -d ' ' -f 2 | sort -r); do echo mount -o remount,ro $mountpoint done
You can also (after remounting all filesystems read only) use your package management system to verify installed packages (save the output). On RedHat based systems, this is rpm -Va, on Debian based system, debsums.
Stopping the system
Before stopping the system, remember to take out all the data you have already collected using e.g. scp or rsync if you extracted it in a tmpfs. You might also want to run 'sync' to make sure that all data is written on disk.
There are two ways of stopping a system:
- Run the normal shutdown procedure: this will close all running program properly, which would preserve e.g. databases, but would write a lot on the disk, potentially destroying evidences
- Simply unplug the power cables of the system: this will kill all running programs, potentially damaging e.g. databases, but would preserve more evidences.
We usually recommend the later.
Offline analysis
Most of the analysis usually happens offline. Here we have less time pressure.
Copying disks
One of the most important rules of forensics is to never worker on the original supports, thus you need to copy the data (some will even recommend to only work on copies of copies in order to avoid copying the original again if you have a corruption).
There are mainly 3 ways of copying a disk :
- Use a disk imaging/mirroring device
- Use an external hard drive docking bay
- Use an internal hot-swappable connector
For the last two possibility, one has to be very cautious not to modify any data:
- The operating system on which the disk is plugged must have any auto-mount feature disabled
- If more than one device is connected, they should be plugged in sequentially, to make sure that their identifiers are properly mapped
- Any command typed can irremediably destroy data and should be checked multiple time
Once drives are identified, we recommend using dd to image the disk:
# inverting if and of will destroy all evidences! dd if=/dev/sdX of=mybigfile.img bs=65536 conv=noerror,sync
Collect filesystem metadata
While filesystem metadata can be collected from live systems, collecting it from cold disk have some advantages:
- While live system analysis can be affected by rootkit, this one cannot
- On ext4 filesystem, normal live tools do not extract the creation time. Forensic tools do
We recommend using fls and mactime from the Sleuth Kit for collecting such metadata, see the corresponding wiki page: http://wiki.sleuthkit.org/index.php?title=Timeline
Carving unallocated blocks
Data on file-systems is not deleted when a file is deleted, but is kept on unallocated data blocks. These blocks might be overridden by the content of new files. Until then, that data can be recovered from the harddrive:
- For a given pattern (e.g. sshd logs), it’s possible (and usually fruitful) to simply grep the raw disk
- Dedicated tools, like PhotoRec from the testdisk utility, can try to rebuild files based on built-in patterns
Exploiting forensic data
There is one basic thing to remember when doing forensics: always try to assert how much you can trust what you see. Most of it is just bits on some disks, which could have been altered by an attacker, thus:
- Never fully trust data from a compromised system, ask yourself if it could have been tempered with and what would that mean
- Try to corroborate different sources, either between different local sources or with external sources
- Check if what you are seeing is possible on a normal system, as part of the normal workflow
Reading file system timelines
When looking a a file system timeline, one should first remember that:
- atime and mtime can easily be faked and are manipulated by the system itself (e.g. when extracting files from archives, downloading files, etc). You should start by ignoring them and only look at them when you understand the basic of the activity on the system
- ctime are less likely to be manipulated
- crtime and dtime are rarely manipulated
We recommend looking for the following artefacts:
- Weird folder names (e.g. '...'), content in temporary and world-writable folders (e.g /var/tmp, /dev/shm, /tmp)
- Binary or library ctime or crtime modified while the package was not updated (no yum log/transaction, man page not updated, ...)
- Traces of compilation: atime on files in /usr/include
- Impossible timestamp combinaisons:
- File/folder created after its last modification (crtime after ctime)
- File/folder created after the last modification of its parent folder (crtime of a file/folder after ctime of its parent folder)
Malware analysis
Malware analysis is complex and, with the ongoing competition between criminals and professional security researchers, is now mostly inaccessible without proper tool or training. If you have some doubts about a library or binary, you can take a quick look by:
- Running string -a and looking for any strange strings. Unfortunately most advanced malware will obfuscate their strings, making such analysis useless. However most legit binary and library will contain help description and text, which absence could be considered suspicious
- Run the malware in a controlled environment (e.g. isolated VM with no network), under strace and ltrace, to look for strange activity. If the binary is supposed to consume input (e.g. password for SSH/SSHD), feed it with some known input and check if that input has been compared with other strings
Resources
Here are some additional resources on forensics.
- Daniel Kouřil provides various training on forensics
- Slides for the forensics SSC 23 workshop: forensics.pptx and forensics-handson.pptx
- Thematic CERN School of Computing on Security 2022: Digital forensics: essentials and data acquisition
- Thematic CERN School of Computing on Security 2022: Digital forensics: data analysis
- Thematic CERN School of Computing on Security 2022: Digital forensics: exercises
- EGI Conference 2019: Intro to forensics
- Live Investigation (2019)
- EGI CSIRT trainings: https://csirt.egi.eu/trainings/
- GÉANT provides forensics trainings, video are available: https://security.geant.org/forensics-training/
Credits
This document, compiled by Vincent Brillault in 2017, is based on:
- Trainings written by Leif Nixon
- A forensics HowTo written by Heiko Reese