This document describes a best-effort approach for preserving and analyzing compromised Linux installations. In many situations, errors in forensics can lead to destruction of evidences, so please take precaution and contact experts if you are unsure. This document proposes commands to run to obtain or analyse some data, but the responsibility of running them is yours: if you do not know them, read the man-page and/or contact experts.

You will find useful references or explanation in the section Forensics resources resources at the bottom of the page.

Preparation before incidents

The difficulty of any forensic varies on the effort attackers took to hide themselves. However, preparing yourself and your servers can ease forensics by a large amount

Prepare yourself

Check your procedures

Some questions should be answered before an incident happens, to make sure they don't hinder you while working on forensics:

Who are you supposed to contact during incidents?
- Local security contacts?
- External CSIRT you depend on?
What is your policy with regard to law enforcement? Will you pursue legal action?
Do you legally have to report security breaches to your users?
Who can decide to take systems and services down?
Who will coordinate actions? (like: Who will answer any press contact?)

As a EGI resource centre the EGI CSIRT Security Incident Handling Procedure should be used to complement your local Incident Response Procedure.

Prepare some hardware

During forensics, you will most likely need:

A USB key containing:
- A Linux distribution of your choice, with auto-mount disabled
- A static copy of forensics tools. We recommend in particular:
  - Testdisk (For data recovery and carving)
  - The Sleuth Kit (For offline file-system analysis and timeline)
  - extundelete (For data recovery on ext3/ext4)
Few large capacity USB hard drive to collect disk images and evidences
[Optional] A SATA->USB converter to image disks offline (potentially with a write-blocking functionality) or a disk mirroring device.

Prepare your systems

Collect all system and audit logs on a remote system with strict access control.
Try to improve your file-system evidences:
- Disable 'prelink' (It alters binary metadata)
- Avoid cronjobs that read all files (Removes any meaning in the 'last access' metadata on file)
- Avoid mounting your system with 'noatime', use 'relatime' only if necessary
Make sure to have basic tools installed on your systems that will help you collecting forensics data (you can't install them once you think you have an incident):
- netstat
- lsof: list of open files
- pstree (from psmisc): tree of processes
- gcore (from gdb): generate core images of processes

Remarks on live analysis

Data collection order

Data should be collected in a specific order, to avoid missing or destroying evidences.

Observation changes the observed object

On a live system:

Every command you run will alter (at least) file system metadata (atime on that binary)
Every data you write on disk can override data from files which were deleted before

Forensic data is volatile

On a live system, most of the data is volatile and usually survive only for a short time:

Processor states, caches and RAM: Nano-seconds
Network states: Milli-seconds
Running processes: Seconds
Disks: minutes

(List borrowed from “Forensic Discovery”, Farmer & Venema, Addison-Wesley 2005)

Please note that except for the last one, all these data are lost when the system is rebooted!

Danger from live system

Compromised tools

When performing live analysis, one has to remember that the system might have been modified to lie to you:

The kernel might have been modified
Binaries might have modified
Libraries might have been modified or are injected

As a result, always doubt what you see and check if it makes sense:

Compare output of different tools (e.g. ps VS pstree VS ls /proc/pid)
Compare with external behaviour (e.g. network activity)
Compare with one own's knowledge (e.g. missing files/activity)

Are you alone?

If the system has been compromised, malicious actors might still be present. If you are unlucky, they might still be around, detect your actions and start deleting evidences!

As a result, as soon as the breach is confirmed and basic network/process forensics evidences have been collected, the system should be isolated from other systems.

Live (Quick & Dirty) forensics

When you are not sure you have a case and/or for physical systems (forensics on virtual machines should concentrate on snapshots (including memory snapshots)), here is a guide to collect initial data from the live system.

All these actions MUST be performed before switching off the system, as some of the evidences would be destroyed otherwise!

Before you start

Log every action you take, in order, with a time-stamp.
- Use paper and pen.
- Consider using script (check "man script") to log all your commands executed.
- If the case is complex, large and/or sensitive (e.g. suspicious of illegal data), consider imposing a 4 eyes policy
If you connect to the system remotely, e.g. ssh, avoid using credentials that can be reused on other systems and consider them lost: change them as soon a possible
Don't store data on the harddrive:
- Set HISTFILE to /dev/null
- Store temporary files in a filesystem backed by a tmpfs (in RAM), remotely or on a USB drive. For example:
```
# pick one in this list, we consider /dev/shm here 
mount | grep tmpfs
# verify that it is big enough
df -h /dev/shm 
mkdir /dev/shm/work && cd /dev/shm/work
```

Collect live data

In your temporary location, start by collecting live data from the system:

Network sockets

netstat -apn | tee netstat_apn.txt

Network environment

ip -4 neigh show | tee ip4_neigh_show.txt
ip -4 route show | tee ip4_route_list.txt
ip -4 link show | tee ip4_link_show.txt
ip -6 neigh show | tee ip6_neigh_show.txt
ip -6 route show | tee ip6_route_list.txt
ip -6 link show | tee ip6_link_show.txt

Users' logins

w > w.txt
last | tee last.txt
lastlog | tee lastlog.txt

Running processes

ps -auxwwwe | tee ps_auxwwwe.txt
pstree -lap | tee pstree_lap.txt

Open sockets/files

lsof -b -l -P -X -n -o -R -U | tee lsof_blPXnoRU.txt
lsof -b -l -P -X -n -o -R | tee lsof_blPXnoR.txt

List of mounted devices

cat /proc/mounts | tee proc_mounts.txt

Loaded kernel modules

cat /proc/modules | tee proc_modules
ls /sys/modules |tee sys_modules

Collect malicious process data

Review rapidly the data you have collected and look for oddities, for example:

Processes listening on a raw socket (not related to dhcp)
Processes listening on ports that you don't know about, can't find on the internet
Processes spawned by a process that should not do it usually
Processes with network activity that should not do it usually

For each of such process:

Keep the PID number in a separate variable and folder:

export PID=12345 # <- INSERT PROCESS-ID (PID) HERE
mkdir $PID
cd $PID

Stop the process (can't be caught):
```
kill -STOP ${PID}
```
Copy the executable:
```
cp /proc/${PID}/exe ${PID}.exe
```
Dump a core of the process:
```
gcore ${PID}
```
- If you don't have gcore, you need to do it manually:
```
gdb -p ${PID}
# Type "gcore"
# Type "detach"
# Type "exit"
```

Copy all interesting open files:

List files opened by the process:
```
lsof -np ${PID}
```

Copy 'deleted' files:

# FDNUM and FILENAME should be taken from the output of lsof
cp /proc/${PID}/fd/${FDNUM} ${FILENAME}

Copy files stored in /dev/shm:

# FILENAME should be taken from the output of lsof
cp /dev/shm/${FILENAME} ${FILENAME}

Copy all files from the proc folder:

tar cvf proc_${PID}.tar /proc/${PID}/{auxv,cgroup,cmdline,comm,environ,limits,maps,sched,schedstat,sessionid,smaps,stack,stat,statm,status,syscall,wchan}

Get out of your temporary folder
```
cd ..
```

Collect filesystem metadata

For each locally mounted filesystem, you should collect their file metadata (access, modification and change times).

The list of local filesystem can usually be found with:

grep '^/dev/' /proc/mounts

For each of them:

Create a mount point:
```
mkdir mountpoint
```
Bind mount the existing mounted point there, e.g. for '/':
```
mount --bind / mountpoint
```
Remount it read-only (this cannot be done in a single step!):
```
mount -o remount,ro mountpoint
```
Get into the mount point:
```
cd mountpoint
```

Get all metadata:

# replace 'root' with the actual filesystem you mounted
find . -print0 \
	| xargs -0 stat -c "%Y %X %Z %A %U %G %s %n" -- \
	| tee ../root.files

Get out of the mountpoint and unmount it:
```
cd ..
umount
mountpoint
```

Once you have collected all this metadata and extracted it to another location, the timeline of each filesystem can be rebuilt using this script:

Timeline rebuild

#! /usr/bin/python

from __future__ import print_function

from datetime import datetime
import sys
try:
  import pytz
except:
  pass

if len(sys.argv) > 1:
  try:
    timezone = pytz.timezone(sys.argv[1])
  except:
    print("Impossible to use this timezone")
    sys.exit(-1)
else:
  timezone = None

def print_line(flags, t, mode, user, group, size, name):
    when = datetime.utcfromtimestamp(float(t))
    if timezone is not None:
        try:
            when = pytz.utc.localize(when).astimezone(timezone)
        except:
            print("Timezone issue!")
            sys.exit(-1)
    print(' '.join([t, when.isoformat(), flags, mode, user, group, size, name]))

for line in sys.stdin:
    line = line[:-1]
    (m, a, c, mode, user, group, size, name) = line.split(" ", 7)
    if m == a:
        if m == c:
            print_line("mac", m, mode, user, group, size, name)
        else:
            print_line("ma-", m, mode, user, group, size, name)
            print_line("--c", c, mode, user, group, size, name)
    else:
        if m == c:
            print_line("m-c", m, mode, user, group, size, name)
            print_line("-a-", a, mode, user, group, size, name)
        else:
            print_line("m--", m, mode, user, group, size, name)
            print_line("-a-", a, mode, user, group, size, name)
            print_line("--c", c, mode, user, group, size, name)

[Optional] Automated tests

You can run some automated tools to try to identify malicious activity/files, but you first need:

To have these tools installed (e.g. chkrootkit, rkhunter, ossec-rootcheck)
To first remount all your local filesystems as read only. A first approach for this remount can use the following code, but you should rather review the list and remount them manually.
```
for mountpoint in $(grep '^/dev/' /proc/mounts | cut -d ' ' -f 2 | sort -r); do
	echo mount -o remount,ro $mountpoint
done
```

You can also (after remounting all filesystems read only) use your package management system to verify installed packages (save the output). On RedHat based systems, this is rpm -Va, on Debian based system, debsums.

Stopping the system

Before stopping the system, remember to take out all the data you have already collected using e.g. scp or rsync if you extracted it in a tmpfs. You might also want to run 'sync' to make sure that all data is written on disk.

There are two ways of stopping a system:

Run the normal shutdown procedure: this will close all running program properly, which would preserve e.g. databases, but would write a lot on the disk, potentially destroying evidences
Simply unplug the power cables of the system: this will kill all running programs, potentially damaging e.g. databases, but would preserve more evidences.

We usually recommend the later.

Offline analysis

Most of the analysis usually happens offline. Here we have less time pressure.

Copying disks

One of the most important rules of forensics is to never worker on the original supports, thus you need to copy the data (some will even recommend to only work on copies of copies in order to avoid copying the original again if you have a corruption).

There are mainly 3 ways of copying a disk :

Use a disk imaging/mirroring device
Use an external hard drive docking bay
Use an internal hot-swappable connector

For the last two possibility, one has to be very cautious not to modify any data:

The operating system on which the disk is plugged must have any auto-mount feature disabled
If more than one device is connected, they should be plugged in sequentially, to make sure that their identifiers are properly mapped
Any command typed can irremediably destroy data and should be checked multiple time

Once drives are identified, we recommend using dd to image the disk:

# inverting if and of will destroy all evidences!
dd if=/dev/sdX of=mybigfile.img bs=65536 conv=noerror,sync

Collect filesystem metadata

While filesystem metadata can be collected from live systems, collecting it from cold disk have some advantages:

While live system analysis can be affected by rootkit, this one cannot
On ext4 filesystem, normal live tools do not extract the creation time. Forensic tools do

We recommend using fls and mactime from the Sleuth Kit for collecting such metadata, see the corresponding wiki page: http://wiki.sleuthkit.org/index.php?title=Timeline

Carving unallocated blocks

Data on file-systems is not deleted when a file is deleted, but is kept on unallocated data blocks. These blocks might be overridden by the content of new files. Until then, that data can be recovered from the harddrive:

For a given pattern (e.g. sshd logs), it’s possible (and usually fruitful) to simply grep the raw disk
Dedicated tools, like PhotoRec from the testdisk utility, can try to rebuild files based on built-in patterns

Exploiting forensic data

There is one basic thing to remember when doing forensics: always try to assert how much you can trust what you see. Most of it is just bits on some disks, which could have been altered by an attacker, thus:

Never fully trust data from a compromised system, ask yourself if it could have been tempered with and what would that mean
Try to corroborate different sources, either between different local sources or with external sources
Check if what you are seeing is possible on a normal system, as part of the normal workflow

Reading file system timelines

When looking a a file system timeline, one should first remember that:

atime and mtime can easily be faked and are manipulated by the system itself (e.g. when extracting files from archives, downloading files, etc). You should start by ignoring them and only look at them when you understand the basic of the activity on the system
ctime are less likely to be manipulated
crtime and dtime are rarely manipulated

We recommend looking for the following artefacts:

Weird folder names (e.g. '...'), content in temporary and world-writable folders (e.g /var/tmp, /dev/shm, /tmp)
Binary or library ctime or crtime modified while the package was not updated (no yum log/transaction, man page not updated, ...)
Traces of compilation: atime on files in /usr/include
Impossible timestamp combinaisons:
- File/folder created after its last modification (crtime after ctime)
- File/folder created after the last modification of its parent folder (crtime of a file/folder after ctime of its parent folder)

Malware analysis

Malware analysis is complex and, with the ongoing competition between criminals and professional security researchers, is now mostly inaccessible without proper tool or training. If you have some doubts about a library or binary, you can take a quick look by:

Running string -a and looking for any strange strings. Unfortunately most advanced malware will obfuscate their strings, making such analysis useless. However most legit binary and library will contain help description and text, which absence could be considered suspicious
Run the malware in a controlled environment (e.g. isolated VM with no network), under strace and ltrace, to look for strange activity. If the binary is supposed to consume input (e.g. password for SSH/SSHD), feed it with some known input and check if that input has been compared with other strings

Resources

Here are some additional resources on forensics.

Daniel Kouřil provides various training on forensics
EGI CSIRT trainings: https://csirt.egi.eu/trainings/
GÉANT provides forensics trainings, video are available: https://security.geant.org/forensics-training/

Credits

This document, compiled by Vincent Brillault in 2017, is based on:

Trainings written by Leif Nixon
A forensics HowTo written by Heiko Reese

Page tree

Forensics Howto

Preparation before incidents

Prepare yourself

Check your procedures

Prepare some hardware

Prepare your systems

Remarks on live analysis

Data collection order

Observation changes the observed object

Forensic data is volatile

Danger from live system

Compromised tools

Are you alone?

Live (Quick & Dirty) forensics

Before you start

Collect live data

Collect malicious process data

Collect filesystem metadata

[Optional] Automated tests

Stopping the system

Offline analysis

Copying disks

Collect filesystem metadata

Carving unallocated blocks

Exploiting forensic data

Reading file system timelines

Malware analysis

Resources

Credits