Sunday, February 21, 2021

Building a Linux profile for Volatility 2 and 3

I needed to analyze a memory dump with Volatility 2/3. These are the steps I followed to build a Linux profile based on Red Hat Enterprise Linux (RHEL) 5.11 x64 (kernel version 2.6.18-398). At the time of writing, the Volatility repository doesn't have a profile for this OS (1)

I installed RHEL511x64 on VMware Workstation v14 using default settings during setup (always press Next). No updates once installed the OS. I then installed VMware Tools for convenience so that I could copy and paste between the host OS and this VM.

From the mounted ISO, I installed these packages that would be needed later (2):

# cd /media/RHEL_5.11\ x86_64\ DVD/Server/
# rpm -i kernel-headers-$(uname -r).x86_64.rpm && rpm -i kernel-debug-$(uname -r).x86_64.rpm && rpm -i kernel-debug-devel-$(uname -r).x86_64.rpm && rpm -i kernel-devel-$(uname -r).x86_64.rpm && rpm -i glibc-headers-*.x86_64.rpm && rpm -i glibc-devel-*.x86_64.rpm && rpm -i gcc-[0-9.-]*.el5.x86_64.rpm && rpm -i elfutils-libelf-devel-*.x86_64.rpm && rpm -i libstdc++-devel-*.x86_64.rpm && rpm -i gcc-c++-*.x86_64.rpm

From this repository:
I downloaded the appropriate kernel-debuginfo packages (3):
  • kernel-debuginfo-common-2.6.18-398.el5.x86_64.rpm
  • kernel-debug-debuginfo-2.6.18-398.el5.x86_64.rpm
  • kernel-debuginfo-2.6.18-398.el5.x86_64.rpm
which I pasted for convenience in the \tmp path.

# cd /tmp
# rpm -i kernel-debuginfo-common-$(uname -r).x86_64.rpm && rpm -i kernel-debug-debuginfo-$(uname -r).x86_64.rpm && rpm -i kernel-debuginfo-$(uname -r).x86_64.rpm

Next step was to download libdwarf/dwarfdump from https://www.prevanders.net/dwarf.html. After trying to compile different versions, I used libdwarf-20180129.tar.gz.

# cd /tmp
# tar -zxvf libdwarf-20180129.tar.gz
# cd dwarf-20180129/
# ./configure
# make
# cp dwarfdump/dwarfdump /usr/local/sbin

From the master branch on GitHub, I pulled the latest version available of Volatility 2.

# cd /tmp
# unzip volatility-master.zip
# mv volatility-master/ volatility/

But make failed with this error when I tried to generate the dwarf module (4).

# cd volatility/tools/linux/
# make

make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CC [M]  /tmp/volatility/tools/linux/module.o
/tmp/volatility/tools/linux/module.c:218: error: redefinition of ‘struct module_sect_attr’
/tmp/volatility/tools/linux/module.c:225: error: redefinition of ‘struct module_sect_attrs’
/tmp/volatility/tools/linux/module.c:379:5: warning: "STATS" is not defined
/tmp/volatility/tools/linux/module.c:395:5: warning: "DEBUG" is not defined
make[2]: *** [/tmp/volatility/tools/linux/module.o] Error 1
make[1]: *** [_module_/tmp/volatility/tools/linux] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
make: *** [dwarf] Error 2

The workaround sed -i 's/PWD/shell pwd/g' Makefile  was of no help (5).

I looked at the source code of module.c (volatility/tools/linux/) and found a section starting with #if LINUX_VERSION_CODE == KERNEL_VERSION(2,6,18). As an attempt to bypass the issue, I deleted the whole section, specifically from line 212 to 239 (6).


I tried make again and this time it worked!

make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CC [M]  /tmp/volatility/tools/linux/module.o
/tmp/volatility/tools/linux/module.c:350:5: warning: "STATS" is not defined
/tmp/volatility/tools/linux/module.c:366:5: warning: "DEBUG" is not defined
  Building modules, stage 2.
  MODPOST
  CC      /tmp/volatility/tools/linux/module.mod.o
  LD [M]  /tmp/volatility/tools/linux/module.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
dwarfdump -di module.ko > module.dwarf
make -C //lib/modules/2.6.18-398.el5/build M="/tmp/volatility/tools/linux" clean
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CLEAN   /tmp/volatility/tools/linux/.tmp_versions
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'

And the last step for Volatility 2 was to create the zip archive for the profile (7).

# cd /tmp
# zip RedHat511.zip volatility/tools/linux/module.dwarf /boot/System.map-$(uname -r)

Regarding Volatility 3, I made a copy of these two files:

# cd /tmp
# cp /boot/System.map-$(uname -r) .
# cp /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmlinuz-$(uname -r)

I suspended the VM and ran vmss2core (included in the installation of VMware Workstation) to create a vmss.core file from the .vmem and .vmss files listed in the picture below. (The presence or not of both files depends on the hypervisor you're using and the conversion to vmss.core is not always needed).

.vmem and .vmss

This is a general description of vmss2core.

"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe"
vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
Missing .vmss filename

   A tool to convert VMware checkpoint state files into formats
   that third party debugger tools understand. It can handle both
   suspend (.vmss) and snapshot (.vmsn) checkpoint state files
   (hereafter referred to as a 'vmss file') as well as both
   monolithic and non-monolithic (separate .vmem file) encapsulation
   of checkpoint state data.

   Usage:

      GENERAL:  vmss2core [[options] | [-l linuxoffsets options]] \
                  <vmss file> [<vmem file>]
[...]
   output can be produced with these options:

[...]
   -N6     Red Hat crash core file for Linux 2.6 (vmss.core).

This is how I ran the tool (8).

C:>"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe" -N6 RHEL511x64-328dacd0.vmss RHEL511x64-328dacd0.vmem
vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
Started core writing.
Writing note section header.
Writing 1 memory section headers.
Writing notes.
... 10 MBs written.
... 20 MBs written.
... 30 MBs written.
...
... 1020 MBs written.
Finished writing core.

The output is a vmss.core file. For its analysis with Volatility 2/3, I used two different Ubuntu VMs to avoid any problem with different versions of Python, virtualenv, dependencies of other tools and so no.


Volatility 2.6.1 (tested with: Ubuntu 16.04.7 LTS, Python 2.7.12) 

These are the steps I followed to install Volatility 2.6.1:

git clone https://github.com/volatilityfoundation/volatility
sudo apt-get install subversion pcregrep libpcre++-dev python-dev yara -y
sudo apt-get install python-pip
pip install PyCrypto distorm3 yara-python
cd volatility
python setup.py build
sudo python setup.py build install
python vol.py -h

I copied the zipped profile for Volatility 2 to volatility/volatility/plugins/overlays/linux and ran python vol.py --plugins=list --info to check if the profile was detected.

Volatility Foundation Volatility Framework 2.6.1

Profiles
--------
LinuxRedHat511x64     - A Profile for Linux RedHat511 x64

These are some of the plugins I tested using the newly created profile:

python vol.py -f ~/Desktop/vmss.core --profile=LinuxRedHat511x64 <linux plugin>

linux_pslist

linux_bash

linux_netstat



Volatility 3 (tested with: Ubuntu 20.04, Python 3.8.5)

Volatility 3 installation (9)

sudo apt install python3-pip
pip3 install leechcorepyc PyCrypto yara-python
git clone https://github.com/volatilityfoundation/volatility3
cd volatility3
python3 setup.py build
sudo python3 setup.py build install
python3 vol.py -h

dwarf2json (10)

sudo apt install golang-go
git clone https://github.com/volatilityfoundation/dwarf2json
cd dwarf2json/
go build

I ran dwarf2json against vmlinux and System.map that I had exported before (11):

./dwarf2json linux --elf vmlinuz-2.6.18-398.el5 --system-map System.map-2.6.18-398.el5 | xz -c > RHEL511x64_2.6.18-398.json.xz

Then I:
  • went to ~/volatility3/volatility3/symbols
  • created a subdirectory named linux
  • copied in here the .json.xz archive created with dwarf2json
I ran Volatility 3 (with the verbosity option) against the vmss.core file to check if the Linux banner was successfully detected.

python3 vol.py -vvvv -f ~/Desktop/vmss.core linux.pslist.PsList
Volatility 3 Framework 1.0.1
INFO     root        : Volatility plugins path: ['/home/user/volatility3/volatility3/plugins', '/home/user/volatility3/volatility3/framework/plugins']
INFO     root        : Volatility symbols path: ['/home/user/volatility3/volatility3/symbols', '/home/user/volatility3/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a linux category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
[...]
DEBUG    volatility3.framework.automagic.linux: Identified banner: b'Linux version 2.6.18-398.el5 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-55)) #1 SMP Tue Aug 12 06:26:17 EDT 2014\n\x00'

The banner appears if it matches the banner stored inside the memory dump.

strings and Linux banner

And this was the ouput of the linux.pstree.PsTree plugin.

python3 vol.py -f ~/Desktop/vmss.core linux.pstree.PsTree


Download
I put the two profiles on my GitHub: https://github.com/forensenellanebbia/volatility-profiles


References
  1. https://github.com/volatilityfoundation/profiles/
  2. http://forensicinsight.org/wp-content/uploads/2013/07/F-INSIGHT-Generating-Volatility-Linux-Profile.pdf
  3. Linux plugins issues: https://github.com/volatilityfoundation/volatility3/issues/231
  4. compilation error module.c against CentOS 5.11 kernel 2.6.18-398.el5:https://github.com/volatilityfoundation/volatility/issues/111
  5. fail to create linux profile: https://github.com/volatilityfoundation/volatility/issues/373
  6. https://vdchuyen.com/blog/2016/01/01/build-volatility-centos-profile.html
  7. https://www.andreafortuna.org/2019/08/22/how-to-generate-a-volatility-profile-for-a-linux-system/
  8. Converting a snapshot file to memory dump using the vmss2core tool: https://kb.vmware.com/s/article/2003941
  9. https://github.com/volatilityfoundation/volatility3
  10. https://github.com/volatilityfoundation/dwarf2json
  11. https://superuser.com/questions/1620234/volatility3-crashes-on-kali/1620643