I needed to analyze a memory dump with Volatility 2/3. These are the steps I followed to build a Linux profile based on Red Hat Enterprise Linux (RHEL) 5.11 x64 (kernel version 2.6.18-398). At the time of writing, the Volatility repository doesn't have a profile for this OS (1).
I installed RHEL511x64 on VMware Workstation v14 using default settings during setup (always press Next). No updates once installed the OS.
Note: if you're following this procedure and you need to upgrade the kernel to a more recent version, first download these packages from https://access.redhat.com/downloads/content/package-browser :
- kernel-<KernelVersion>.x86_64.rpm
- kernel-headers-<KernelVersion>.x86_64.rpm
- kernel-debug-<KernelVersion>.x86_64.rpm
- kernel-debug-devel-<KernelVersion>.x86_64.rpm
- kernel-devel-<KernelVersion>.x86_64.rpm
Install them as root, reboot, on the boot menu choose the new kernel to load and then proceed to install the packages listed right below (skip the kernel* ones).
Since the VM was already running the kernel version I needed, from the mounted ISO I installed these packages that would be needed later (2):
cd /media/RHEL_5.11\ x86_64\ DVD/Server/
rpm -iv --force kernel-headers-$(uname -r)*.rpm kernel-debug-$(uname -r)*.rpm kernel-debug-devel-$(uname -r)*.rpm kernel-devel-$(uname -r)*.rpm glibc-headers-*.x86_64.rpm glibc-devel-*.x86_64.rpm cpp-*.x86_64.rpm gcc-[0-9.-]*.x86_64.rpm zlib-devel-*.x86_64.rpm elfutils-libelf-devel-*.x86_64.rpm libstdc++-devel-*.x86_64.rpm gcc-c++-*.x86_64.rpm
I then installed the VMware Tools for convenience so that I could copy and paste between the host OS and this VM.
From this repository:
I downloaded the appropriate kernel-debuginfo packages (3):
- kernel-debuginfo-common-2.6.18-398.el5.x86_64.rpm
- kernel-debug-debuginfo-2.6.18-398.el5.x86_64.rpm
- kernel-debuginfo-2.6.18-398.el5.x86_64.rpm
which I pasted for convenience in the /tmp path.
cd /tmp
rpm -iv kernel-debuginfo-common*-$(uname -r)*.rpm kernel-debug-debuginfo-$(uname -r)*.rpm kernel-debuginfo-$(uname -r)*.rpm
Next step was to download libdwarf/dwarfdump from https://www.prevanders.net/dwarf.html#releases. After trying to compile different versions, I eventually succeeded with libdwarf-20180129.tar.gz.
cd /tmp
tar -zxvf libdwarf-20180129.tar.gz
cd dwarf-20180129/
./configure
make
cp dwarfdump/dwarfdump /usr/local/sbin/
From the master branch on GitHub, I pulled the latest version available of Volatility 2.
cd /tmp
unzip volatility-master.zip
mv volatility-master/ volatility/
But make failed with this error when I tried to generate the dwarf module (4).
# cd volatility/tools/linux/ # make
make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
CC [M] /tmp/volatility/tools/linux/module.o
/tmp/volatility/tools/linux/module.c:218: error: redefinition of ‘struct module_sect_attr’
/tmp/volatility/tools/linux/module.c:225: error: redefinition of ‘struct module_sect_attrs’
/tmp/volatility/tools/linux/module.c:379:5: warning: "STATS" is not defined
/tmp/volatility/tools/linux/module.c:395:5: warning: "DEBUG" is not defined
make[2]: *** [/tmp/volatility/tools/linux/module.o] Error 1
make[1]: *** [_module_/tmp/volatility/tools/linux] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
make: *** [dwarf] Error 2
The workaround sed -i 's/PWD/shell pwd/g' Makefile was of no help (5).
I looked at the source code of module.c (volatility/tools/linux/) and found a section starting with #if LINUX_VERSION_CODE == KERNEL_VERSION(2,6,18). As an attempt to bypass the issue, I deleted the whole section, specifically from line 212 to 239 (6).
I tried make again and this time it worked!
make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64' CC [M] /tmp/volatility/tools/linux/module.o /tmp/volatility/tools/linux/module.c:350:5: warning: "STATS" is not defined /tmp/volatility/tools/linux/module.c:366:5: warning: "DEBUG" is not defined Building modules, stage 2. MODPOST CC /tmp/volatility/tools/linux/module.mod.o LD [M] /tmp/volatility/tools/linux/module.ko make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64' dwarfdump -di module.ko > module.dwarf make -C //lib/modules/2.6.18-398.el5/build M="/tmp/volatility/tools/linux" clean make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64' CLEAN /tmp/volatility/tools/linux/.tmp_versions make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
And the last step for Volatility 2 was to create the zip archive for the profile (7).
cd /tmp
zip RedHat511.zip volatility/tools/linux/module.dwarf /boot/System.map-$(uname -r)
Regarding Volatility 3, I made a copy of these two files:
cd /tmp
cp /boot/System.map-$(uname -r) .
cp /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmlinuz-$(uname -r)
I suspended the VM and ran vmss2core (included in the installation of VMware Workstation or downloadable from vmware Flings) on my host to create a vmss.core file from the .vmem and .vmss files listed in the picture below. (The presence or not of both files depends on the hypervisor you're using and the conversion to vmss.core is not always needed).
 |
| .vmem and .vmss |
This is a general description of vmss2core.
"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe" vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved. Missing .vmss filename A tool to convert VMware checkpoint state files into formats that third party debugger tools understand. It can handle both suspend (.vmss) and snapshot (.vmsn) checkpoint state files (hereafter referred to as a 'vmss file') as well as both monolithic and non-monolithic (separate .vmem file) encapsulation of checkpoint state data. Usage: GENERAL: vmss2core [[options] | [-l linuxoffsets options]] \ <vmss file> [<vmem file>] [...] output can be produced with these options: [...] -N6 Red Hat crash core file for Linux 2.6 (vmss.core).
This is how I ran the tool (8).
C:>"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe" -N6 RHEL511x64-328dacd0.vmss RHEL511x64-328dacd0.vmem
vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
Started core writing.
Writing note section header.
Writing 1 memory section headers.
Writing notes.
... 10 MBs written.
... 20 MBs written.
... 30 MBs written.
...
... 1020 MBs written.
Finished writing core.
The output is a vmss.core file. For its analysis with Volatility 2/3, I used two different Ubuntu VMs to avoid any problem with different versions of Python, virtualenv, dependencies of other tools and so no.
Volatility 2.6.1 (tested with: Ubuntu 16.04.7 LTS, Python 2.7.12)
These are the steps I followed to install Volatility 2.6.1:
git clone https://github.com/volatilityfoundation/volatility
sudo apt-get install libpcre++-dev pcregrep python-dev python-pip subversion yara
pip install distorm3 PyCrypto yara-python
cd volatility
python setup.py build
sudo python setup.py build install
python vol.py -h
I copied the zipped profile for Volatility 2 to volatility/volatility/plugins/overlays/linux and ran python vol.py --plugins=list --info to check if the profile was detected.
Volatility Foundation Volatility Framework 2.6.1 Profiles -------- LinuxRedHat511x64 - A Profile for Linux RedHat511 x64
These are some of the plugins I tested using the newly created profile:
python vol.py -f ~/Desktop/vmss.core --profile=LinuxRedHat511x64 <linux plugin>
 |
| linux_pslist |
 |
| linux_bash |
 |
| linux_netstat |
Volatility 3 v2.3.0 (tested with: Ubuntu 20.04.4, Python 3.8.10)
Volatility 3 installation (9)
sudo apt install git python3-pip
sudo pip3 install capstone jsonschema leechcorepyc pefile pycryptodome python-snappy==0.6.0 yara-python
git clone https://github.com/volatilityfoundation/volatility3
cd volatility3
python3 setup.py build
sudo python3 setup.py build install
python3 vol.py -h
dwarf2json (10)
sudo snap install go --classic
git clone https://github.com/volatilityfoundation/dwarf2json
cd dwarf2json/
go build
./dwarf2json linux
To create a "profile" for Volatility 3 (11), I ran dwarf2json against the files vmlinux and System.map that I had exported before from the VM (12):
./dwarf2json linux --elf vmlinuz-2.6.18-398.el5 --system-map System.map-2.6.18-398.el5 | xz -c > RHEL511x64_2.6.18-398.json.xz
If the dwarf2json step fails, try to increase the amount of memory assigned to the VM (8GB+).
Then I:
- went to ~/volatility3/volatility3/symbols
- created a subdirectory named linux
- copied in here the .json.xz archive created with dwarf2json
I ran Volatility 3 (with the verbosity option) against the vmss.core file to check if the Linux banner was successfully detected.
python3 vol.py -vvvv -f ~/Desktop/vmss.core linux.pslist.PsList Volatility 3 Framework 1.0.1 INFO root : Volatility plugins path: ['/home/user/volatility3/volatility3/plugins', '/home/user/volatility3/volatility3/framework/plugins'] INFO root : Volatility symbols path: ['/home/user/volatility3/volatility3/symbols', '/home/user/volatility3/volatility3/framework/symbols'] INFO volatility3.framework.automagic: Detected a linux category plugin INFO volatility3.framework.automagic: Running automagic: ConstructionMagic [...] DEBUG volatility3.framework.automagic.linux: Identified banner: b'Linux version 2.6.18-398.el5 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-55)) #1 SMP Tue Aug 12 06:26:17 EDT 2014\n\x00'
The banner appears if it matches the banner stored inside the memory dump.
 |
| strings and Linux banner |
And this was the ouput of the linux.pstree.PsTree plugin.
 |
| python3 vol.py -f ~/Desktop/vmss.core linux.pstree.PsTree |
Download
I put the two profiles on my GitHub: https://github.com/forensenellanebbia/volatility-profiles
References
- https://github.com/volatilityfoundation/profiles/
- http://forensicinsight.org/wp-content/uploads/2013/07/F-INSIGHT-Generating-Volatility-Linux-Profile.pdf
- Linux plugins issues: https://github.com/volatilityfoundation/volatility3/issues/231
- compilation error module.c against CentOS 5.11 kernel 2.6.18-398.el5:https://github.com/volatilityfoundation/volatility/issues/111
- fail to create linux profile: https://github.com/volatilityfoundation/volatility/issues/373
https://vdchuyen.com/blog/2016/01/01/build-volatility-centos-profile.html- https://www.andreafortuna.org/2019/08/22/how-to-generate-a-volatility-profile-for-a-linux-system/
- Converting a snapshot file to memory dump using the vmss2core tool: https://kb.vmware.com/s/article/2003941
- https://github.com/volatilityfoundation/volatility3
- https://github.com/volatilityfoundation/dwarf2json
- https://volatility3.readthedocs.io/en/latest/vol2to3.html#symbols-and-types
- https://superuser.com/questions/1620234/volatility3-crashes-on-kali/1620643
