Building a Linux profile for Volatility 2 and 3

I needed to analyze a memory dump with Volatility 2/3. These are the steps I followed to build a Linux profile based on Red Hat Enterprise Linux (RHEL) 5.11 x64 (kernel version 2.6.18-398). At the time of writing, the Volatility repository doesn't have a profile for this OS (1). 

I installed RHEL511x64 on VMware Workstation v14 using default settings during setup (always press Next). No updates once installed the OS. 

Note: if you're following this procedure and you need to upgrade the kernel to a more recent version, first download these packages from https://access.redhat.com/downloads/content/package-browser :

• kernel-<KernelVersion>.x86_64.rpm
• kernel-headers-<KernelVersion>.x86_64.rpm
• kernel-debug-<KernelVersion>.x86_64.rpm
• kernel-debug-devel-<KernelVersion>.x86_64.rpm
• kernel-devel-<KernelVersion>.x86_64.rpm

Install them as root, reboot, on the boot menu choose the new kernel to load and then proceed to install the packages listed right below (skip the kernel* ones).

Since the VM was already running the kernel version I needed, from the mounted ISO I installed these packages that would be needed later (2):

cd /media/RHEL_5.11\ x86_64\ DVD/Server/
rpm -iv --force kernel-headers-$(uname -r)*.rpm kernel-debug-$(uname -r)*.rpm kernel-debug-devel-$(uname -r)*.rpm kernel-devel-$(uname -r)*.rpm glibc-headers-*.x86_64.rpm glibc-devel-*.x86_64.rpm cpp-*.x86_64.rpm gcc-[0-9.-]*.x86_64.rpm zlib-devel-*.x86_64.rpm elfutils-libelf-devel-*.x86_64.rpm libstdc++-devel-*.x86_64.rpm gcc-c++-*.x86_64.rpm

I then installed the VMware Tools for convenience so that I could copy and paste between the host OS and this VM.

From this repository:

• http://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/x86_64/Debuginfo/

I downloaded the appropriate kernel-debuginfo packages (3):

• kernel-debuginfo-common-2.6.18-398.el5.x86_64.rpm
• kernel-debug-debuginfo-2.6.18-398.el5.x86_64.rpm
• kernel-debuginfo-2.6.18-398.el5.x86_64.rpm

which I pasted for convenience in the /tmp path.

cd /tmp
rpm -iv kernel-debuginfo-common*-$(uname -r)*.rpm kernel-debug-debuginfo-$(uname -r)*.rpm kernel-debuginfo-$(uname -r)*.rpm

Next step was to download libdwarf/dwarfdump from https://www.prevanders.net/dwarf.html#releases. After trying to compile different versions, I eventually succeeded with libdwarf-20180129.tar.gz.

cd /tmp
tar -zxvf libdwarf-20180129.tar.gz
cd dwarf-20180129/
./configure
make
cp dwarfdump/dwarfdump /usr/local/sbin/

From the master branch on GitHub, I pulled the latest version available of Volatility 2.

cd /tmp
unzip volatility-master.zip
mv volatility-master/ volatility/

But make failed with this error when I tried to generate the dwarf module (4).

# cd volatility/tools/linux/
# make

make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CC [M]  /tmp/volatility/tools/linux/module.o
/tmp/volatility/tools/linux/module.c:218: error: redefinition of ‘struct module_sect_attr’
/tmp/volatility/tools/linux/module.c:225: error: redefinition of ‘struct module_sect_attrs’
/tmp/volatility/tools/linux/module.c:379:5: warning: "STATS" is not defined
/tmp/volatility/tools/linux/module.c:395:5: warning: "DEBUG" is not defined
make[2]: *** [/tmp/volatility/tools/linux/module.o] Error 1
make[1]: *** [_module_/tmp/volatility/tools/linux] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
make: *** [dwarf] Error 2

The workaround sed -i 's/PWD/shell pwd/g' Makefile  was of no help (5).

I looked at the source code of module.c (volatility/tools/linux/) and found a section starting with #if LINUX_VERSION_CODE == KERNEL_VERSION(2,6,18). As an attempt to bypass the issue, I deleted the whole section, specifically from line 212 to 239 (6).

I tried make again and this time it worked!

make -C //lib/modules/2.6.18-398.el5/build CONFIG_DEBUG_INFO=y M="/tmp/volatility/tools/linux" modules
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CC [M]  /tmp/volatility/tools/linux/module.o
/tmp/volatility/tools/linux/module.c:350:5: warning: "STATS" is not defined
/tmp/volatility/tools/linux/module.c:366:5: warning: "DEBUG" is not defined
  Building modules, stage 2.
  MODPOST
  CC      /tmp/volatility/tools/linux/module.mod.o
  LD [M]  /tmp/volatility/tools/linux/module.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
dwarfdump -di module.ko > module.dwarf
make -C //lib/modules/2.6.18-398.el5/build M="/tmp/volatility/tools/linux" clean
make[1]: Entering directory `/usr/src/kernels/2.6.18-398.el5-x86_64'
  CLEAN   /tmp/volatility/tools/linux/.tmp_versions
make[1]: Leaving directory `/usr/src/kernels/2.6.18-398.el5-x86_64'

And the last step for Volatility 2 was to create the zip archive for the profile (7).

cd /tmp
zip RedHat511.zip volatility/tools/linux/module.dwarf /boot/System.map-$(uname -r)

Regarding Volatility 3, I made a copy of these two files:

cd /tmp
cp /boot/System.map-$(uname -r) .
cp /usr/lib/debug/lib/modules/$(uname -r)/vmlinux vmlinuz-$(uname -r)

I suspended the VM and ran vmss2core (included in the installation of VMware Workstation or downloadable from vmware Flings) on my host to create a vmss.core file from the .vmem and .vmss files listed in the picture below. (The presence or not of both files depends on the hypervisor you're using and the conversion to vmss.core is not always needed).

.vmem and .vmss

This is a general description of vmss2core.

"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe"
vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
Missing .vmss filename

   A tool to convert VMware checkpoint state files into formats
   that third party debugger tools understand. It can handle both
   suspend (.vmss) and snapshot (.vmsn) checkpoint state files
   (hereafter referred to as a 'vmss file') as well as both
   monolithic and non-monolithic (separate .vmem file) encapsulation
   of checkpoint state data.

   Usage:

      GENERAL:  vmss2core [[options] | [-l linuxoffsets options]] \
                  <vmss file> [<vmem file>]
[...]
   output can be produced with these options:

[...]
   -N6     Red Hat crash core file for Linux 2.6 (vmss.core).

This is how I ran the tool (8).

C:>"C:\Program Files (x86)\VMware\VMware Workstation\vmss2core.exe" -N6 RHEL511x64-328dacd0.vmss RHEL511x64-328dacd0.vmem
vmss2core version 14921873 Copyright (C) 1998-2019 VMware, Inc. All rights reserved.
Started core writing.
Writing note section header.
Writing 1 memory section headers.
Writing notes.
... 10 MBs written.
... 20 MBs written.
... 30 MBs written.
...
... 1020 MBs written.
Finished writing core.

The output is a vmss.core file. For its analysis with Volatility 2/3, I used two different Ubuntu VMs to avoid any problem with different versions of Python, virtualenv, dependencies of other tools and so no.

Volatility 2.6.1 (tested with: Ubuntu 16.04.7 LTS, Python 2.7.12) 

These are the steps I followed to install Volatility 2.6.1:

git clone https://github.com/volatilityfoundation/volatility
sudo apt-get install libpcre++-dev pcregrep python-dev python-pip subversion yara
pip install distorm3 PyCrypto yara-python
cd volatility
python setup.py build
sudo python setup.py build install
python vol.py -h

I copied the zipped profile for Volatility 2 to volatility/volatility/plugins/overlays/linux and ran python vol.py --plugins=list --info to check if the profile was detected.

Volatility Foundation Volatility Framework 2.6.1

Profiles
--------
LinuxRedHat511x64     - A Profile for Linux RedHat511 x64

These are some of the plugins I tested using the newly created profile:

python vol.py -f ~/Desktop/vmss.core --profile=LinuxRedHat511x64 <linux plugin>

linux_pslist

linux_bash

linux_netstat

Volatility 3 v2.3.0 (tested with: Ubuntu 20.04.4, Python 3.8.10)

Volatility 3 installation (9)

sudo apt install git python3-pip
sudo pip3 install capstone jsonschema leechcorepyc pefile pycryptodome python-snappy==0.6.0 yara-python
git clone https://github.com/volatilityfoundation/volatility3
cd volatility3
python3 setup.py build
sudo python3 setup.py build install
python3 vol.py -h

dwarf2json (10)

sudo snap install go --classic
git clone https://github.com/volatilityfoundation/dwarf2json
cd dwarf2json/
go build
./dwarf2json linux

To create a "profile" for Volatility 3 (11), I ran dwarf2json against the files vmlinux and System.map that I had exported before from the VM (12):

./dwarf2json linux --elf vmlinuz-2.6.18-398.el5 --system-map System.map-2.6.18-398.el5 | xz -c > RHEL511x64_2.6.18-398.json.xz

If the dwarf2json step fails, try to increase the amount of memory assigned to the VM (8GB+).

Then I:

• went to ~/volatility3/volatility3/symbols
• created a subdirectory named linux
• copied in here the .json.xz archive created with dwarf2json

I ran Volatility 3 (with the verbosity option) against the vmss.core file to check if the Linux banner was successfully detected.

python3 vol.py -vvvv -f ~/Desktop/vmss.core linux.pslist.PsList
Volatility 3 Framework 1.0.1
INFO     root        : Volatility plugins path: ['/home/user/volatility3/volatility3/plugins', '/home/user/volatility3/volatility3/framework/plugins']
INFO     root        : Volatility symbols path: ['/home/user/volatility3/volatility3/symbols', '/home/user/volatility3/volatility3/framework/symbols']
INFO     volatility3.framework.automagic: Detected a linux category plugin
INFO     volatility3.framework.automagic: Running automagic: ConstructionMagic
[...]
DEBUG    volatility3.framework.automagic.linux: Identified banner: b'Linux version 2.6.18-398.el5 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-55)) #1 SMP Tue Aug 12 06:26:17 EDT 2014\n\x00'

The banner appears if it matches the banner stored inside the memory dump.

strings and Linux banner

And this was the ouput of the linux.pstree.PsTree plugin.

python3 vol.py -f ~/Desktop/vmss.core linux.pstree.PsTree

Download

I put the two profiles on my GitHub: https://github.com/forensenellanebbia/volatility-profiles

References

1. https://github.com/volatilityfoundation/profiles/
2. http://forensicinsight.org/wp-content/uploads/2013/07/F-INSIGHT-Generating-Volatility-Linux-Profile.pdf
3. Linux plugins issues: https://github.com/volatilityfoundation/volatility3/issues/231
4. compilation error module.c against CentOS 5.11 kernel 2.6.18-398.el5:https://github.com/volatilityfoundation/volatility/issues/111
5. fail to create linux profile: https://github.com/volatilityfoundation/volatility/issues/373
6. https://vdchuyen.com/blog/2016/01/01/build-volatility-centos-profile.html
7. https://www.andreafortuna.org/2019/08/22/how-to-generate-a-volatility-profile-for-a-linux-system/
8. Converting a snapshot file to memory dump using the vmss2core tool: https://kb.vmware.com/s/article/2003941
9. https://github.com/volatilityfoundation/volatility3
10. https://github.com/volatilityfoundation/dwarf2json
11. https://volatility3.readthedocs.io/en/latest/vol2to3.html#symbols-and-types
12. https://superuser.com/questions/1620234/volatility3-crashes-on-kali/1620643

Let's combine EvtxEcmd with LogonTracer

This blog post aims to show how to combine EvtxECmd (v0.6.0.3) with LogonTracer (v1.5.0) during the analysis of Windows Event Log events.

To have some data to work on, download for instance the Windows EVTX Samples shared by Samir @SBousseaden on GitHub.

Extract the EVTX files to a temporary location like C:\TEMP\EVTX-ATTACK-SAMPLES-master.

Use EvtxECmd to extract only the events that are currently supported by LogonTracer: this is especially useful when dealing with cases that contain GBs of EVTX files from different systems and you want to load into LogonTracer just the event IDs that this tool can interpret, speeding up the whole process.

EvtxECmd.exe -d <PATH> --xml <PATH> --xmlf evtxecmd.xml --inc 4624,4625,4768,4769,4776,4672

Now we need to install LogonTracer. For convenience, I'll install the docker version of the tool on a REMnux v7 VM which runs Ubuntu 20.04. You may need to increase the amount of memory assigned to the VM (at least 4 GB).

sudo apt-get update && sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent software-properties-common && curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && sudo apt-key fingerprint 0EBFCD88 && sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && sudo apt-get update && sudo apt-get install docker-ce docker-ce-cli containerd.io && sudo docker pull jpcertcc/docker-logontracer && sudo usermod -aG docker remnux

From the terminal in REMnux, launch LogonTracer:

docker run --detach --publish=7474:7474 --publish=7687:7687 --publish=8080:8080 -e LTHOSTNAME=127.0.0.1 jpcertcc/docker-logontracer

Open the browser on REMnux and go to: http://127.0.0.1:8080/

Scroll down at the very bottom, till you see on the left the button "Upload Event Log".

Click on it and: 

• select "XML" in the next window that appears
• set the time zone
• choose the evtxecmd.xml file that we created before. 

Once you click "Upload", the parsing fails due to an error saying "This file is not XML format".

The XML structure of evtxecmd.xml has to be tweaked a little in order to work with LogonTracer. After comparing it with an XML file created using Microsoft Event Viewer, and based on my tests, in order to make the file work you need to:

• Add the following header: <?xml version="1.0" encoding="utf-8" standalone="yes"?><Events>
• Add this footer: </Events>
• Replace all the <Event> tags with <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
• Merge all lines into one
• Remove all the spaces between ">" and "<"

We can use PowerShell to automate the above steps. 

$evtxecmd = "C:\TEMP\EVTX-ATTACK-SAMPLES-master\evtxecmd.xml";$logontracer = '<?xml version="1.0" encoding="utf-8" standalone="yes"?><Events>' + [System.IO.File]::ReadLines($evtxecmd) + "</Events>";$logontracer = ($logontracer -join ("")) -replace "<Event>","<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" -replace ">[ ]*<","><";$logontracer | Out-File logontracer.xml -Encoding utf8

Let's try again with LogonTracer and this time the parsing succeeds!

[UPDATE 2020-12-07]: Just published a script (ConvertTo-LogonTracer.ps1) that can do the conversion, searches for keywords and splits a large input file to smaller chunks. Edit the settings inside the script before running it. Give it a try! The script is on my GitHub repo: https://github.com/forensenellanebbia/powershell-scripts/blob/master/ConvertTo-LogonTracer.ps1

References

• https://binaryforay.blogspot.com/2019/04/introducing-evtxecmd.html
• https://ericzimmerman.github.io
• https://github.com/JPCERTCC/LogonTracer/wiki
• https://remnux.org/
• https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES
• http://www.happysysadm.com/2014/10/reading-large-text-files-with-powershell.html

Nest camera app (DFRWS2018 Challenge)

Last February, I started playing the DFRWS 2018 challenge:

• http://dfrws.org/dfrws-forensic-challenge
• https://github.com/dfrws/dfrws2018-challenge

This is a brief description of the challenge taken from the website:

"The DFRWS 2018 challenge is about Internet of Things (IoT), defined generally to include network and Internet connected devices usually for the purpose of monitoring and automation tasks. Consumer-grade “Smart” devices are increasing in popularity and scope. These devices and the data they collect are potentially interesting for digital investigations, but also come with a number of new investigation challenges."

Unfortunately for me, I soon had to dedicate my free time to something else and forgot about the challenge. Since the submission window is now closed (the submission deadline was March 20, 2019), I'd like to share my findings about an extension-less SQLite database named frame_database belonging to the Nest app for Android installed on the mobile device found at crime scene. This post is not a full analysis of the app.

The file is located here:

• Evidence: Samsung GSM_SM-G925F Galaxy S6 Edge
• Physical image: blk0_sda.bin
• Database path: P18\data\com.nest.android\cache\f315c6e2b5434a5381f1f5be6f73b4b3

The frame_raw_data_table table in the database caught my attention due to its name and I thought it could contain pictures or videos. The table contains 4548 records and its column names are:

frame_time | chunk_id | chunk_version | gop_start_rowid | sps_bytes | pps_bytes | frame_bytes | chunk_complete

A bit of googling made me find out that gop, sps and pps are parameters related to H.264 videos and stand for:

• GOP = Group of pictures
• SPS = Sequence Parameter Set
• PPS = Picture Parameter Set

This post at stackoverflow was also useful to understand that a stream could look like this:

(AUD)(SPS)(PPS)(I-Slice)(PPS)(P-Slice)(PPS)(P-Slice) ... (AUD)(SPS)(PPS)(I-Slice)

The explanation of all these terms is beyond the scope of this article. The only thing to care about is that there's a data sequence that needs to be rebuilt in order to get meaningful data to analyze.

Having said that, this is a data snippet of what is stored in the database:

The chunk_id field is a timestamp. Frames belonging to the same group have the same chunk_id. The last frame of each group has the chunk_complete value set to 1. The frame_time field is a timestamp too, but I haven't investigated the relationship between this field and the chunk_id. I noticed there's a significant time difference between the two fields. For instance:

• frame_time 1526289810036 translates to 14/05/2018 09:23:30 Unix milliseconds
• chunk_id 1526288836 translates to 14/05/2018 09:07:16 Unix seconds

After several failed data combination attempts, I found out that the following fields containing blob data have to be concatenated in this order to build a working group of frames:

sps_bytes || pps_bytes || frame_bytes (first) || frame_bytes (second) || ... || frame_bytes (last)

I achieved this operation by using a SQL query built with the concatenation operator || and the GROUP_CONCAT function:

SELECT chunk_id,
       ( Hex(sps_bytes) || Hex(pps_bytes)|| frame_bytes ) AS frame_bytes
FROM   (SELECT chunk_id,
               sps_bytes,
               pps_bytes,
               GROUP_CONCAT(Hex(frame_bytes), '') AS frame_bytes
        FROM   frame_raw_data_table
        GROUP  BY chunk_id) 

Then I wrote a quick Python script to automate the data extraction and write data to binary files. To try the script, just copy the code below and customize the paths of db and output_path.

To make Python 3.7.2 correcly execute the SQL query, I had to download the latest version of the SQLite dll from here (I used 64-bit DLL x64 for SQLite version 3.27.2) and save it to C:\Python37\DLLs and overwrite the existing one.

from datetime import datetime
import binascii
import sqlite3

db          = "C:\\temp\\frame_database"
output_path = "C:\\temp\\"

conn = sqlite3.connect(db)
conn = conn.execute("SELECT chunk_id,(hex(sps_bytes) || hex(pps_bytes) || 

frame_bytes) AS frame_bytes FROM (SELECT chunk_id,sps_bytes,pps_bytes,

GROUP_CONCAT(hex(frame_bytes),'') AS frame_bytes FROM 

frame_raw_data_table GROUP BY chunk_id)")

rows = conn.fetchall()

for row in rows:
    chunk_id    = row[0]
    frame_bytes = row[1]
    timestamp   = datetime.utcfromtimestamp(float(chunk_id))
    timestamp   = str(timestamp).replace(" ","_").replace(":","")
    filename    = output_path + str(chunk_id) + "_" + timestamp + ".h264"
    video       = open(filename,"wb")
    video.write(binascii.unhexlify(frame_bytes))

The very small video files (9KB/83KB) can be played with VLC media player (3.0.6):

There's always something new to learn.

Thanks DFRWS for the challenge!

Using small details to add additional context to other artifacts

This is a quick post on some files that could add additional context to other artifacts.

VLC media player (tested version: 3.0.5 / 3.0.6)
Artifact: "restart the playback where left off"
So what?: instead of simply saying that a file was opened with VLC, it could be possible to prove that a media file was also watched from beginning to a certain (milli)second.
Description: if a media file is partially played, VLC tracks the last played position to allow the user to resume playback when reopening the same file. Depending on the operating system, the last played position value is stored in the following files:

• Windows: C:/Users/<username>/AppData/Roaming/vlc/vlc-qt-interface.ini
• Ubuntu: /home/<username>/.config/vlc/vlc-qt-interface.conf
• macOS : /Users/<username>/Library/Preferences/org.videolan.vlc.plist

VLC for Windows and Ubuntu stores the values in the [RecentsMRL] section within the vlc-qt-interface file and these values are expressed in milliseconds. VLC for macOS stores the values in the recentlyPlayedMedia array within a .plist file and the values are expressed in seconds. Based on my tests, a zero value may either mean that a media file has been fully played or that less than five percent of the file contents has been played. 

To automate the parsing I wrote a Python 2.7 script. Here's an example of the output of the script VLC_LastPlayedPosition.py:

C:\>VLC_LastPlayedPosition.py vlc-qt-interface.ini

Analyzing file: C:\temp\vlc-qt-interface.ini...

------------------------------------------
 VLC media player ('RecentsMRL' section)
 The entries are listed by default from
 the most recent to the oldest
------------------------------------------

# | Last Played Position (h:mm:ss) | Media file
1 | 0:04:05 | file:///C:/DATI/audio-video/file2.mp4
2 | 1:26:16 | file:///C:/DATI/audio-video/file1.avi

Output saved to: 20190127_150900_vlc.csv

Adblock Plus (3.4.2) add-on for Firefox (64.0.2)

Artifact: websites that have been whitelisted by the user
So what?: the user visited a website that required ad blockers to be turned off. The user had to manually disable Adblock Plus for that website.

Description: Adblock Plus is a popular ad blocker. The user can choose to allow a site to show ads by "whitelisting" it. When that happens, an entry is added to a storage.js file which is located at:

• Windows: C:\Users\<username>\AppData\Roaming\Mozilla\Firefox\Profiles\<profileID>.default\browser-extension-data\{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}\storage.js
• Ubuntu: /home/<username>/.mozilla/firefox/<profileID>.default/browser-extension-data/{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}/storage.js
• macOS: /Users/<username>/Library/Application Support/Firefox/Profiles/<profileID>.default/browser-extension-data/{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}/storage.js

On a live machine, the list of all the websites that have been whitelisted can be viewed from Firefox by going to:

• about:addons | Adblock Plus options | Whitelisted websites tab

That list can be directly extracted from the file mentioned above by using the following regular expression:

• \[Subscription\]","url=~user~\d*","defaults=whitelist","","\[Subscription filters\]

The entries we're looking for are listed right after the string matching the regular expression. 

This is my script to automate the parsing: Firefox_AdblockPlus.py. Here's an example of the output:

C:\>Firefox_AdblockPlus.py --default-path

# Analysis of Adblock Plus for Firefox #
File: C:/Users/xxxx/AppData/Roaming/Mozilla/Firefox/Profiles/
xxxx.default/browser-extension-data/{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}/storage.js

Whitelisted websites added by user: 2
- macworld.com
- windowscentral.com

I know that this add-on is also available for Chrome, but Chrome uses LevelDBs instead of the storage.js file and at the moment I don't know how to extract what I need in a reliable way.

NoScript (10.2.1) add-on for Firefox (64.0.2)

Artifact: web sites that have been manually set to "trusted" or "untrusted" by the user.
So what?: The user visited a website and, within the browser, manually trusted or untrusted the domain by changing the default settings through the NoScript icon.

Description: NoScript is a security add-on. When the user sets a domain to "trusted" or "untrusted", an entry containing the domain (not the full URL) is added to a .sqlite database named storage-sync.sqlite:

• Windows: C:/Users/<username>/AppData/Roaming/Mozilla/Firefox/Profiles/<profileID>.default/storage-sync.sqlite
• Ubuntu: /home/<username>/.mozilla/firefox/<profileID>.default/storage-sync.sqlite
• macOS: /Users/<username>/Library/Application Support/Firefox/Profiles<profileID>.default/storage-sync.sqlite

From Firefox, NoScript settings can be reviewed by going to:

• about:addons | NoScript options | Per-site Permissions tab

The same data can be retrieved by directly analyzing the collection_data table within the storage-sync.sqlite file. The record having collection_name = default/{73a6fe31-595d-460b-a920-fcc0f8843232} and record_id = key-policy contains the entries we're looking for. The entries have no timestamp and are stored in JSON format in the record field in the sites object. Some domains are already in the file as part of NoScript default settings.

For instance, I navigated to https://www.cnn.com and set some domains to "trusted" and some others to "untrusted".

From the list alone extracted from the .sqlite file, we don't know if the entries refer to domains visited by the user or domains simply "trusted/untrusted" by the user when visiting something else.

After several trial and error attempts, I've noticed it's possible to distinguish between "visited sites" and "sites not visited" by using HTTP requests and observing the responses. It's possible that the user has visited a domain if the domain:

• returns a HTTP response
• doesn't redirect to another domain
• doesn't have a very small Content-Length size

Moreover, if a domain loads scripts residing on another domain, it's highly probable that the former is the domain that was directly visited by the user.

If all or some of the above conditions are not met, then it's possible that the user has set a domain to "trusted" or "untrusted" when visiting any of the visited domains.

The script I've developed can be downloaded here: Firefox_NoScript.py.
If the script is used with a "-r" option, it will send a HTTP request to each site found in the file and try to separate the results. The more entries there are in the .sqlite file, the more accurate the script output is.

C:\>Firefox_NoScript.py --default-path -r

Firefox_NoScript v.20190127
Script to extract the permissions that have been manually added to NoScript add-on

Analyzing file: C:/Users/xxxx/AppData/Roaming/Mozilla/Firefox/Profiles/

xxxx.default/storage-sync.sqlite ...

Non-default permissions found: (6)

Sending HTTP requests to 6 domains found in the file...

   Based on the HTTP responses received, it's possible that:
     ==> the user directly visited 1 domain(s):
      - cnn.com

     ==> the trust level for 5 domain(s) was set by the user
     when visiting other domains:
      - chartbeat.com
      - cookielaw.org
      - optimizely.com
      - postrelease.com
      - sharethrough.com

Output saved to: 20190127_141136_NoScript.csv

The Tor Browser (8.0.4) comes with NoScript (10.2.1) pre-installed. The .sqlite file is located at:

• C:\Users\<username>\Desktop\Tor Browser\Browser\TorBrowser\Data\Browser\profile.default\storage-sync.sqlite

Differently from Firefox, and based on my tests, the storage-sync.sqlite file will retain NoScript last session's settings when the Tor Browser is closed. The settings are lost when the Tor Browser is reopened.

What was my IP? Ask DoSvc on Windows 10

Introduction

   I recently watched the recording of the interesting talk Windows Forensics: Event Trace Logs that Nicole Ibrahim gave at SANS DFIR Summit 2018. I then used the tool ETLParser to dump the contents of all the ETL files stored on my own workstation. I glanced through the giant CSV output file looking for anything of interest and accidentally noticed a string "ExternalIpAddress". As you can see, the string is followed by an IP address.

That IP address is my current public IP address. Why is it there?

The string "ExternalIpAddress" is located next to other interesting strings like "GEO: response", "CountryCode" and a precious timestamp. If this is a geolocation response, what triggered it? Since "ExternalIpAddress"appears several times in the log files, how many geolocation requests have been made so far and why?

DoSvc - Delivery Optimization

   All the hits were found within some logs whose names begin with "dosvc". I searched on Google and found out that "dosvc" stands for Delivery Optimization which is the update delivery service for Windows 10 clients. In the online documentation Optimize Windows 10 update delivery, Microsoft explains what this service does:

Delivery Optimization is a new peer-to-peer distribution method in Windows 10. Windows 10 clients can source content from other devices on their local network that have already downloaded the updates or from peers over the internet.

Depending on the version of Windows 10, the various Event Trace Log (ETL) files created by the Delivery Optimization service (DoSvc) are stored here:

OS Version	Default path	Filename
Win10 (1507)	C:\Windows\Logs\dosvc	dosvc.\d*.\d.etl (e.g. dosvc.1377765.1.etl)
Win10 (1709/1803)	C:\Windows\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Windows\DeliveryOptimization\Logs	dosvc.yyyyMMdd_HHmmss_\d*.etl (e.g. dosvc.20181111_180339_399.etl)

On my computer running Win10 (1803), which is always On and connected to the internet 24/7, the default DoSvc log path contains each day about 140 log files. Based on what I've observed, Win10 (1803) daily removes the ETL files older than 57/58 days. In such a scenario, there may be the chance of extracting several public/external IP addresses from the logs.

Even though the log files lead in the direction of "Delivery Optimization", I think something else might be responsible for the geolocation calls. On my computer, the "Delivery Optimization" service is off. Even the "Location" service is turned off.

(Delivery Optimization)

(Location)

How to parse the logs

   After some trial and error trying to figure out what was the best way to extract the data I needed from the CSV output file created by ETLParser, I found out that Win10 has a built-in Powershell cmdlet named "Get-DeliveryOptimizationLog":

• Announcing Windows 10 Insider Preview Build 17074 for PC: https://blogs.windows.com/windowsexperience/2018/01/11/announcing-windows-10-insider-preview-build-17074-pc/

This cmdlet retrieves decoded logs for Delivery Optimization. If no parameter is given, the cmdlet parses the default DoSvc path. The parameter "-Path" is required to parse other locations:

Get-DeliveryOptimizationLog -Path C:\CustomPath\*

Here I used the cmdlet to search for the keyword "ExternalIpAddress".

PS> Get-DeliveryOptimizationLog | Where-Object Message -Like "*ExternalIpAddress*"

The output was:

(27/Nov/2018 11:05:47)

I also spotted the IP that I was assigned to when using CyberGhost VPN.

(24/Nov/2018 23:26:39)

What triggered the geolocation requests? Using the two examples shown above, I searched for "ProcessId" 13104 and 36192 and noticed that some events contain the message: "Create job name = WU Client Download".

TimeCreated : 27/11/2018 11:05:47
ProcessId   : 13104
ThreadId    : 9952
Level       : 4
LevelName   : Info
Message     : Create job name = WU Client Download, jobId = 4d66d186-68e2-4bfc-8d74-f40de415fc20, type = 0. hr = 0
Function    : CDeliveryOptimizationManager::CreateJob
LineNumber  : 495

TimeCreated : 24/11/2018 23:26:43
ProcessId   : 36192
ThreadId    : 33560
Level       : 4
LevelName   : Info
Message     : Create job name = WU Client Download, jobId = bc698001-4916-4c93-b513-cdcfe325ae9d, type = 0. hr = 0
Function    : CDeliveryOptimizationManager::CreateJob
LineNumber  : 495

What was downloaded and probably installed by the "Windows Update" (WU) client?

PS> Get-WuaHistory | Format-Table

Get-WuaHistory is a third party cmdlet.

This seems to be the answer:

Result    Date                Title                                                                                                       
------    ----                -----                                                                                                       
Succeeded 27/11/2018 11:04:59 Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.281.899.0)                       
Succeeded 27/11/2018 11:04:59 Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.281.899.0)                       

Succeeded 24/11/2018 23:25:54 Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.281.756.0)                       
Succeeded 24/11/2018 23:25:54 Definition Update for Windows Defender Antivirus - KB2267602 (Definition 1.281.756.0) 

I see from my Windows Update history that Windows Defender Antivirus is updated on a daily basis. Based on the log files, it seems that "WU Client" makes a geolocation call before downloading any available update. That could explain why I have at least one geolocation response per day in the logs. Additionally, the creation time (TimeCreated) of each "GEO: response" event message always matches or is very close to the installation date of each antivirus definition update. 

A hive file named "dosvcState.dat" is also involved in the process, but I haven't had the time yet to check what it contains. The hive can be found here:

C:\WINDOWS\ServiceProfiles\NetworkService\AppData\Local\Microsoft\Windows\DeliveryOptimization\State\dosvcState.dat 

Scripting

   I wrote a Powershell script that adapts to my needs the output provided by the mentioned cmdlet. The script adds to the output the name of the log files from which the information was extracted and shows the contents of the "Message" object in a different way. The script will generate two output files in both CSV and JSON format:

• <timestamp>_dosvc_ExtIpAddress: contains the extraction of each IP found in the ETL files;
• <timestamp>_dosvc_ip2location: contains additional details about each unique IP found like the Internet service provider, latitude and longitude. The script uses an external API.

Filenames are prepended with the timestamp of when the script was executed. To use the script, just provide the path containing the ETL files to parse:

PS> .\Get-DoSvcExternalIP.ps1 C:\LogPath

From the logs of the computer I mentioned above (Win10 - 1803), I managed to extract 124 IP addresses whose dates range from October 10th 2018 to yesterday.

I also used the script against the DoSvc ETL files that I extracted from a laptop (Win10 - 1709) that I last used in May in Las Vegas during the Magnet User Summit 2018. I connected to the Wi-Fi network in the hotel a couple of times for a short moment, but that was long enough for Win10.

This is an example of what I could extract from the logs:

PS> (Get-Content 20181205_143620_dosvc_ExtIpAddress.json | ConvertFrom-Json) | Where-Object ExternalIpAddress -eq "xx.xxx.xx.x6"

LogName                  : C:\TEMP\ETL_Laptop2\dosvc.20180522_170803_773.etl

TimeCreated              : 22/05/2018 17:09:04
ExternalIpAddress        : xx.xxx.xx.x6
CountryCode              : US
ProcessId                : 7840
ThreadId                 : 776
Level                    : 4
LevelName                : Info
KeyValue_EndpointFullUri : https://kv801-prod.do.dsp.mp.microsoft.com/all
Version                  : <omissis>
Function                 : CGeoInfoProvider::RefreshConfigs
LineNumber               : 58

These are the additional details provided by the script by using an external API:

C:\> type 20181205_143620_dosvc_ip2location.json | jq

  {

    "as": "ASxxxxx Cox Communications Inc.",
    "city": "Las Vegas",
    "country": "United States",
    "countryCode": "US",
    "isp": "Cox Communications Inc",
    "lat": xx.x892,
    "lon": -xxx.x63,
    "org": "HOSPITALITY NETWORK, LLC",
    "query": "xx.xxx.xx.x6",
    "region": "NV",
    "regionName": "Nevada",
    "status": "success",
    "timezone": "America/Los_Angeles",
    "zip": "89106"
  },

I hope you find this long blog post useful!

You can download the script from my GitHub repository here.


[UPDATE March 31, 2019]:
the peer-reviewed version of this article can be read at DFIR Review.

[UPDATE April 5, 2019]:
I improved the script and created a "Get-DoSvcExternalIP" module for KAPE.

Calllog.db and SMS data on Android 7.0 Nougat

A few weeks ago, Jamie McQuaid at Magnet Forensics wrote an interesting article titled Android Messaging Forensics – SMS/MMS and Beyond. The article is a great overview of the different databases that Android uses to store SMS/MMS data.

Based on my recent findings, I think that another database should be mentioned: calllog.db. During the analysis of two Samsung smartphones (SM-G935F and SM-G930F) running Android 7.0, I've found out that calllog.db also contains SMS data.

I was able to find and analyze this db because I had created a physical dump of the two non-rooted smartphones by using UFED 4PC. I haven't verified if the file can be obtained with other methods like a ADB backup.

This sqlite database can be found here:

\data\com.android.providers.contacts\databases\calllog.db

The table calls within the database has a column named m_content. If the length of this field is greater than 0, it means that the record contains a text message (if zero, it's a phone call log).

At a quick glance, the most relevant fields in the table are:

• number: sender's or recipient's phone number;
• date: message date;
• type: its value indicates if it's a sent or received message; 
• name: contact name associated with the phone number;
• last_modified: message date (it usually matches the date of the field date);
• m_content: first 50 characters of SMS message body.

This is a quick query to view the contents of these fields:

SELECT number,datetime(date/1000,"unixepoch","utc") AS date,type,name,datetime(last_modified/1000,"unixepoch","utc") AS last_modified,m_content FROM calls WHERE m_content <>""

If I run the query SELECT DISTINCT type FROM calls WHERE m_content <>"", I can see that the field type has value "1" or "2". After some comparison with other messages, I deduced that "1" means "received" and "2" is "sent".

As of writing, by default the tools UFED Physical Analyzer (7.9.0.223) and Magnet AXIOM (2.5.1.11408 - with custom artifact plugin) only extract call logs from calllog.db.

This is how I adapted the tools to my needs.

UFED Physical Analyzer

By using the internal tool SQLite Wizard within the Physical Analyzer, I created a query to do the parsing:

Select calls.number,
  calls.date,
  calls.type,
  calls.name,
  calls.last_modified,
  calls.m_content
From calls
Where Length(calls.m_content) > 0

I then mapped the fields by drag&drop and customized the conditions of the field type.

I ran the query and successfully retrieved 500 additional SMS records that I added to my report.

The table shows the big difference in the number of SMS found before and after parsing the file calllog.db.

(before)

(after)

Some messages were duplicates, but many others were not. This is an example sorted by message body:

While creating the query, I tried with and without the "include deleted rows" option. In the end I decided to keep it unticked since I was getting too many false results.

Anyway you can download both query versions from here.

How to import and run the query: Physical Analyzer | Tools | SQLite Wizard | Open SQLite query manager | Import | select the query file to use | Run.


Magnet Axiom

When creating a new case with Magnet AXIOM Process, I recommend to use the Dynamic App Finder ("Find more artifacts" turned ON). It's a useful feature that allows to discover additional databases that may contain relevant data. Once the search is complete, a window pops up asking to select any of the found databases and to map the needed fields.

When done, click on "Save selected artifacts". Magnet AXIOM Examine will show these custom artifacts under the category "Custom".

My custom artifact can be downloaded from here. Just import it from the menu in Magnet AXIOM Process (Tools | Manage custom artifacts | Add new custom articact) or simply copy the file to the path "C:\Program Files\Magnet Forensics\Magnet AXIOM\AXIOM Process\plugins".

Converting from .heic to .jpg

A file with a .heic extension is an image using the new High Efficiency Image File Format (HEIF).

If somehow you've never heard about it, here are some sample files:
https://trac.ffmpeg.org/ticket/6521

At the time of this writing, not all forensic tools fully support it. For instance, among the tools I have:

• X-Ways Forensics (v19.7 SR-2) does no picture preview or metadata extraction. It has by the way a file carving algorithm for .heic images since version 19.5 Preview 2;
• Magnet AXIOM (v2.5.1.11408) is able to preview the images but it doesn't extract (for now) any EXIF data from the HEIF format;
• Cellebrite Physical Analyzer (v7.9.0.223) stands out from the crowd. If you have a set of images, you can easily view them and parse their metadata by simply choosing: File | Open (advanced) | Blank project | Folder | select the folder containing the pictures to analyze | Finish | Start decoding

Depending on the situation, it could be useful to use external viewers like XnView / CopyTrans HEIC or to convert .heic images to .jpg to make them "compatible" with other tools.

What follows is my method to do the conversion and can be used within a script.

Download and install ImageMagick (free - tested version on Win10: v7.0.8-12 7.0.8-28-Q16-x64-dll). Then open the command prompt and type:

magick SrcFile.heic DstFile.jpg

This will create a JPEG version of the HEIC file, but for some reason the EXIF metadata of the newly created file will be ignored and not parsed by many tools. After some trial and error, I managed to fix this issue with ExifTool (v11.10):

exiftool -overwrite_original -all= -TagsFromFile SrcFile.heic DstFile.jpg

The meaning of each option is the following:

-overwrite_original	Overwrite the destination file without creating any backup copy of the destination file
-all=	Strip off all metadata from the destination file
-TagsFromFile	Copy all metadata tags from the SourceFile into the DestinationFile
SrcFile.heic	Source file
DstFile.jpg	Destination file

If you need to set the filesystem timestamps "last modified date" and "creation date" equal to the ones of the source .heic file, run ExifTool with these options:

exiftool -overwrite_original -TagsFromFile SrcFile.heic -FileModifyDate -FileCreateDate DstFile.jpg

For more details, you may check the ExifTool Documentation.

Additional resources on the HEIF format:

• CCL Blog - iOS 11: HEVC and HEIF (heic) files
• Cheeky4n6Monkey -  Monkey takes a .heic

[UPDATE February 20, 2019]: Thanks to Phill Moore for letting me know that the metadata step described in the article is no longer needed when using the latest version of ImageMagick. Metadata is now properly added to the converted files.