Windows Memory and Process Analysis: Volatility3 Walkthrough
- DFIRHive
- Oct 6
- 9 min read
Updated: Oct 9
Title: When the Files Go Missing
Challenge Source: MemLabs Lab 5
Case Introduction:
We recently received a memory dump from a client who suspected unauthorized access to his system. He mentioned noticing strange filenames — random strings of letters and numbers — that didn’t make any sense. Even worse, his favorite application began crashing every time he tried to run it.
Was this a simple case of software corruption, or was something malicious hiding in the system? Let’s find out.
In memory forensics challenges like this, the case description often hides subtle clues about the underlying issue. Reading between the lines helps us focus our investigation more effectively.
Here’s what immediately stood out:
Unreadable filenames: Likely temporary encrypted data or files automatically generated.
Frequent crashes of a favorite app: Could indicate DLL injection, tampering, or process manipulation.
Three flags: Suggests a staged challenge — where each discovery leads logically to the next.
These indicators suggest we’re likely dealing with hidden processes, unusual file activity, and possible code injection artifacts.
Memory Dump Analysis
Let's begin working with the provided memory image file MemLabs_Lab5.raw.
For this analysis, I used Volatility 3. The challenge was originally designed for Volatility 2, so some plugin names differ — but the underlying forensic logic remains the same.
Unlike Volatility 2, Volatility 3 does not require a profile to be specified. It automatically detects OS structures, allowing us to move straight into process analysis.Process analysis — first pass
To identify suspicious activity, I started by enumerating running processes and analyzing parent-child relationships using Volatility 3’s windows.pslist and windows.pstree plugins.
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.pslist
:
A few things jumped out immediately: DumpIt.exe, WinRAR.exe, and multiple instances of NOTEPAD.EXE.Most of them shared the same parent process — PID 1580, which turned out to be explorer.exe.
That alone looked odd. DumpIt is typically used for memory acquisition (and was mentioned as genuine by the challenge author). However, WinRAR implies archiving activity, and a flood of Notepad processes rarely points to normal user behavior — definitely something worth investigating.
To understand how these processes were related, I used the pstree plugin for a hierarchical view.
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.pstree
From the tree view, it became clear that everything originated from explorer.exe, which in turn spawned several other processes:
1580 explorer.exe
├── 2144 VBoxTray.exe
├── 2208 DumpIt.exe
├── 2924 WinRAR.exe
├── 2724 NOTEPAD.EXE
├── 2056 NOTEPAD.EXE
├── 1388 NOTEPAD.EXE
└── 2744 notepad.exe
Multiple capitalized and lowercase Notepad processes, alongside WinRAR, all executing within the same minute — definitely a red flag.
Check command lines and paths
To understand how these executables were launched, I examined their command-line arguments. This helps reveal where the processes ran from, what parameters were used, and sometimes even which files they interacted with.
We can use either the cmdline plugin directly or parse our pstree output. Here, I chose pstree:
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.cmdline > cmdline.txt
grep -iE 'notepad|winrar|dumpit|explorer' cmdline.txt
Findings:
A NOTEPAD.EXE process was running from C:\Users\SmartNet\Videos\NOTEPAD.EXE — not the legitimate C:\Windows\System32 path. That’s immediately suspicious.
WinRAR.exe had opened the file SW1wb3J0YW50.rar — which looks like encoded text. That stood out as our next lead.
So at this point, my two primary suspicions became:
The rogue Notepad in the Videos directory, and
The suspicious archive SW1wb3J0YW50.rar that may contain hidden data.
Timestamps from pslist and pstree confirmed everything happened in a tight time window:
PID | Process | Timestamp (UTC) |
1580 | explorer.exe | 2019-12-20 03:46:49 |
2144 | VBoxTray.exe | 2019-12-20 03:46:50 |
2924 | WinRAR.exe | 2019-12-20 03:47:13 |
2744 | notepad.exe | 2019-12-20 03:47:21 |
2724 | NOTEPAD.EXE | 2019-12-20 03:47:53 |
1388 | NOTEPAD.EXE | 2019-12-20 03:48:00 |
Everything occurred within two minutes — a very narrow execution window, suggesting a single user or automated action chain triggered the sequence.
Find hidden or terminated processes (psscan)
Attackers often try to conceal their activity by unlinking process structures from the active process list. To detect such behavior, I scanned the raw memory image for EPROCESS objects using the windows.psscan plugin and compared the results against pslist.
Command:
volatility3 vol.py -f MemLabs_Lab5.raw windows.psscan > psscan.txt
This plugin examines memory for all process structures — even those that have been terminated, hidden, or unlinked from the active list. Comparing pslist (active processes) and psscan (raw memory scan) results helps uncover inconsistencies that may indicate stealth activity.
To simplify the comparison, I extracted only the PID and process name fields from both results:
awk '{print $1, $3}' > pslist.txt | sort > pslist_processed.txt
awk '{print $1, $3}' > psscan.txt | sort > psscan_processed.txt
Interestingly, both explorer.exe and multiple NOTEPAD.EXE instances appeared in both pslist and psscan outputs.
This indicates they were not hidden, but actively involved around the same short time window — consistent with earlier observations from the process tree.
Inspect WinRAR (PID 2924): handles & DLLs
After identifying WinRAR.exe as a key process linked to the suspicious archive, I examined which files it interacted with and what DLLs it had loaded. These two aspects — handles and DLLs — often provide strong leads about file access patterns and process behavior.
Command:
python3 vol.py -f MemoryDump_Lab5.raw windows.handles --pid 2924
The handles output revealed several open file objects, including entry related to Documents — which immediately stood out, given our archive extraction activity.
Next, I listed the DLLs loaded by the process:
Command:
python3 vol.py -f MemoryDump_Lab5.raw windows.dlllist --pid 2924
The DLL list didn’t reveal anything particularly malicious — most were standard WinRAR dependencies. However, dlllist can still confirm that the process executed properly and didn’t dynamically load external malicious libraries during runtime.
Searching for referenced files in memory
Since I saw document paths in the handle output, I ran a filescan to cross-check whether those files (or similar ones) were still mapped in memory. This plugin scans raw memory for FILE_OBJECT structures — helping recover deleted or recently accessed files.
Command:
python3 vol.py -f MemoryDump_Lab5.raw windows.filescan | grep -i documents
This produced multiple file hits, including our suspicious .rar entry. The scan provided physical memory addresses (physaddr), which could then be used to dump the file directly from memory.
Dumping the RAR file from memory
Using the physical address obtained earlier, I extracted the suspected archive using the dumpfiles plugin:
Command:
python3 vol.py -f MemoryDump_Lab5.raw windows.dumpfiles --physaddr <>
The dumped file was saved as a .dat file. Running the file command on it confirmed it was a RAR archive, matching the suspicious name SW1wb3J0YW50.rar we had seen in WinRAR’s command line earlier.
file <file.0...>
e.0x....dat SW1wb3J0YW50.rar
unrar t SW1wb3J0YW50.rar
# will prompt for password if protected
Result: The archive existed — but was password-protected, marking the next stage of our investigation: recovering the password from memory.
Search for the archive password (explorer + WinRAR memory)
At this point, the RAR file (SW1wb3J0YW50.rar) was our central artifact — it existed in memory but was password-protected. The next step was to determine where that password might have been cached within volatile memory.
In practice, passwords or related credentials often reside in memory regions belonging to these processes:
explorer.exe — the Windows shell; often retains clipboard contents, file dialog text, and recent user interaction data.
rdpclip.exe — responsible for clipboard redirection during Remote Desktop sessions.
WinRAR.exe — may temporarily store the password in heap memory while decrypting or opening an archive.
Since explorer.exe (PID 1580) was the parent of WinRAR.exe (PID 2924), both processes became prime targets for memory extraction.
Step 1: Dump the target processes
To extract any in-memory password remnants, I dumped both processes using Volatility’s memdump plugin:
python3 vol.py -f MemLabs_Lab5.raw windows.memdump --pid 1580 --dump
python3 vol.py -f MemLabs_Lab5.raw windows.memdump --pid 2924 --dump
This created raw memory segment files (1580.data, 2924.data), which can then be searched for text or encoded password traces.
I converted these dumps into .txt format for easier review in Notepad++ and further string analysis.
Step 2: Manual inspection in Notepad++
I opened both converted memory dumps in Notepad++, manually searching for potential clues such as rar, password, Documents, and file paths that might indicate password-related activity.
During this inspection, I found several file names and partial paths that did not appear in the filescan results — suggesting that certain files had been deleted or unlinked but their remnants still persisted in memory.
For instance, I noticed references to files within the Documents directory and an image named Password.png, indicating previously accessed or created items tied to sensitive data.
These entries pointed to past interactions with the suspicious archive and other related files stored in the user’s Documents path.
Step 3: Focused string search (Explorer memory)
Since WinRAR.exe was launched by Explorer.exe, any file browsing, password entry, or dialog interaction was likely cached in Explorer’s memory.
To refine the search, I extracted all human-readable ASCII strings (minimum 8 characters) and filtered for occurrences of “rar” to detect any RAR-related data.
Command:
strings -n 8 1580.data | grep -i "rar" > out.txt
This command isolates printable text segments containing “rar,” helping identify file paths, dialog prompts, or potentially encoded password fragments stored during the RAR file operation.
Step 4: Identifying suspicious artifacts
In the output, I found multiple readable lines, and one in particular caught my eye —a filename ending in .lnk but containing a long, random-looking string of letters and numbers (for example: ZmxhZ3...==.lnk).
That clearly didn’t look like a regular Windows shortcut. The structure — mixed uppercase/lowercase characters with “=” at the end —strongly resembled Base64-encoded data.
the pattern itself suggested it was encoded text rather than a normal file name. Later, I confirmed that decoding it revealed the first flag, meaning the password or related encoded data was indeed present in memory.
Step 5: Optional pattern expansion (if you want to confirm encoding)
The initial command I used —strings -n 8 1580.txt | grep -i "rar" —is not Base64-specific. It simply extracts human-readable ASCII text from the dump. In this case, it worked because the encoded filename containing the flag also included the keyword “rar.”
To broaden the search and confirm whether more encoded artifacts existed, we can expand the pattern search using egrep for Base64 and hex-encoded strings:
egrep -o '[A-Za-z0-9+/]{8,}={0,2}' 1580.txt | sort -u > base64_candidates.txt
egrep -o '\b[0-9A-Fa-f]{16,}\b' 1580.txt | sort -u > hex_candidates.txt
These expressions extract Base64 or hex-encoded strings that could represent hidden credentials, file references, or encoded data remnants.
Even though my original ASCII-based search already surfaced the relevant Base64 string, running these broader pattern searches helps ensure that no additional encoded data is missed — a useful step in comprehensive memory forensics.
Step 6: Validation with MFT scan
To validate the finding, I ran the MFT scan plugin and exported the results as a text file.When reviewing the output, I noticed multiple entries — including .lnk and .bmp files — all sharing the same encoded string.
Both file types were created or accessed within seconds of each other, a strong indicator that the same data was copied, cached, or temporarily reused in multiple forms.
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.mftscan > mftscan.txt
(Some data may appear with .lnk or .bmp extensions — these are heuristic guesses made by the tool, not actual file types.)
Step 7: Applying the Password & Verifying
After identifying the Base64-encoded password, I used it to extract the contents of the protected archive.
Inspecting the Suspicious notepad.exe
After successfully unlocking the RAR archive, one file immediately stood out — a suspicious notepad.exe located in the Videos directory, rather than the legitimate C:\Windows\System32 path.
This same executable had been observed earlier in the process tree, launched multiple times under slightly different names (NOTEPAD.EXE, notepad.exe).Given the challenge description mentioning that “the user’s favorite application keeps crashing,” this was almost certainly the culprit.
Step 1: Preliminary Inspection (Handles & DLLs)
Before performing a full memory dump, I first ran the handles and dlllist plugins to check which resources and libraries the process interacted with
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.handles --pid 2724
python3 vol.py -f MemLabs_Lab5.raw windows.dlllist --pid 2724
Both outputs were fairly normal — no abnormal DLL injections or external handle references were detected. However, the repetition of this same process (NOTEPAD.EXE) in short intervals hinted at either process tampering or repeated execution attempts due to crashes.
Step 2: Dumping the Notepad Process Memory
Once the preliminary checks were complete, I dumped the entire process memory for deeper analysis:
Command:
python3 vol.py -f MemLabs_Lab5.raw windows.memmap --pid 2724 --dump
python3 vol.py -f MemLabs_Lab5.raw windows.dumpfiles --physaddr<>
"This command provided 3 outputs- 2 executables and 1 text"
Note:
When extracting executables from memory, both memmap and dumpfiles can yield .exe files — but they come from different layers of memory.dumpfiles reconstructs file objects mapped in memory (representing data as stored on disk), while memmap extracts the active, in-memory process space reflecting its runtime state.
In this case, the recovered notepad.exe via dumpfiles was the suspicious binary loaded from the Videos folder, confirming its execution path and making it suitable for reverse engineering.
Step 4: Reverse Engineering Preparation
For deeper inspection, We extracted the dumped notepad.exe executable and prepared it for static analysis. Tools such as Ghidra, IDA Pro, or Cutter can be used to:
Identify embedded payloads or encoded strings.
Verify whether the binary drops secondary executables or makes suspicious API calls.
Recover any remaining challenge flags (in this case, the third one).
While this walkthrough focused on memory-level investigation, further reverse engineering of the dumped executable could reveal the final payload and complete the challenge chain.