Recovering old Virtualbox VM's from an Encrypted disk.
If you are reading this, I have succeeded in what I set out to achieve this post. HIGH FIVE.
The scene.
At an old kitchen table, I built a dual purpose machine. Its tasks?
- Serve as a headless VM server for me to freelance development work in VMs
- Perform GPU (crypto currency) tasks with a Radeon 6950 and a 5770 to which I had overclocked, undervolted, unlocked shaders and turned into heaters which sounded like a 747 taking off.
Actual 5770 pictured w/ AMD phenom II stock CPU fan cable tied to it - Yes, this actually mined BTC.
To begin with, I had tried PCIe passthrough with VMWare ESXi to allow GPU tasks to happen on a VM, but to my dismay and many hours not crunching those sweet sweet hashes (read: making pocket money while BTC was upcoming $1000/btc). It wasn't to be, so I settled for an Ubuntu (swahili for: "can't configure debian") LTS server install with fluxbox and VirtualBox to run my VMs. And for a time, all was good.
The base disk was 250gb and encrypted using LUKS a form of software encryption and the default encryption option in the Ubuntu installer. I had chosen a strong passphrase and I rarely needed to enter it. My personal home directory containing the mining software and VMs were also encrypted on a ~230gb partition. Each VM I provisioned was naturally done the same way. While there was significant performance overhead at times, my concern was that some of my work was development for local gov clients along with some forensic incident response for another client, so VM isolation and disk encryption was a higher priority than performance.
Fast-Forward nearly 3 years.
Profiting from GPU mining was long out of date, I was sick of the noise/heat/electricity use of the machine and I had purchased 2 new gaming rigs. I'd provisionned a new ESXi box from old parts (even a watercooled CPU) and I'd thrown the HDD into box when I'd moved house - my unpublished personal tech blog was on one of the VMs.
It's now 3am on Monday morning. I have to work in 5 hours, and I want my data. I'd forgotten the depth of my past setup.
So I find the box that contains the drive; easy part done. I grab a USB to SATA adapter from the same box (smart enough to pack those 2 together) I attach the disk to my Windows desktop. First mistake.
I promptly asked Jeeves and he brings me a ext3/4 driver for Windows and a program that claims to allow you to mount VDIs. Cool. That's all I should need, right? Wrong. I remember the LUKS encryption. Fuck. What the fuck is my passphrase and how the fuck am I going to enter it?
Okay, be chill, connect it to an existing Ubuntu VM, easy. Wrong. Hard.
Connecting the disk to the VM was the easy part. It appeared as /dev/sdb, time to check fdisk.
:~$ sudo fdisk -l /dev/sdb
Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 2048 499711 248832 83 Linux
/dev/sdb2 501758 488396799 243947521 5 Extended
/dev/sdb5 501760 488396799 243947520 83 Linux
While /dev/sdb1 has the boot flag set, and what appears to be an MBR, its not bootable.
:~$ sudo xxd -l 511 /dev/sdb
0000000: faeb 2101 b401 4c49 4c4f 1800 73cf 1853 ..!...LILO..s..S
...
Fucking LILO. I never use LILO, but there was some reason I can't recall that I had used it instead of Grub. I figured fixing the bootloader for a system I wasn't sure even ran on virtual hardware was a less direct route to my data than just trying to mount the disk.
Moving on, sdb5 is almost definitely the /home partition Time to mount it... but I definitely don't remember how to mount a LUKS off the top of my head...
I asked Jeeves a few more questions and I needed to RTFM cryptsetup and cryptsetup open --type LUKS <device><name>
Alright cool, but cryptsetup isn't installed, sudo apt-get install cryptsetup
Now the dreaded Passphrase:.
After bashing on my keyboard for a good 35mins at least, I'd remembered the passphrase based on an old password scheme I used to use. Finally, the disk was setup.
Now to mount it. Easy. sudo mount /dev/mapper/tmp tmp
Nope, Hard. Unknown filesystem type 'LVM2_member' Fuck.
Now personally I dislike LVM, not because it's not feature rich, not because it's not flexible, not because it's not good at x, y or z. I disklike LVM because of moments like this. All I want to do is mount the goddamn disk. Is that so much to ask?
Again, I turned to my butler Jeeves and asked him about it.
Good news, blog posts on how to do it. ~= s/Good/Bad/
If there is one thing the uptake on using *nix operating systems has bought about, it's terrible blog posts with 15 1 line commands that don't work because they are out of context. I'm not saying my blog is any better, I'm just saying 99% of Stackoverflow could lead to actual stack overflows.
So I installed lvm2 sudo apt-get install lvm2 Let's try again, starting with the kernel module. sudo modprobe dm-mod
hmm
:~$ sudo lvscan
inactive '/dev/<redacted>-vg/root' [228.64 GiB] inherit
inactive '/dev/<redacted>-vg/swap_1' [4.00 GB] inherit
Okay, it's not /home, its /, still, good news.
Both VolumeGroups are inactive, so time to activate them... more late night chats with Jeeves and RTFM. Jeeves told me about a post on Pissed Off Admins - seemed fitting, but also accurate.
:~$ sudo vgchange -ay
2 logical volume(s) in volume group "<redacted>-vg" now active
:~$ sudo lvscan
ACTIVE '/dev/<redacted>-vg/root' [228.64 GiB] inherit
ACTIVE '/dev/<redacted>-vg/swap_1' [4.00 GB] inherit
Nice. Steps in the right direction. Surely I must almost be at my data.
I can just mount those, right?
:~$ sudo mount /dev/<redacted>-vg/root ~/tmp
:~$ cd tmp
:~/tmp$ ls -al
total 128
...
Awwww yeahhh, disk mounted! I can smell my stuff, it's so close now, a few more commands and I'll be there! Very Wrong.
:~/tmp$ cd home/<redacted>
:~/tmp/home/<redacted>$ ls -la
total 12
dr-x------ 2 <redacted> <redacted> 4096 <redacted> 25 2013 .
drwxr-xr-x 4 root root 4096 <redacted> 25 2013 ..
lrwxrwxrwx 1 <redacted> <redacted> 56 <redacted> 25 2013 Access-Your-Private-Data.desktop -> /usr/share/ecryptfs-utils/ecryptfs-mount-private.desktop
-rw------- 1 root root 176 <redacted> 25 2013 .bash_history
lrwxrwxrwx 1 <redacted> <redacted> 35 <redacted> 25 2013 .ecryptfs -> /home/.ecryptfs/<redacted>/.ecryptfs
lrwxrwxrwx 1 <redacted> <redacted> 34 <redacted> 25 2013 .Private -> /home/.ecryptfs/<redacted>/.Private
lrwxrwxrwx 1 <redacted> <redacted> 52 <redacted> 25 2013 README.txt -> /usr/share/ecryptfs-utils/ecryptfs-mount-private.txt
Shit. Broken symlinks. Maybe my .bash_history will have some clues... Nope, a few entries restarting nodm, xorg and some other things I'd run as root early in my installation, breaking the ownership of my .bash_history meaning none of my history had been logged anyway.
This is 'basic ubuntu' configuration though, surely the README.txt will give me some hints. The readme points to a file which is part of the package ecrypt-utils, So I installed that sudo apt-get install ecrypt-utils and started to wander my way around.
THIS DIRECTORY HAS BEEN UNMOUNTED TO PROTECT YOUR DATA.
From the graphical desktop, click on:
"Access Your Private Data"
or
From the command line, run:
ecryptfs-mount-private
okay...
ecryptfs-mount-private
ERROR: Encrypted private directory is not setup properly
Well I didn't see that coming.
So how do I mount it? Jeeves was telling me to type ecryptfs-recover-private. With my faithful butler at my side, I continued.
:~/tmp/home/<redacted>$ sudo ecryptfs-recover-private
INFO: Searching for encrypted private directories (this might take a while)...
Wait what, searching my whole machine? but it has network storage attached that is totally irrelevant.
^C
:~/tmp/home/<redacted>$ man ecrypt-recovery-private
Okay, so I can pass a directory to it, but these symlinks are broken, so I'll go to their original location.
:~/tmp/home/<redacted>$ cd ../.ecryptfs/<redacted>
:~/tmp/home/.ecryptfs/<redacted>$ sudo ecryptfs-recover-private .
INFO: Found [.].
Try to recover this directory? [Y/n]:
INFO: Could not find your wrapped passphrase file.
INFO: To recover this directory, you MUST have your original MOUNT passphrase.
INFO: When you first setup your encrypted private directory, you were told to record
INFO: your MOUNT passphrase.
INFO: It should be 32 characters long, consisting of [0-9] and [a-f].
Enter your MOUNT passphrase:
ERROR: Failed to mount private data at [/tmp/ecryptfs.ejg0l6zo].
Okay, so this isn't going to be a passphrase I know... but I've never had to enter it before, so it must be somewhere.
:~/tmp/home/.ecryptfs/<redacted>$ ls -la .Private/
total 3483220
drwx------ 17 <redacted> <redacted> 12288 <redacted> 26 2014 .
drwxr-xr-x 4 <redacted> <redacted> 4096 <redacted> 25 2013 ..
drwxrwxr-x 5 <redacted> <redacted> 4096 <redacted> 6 2014
lrwxrwxrwx 1 <redacted> <redacted> 124 <redacted> 25 2013
...
:~/tmp/home/.ecryptfs/<redacted>$ ls -la .ecryptgfs
total 20
drwx------ 2 <redacted> <redacted> 4096 <redacted> 25 2013 .
drwxr-xr-x 4 <redacted> <redacted> 4096 <redacted> 25 2013 ..
-rw-r--r-- 1 <redacted> <redacted> 0 <redacted> 25 2013 auto-mount
-rw-r--r-- 1 <redacted> <redacted> 0 <redacted> 25 2013 auto-umount
-rw------- 1 <redacted> <redacted> 16 <redacted> 25 2013 Private.mnt
-rw------- 1 <redacted> <redacted> 34 <redacted> 25 2013 Private.sig
-rw------- 1 <redacted> <redacted> 48 <redacted> 25 2013 wrapped-passphrase
Things! Okay this is good. I can see there are files and folders, and I can see a wrapped passphrase, and the manual showed a unwrapping command!
:~/tmp2/home/.ecryptfs/<redacted>$ ecryptfs-unwrap-passphrase .ecryptfs/wrapped-passphrase
Passphrase:
According to Jeeves, this would be a password I know, though maybe not my login password... Success! After a few tries (maybe 20 minutes but who's counting now), it was in fact a passphrase I could derive from an old password scheme. It's given me a 32 character hex string.
Again I tried to use ecrypt-recover-private but to no avail. Checking dmesg I could see this.
process_request_key_err: No key
Could not find valid key in user session keyring for sig specified in mount option: [<redacted>]
One or more global auth toks could not properly register; rc = [-2]
Error parsing options; rc = [-2]
Apparently I needed to insert the tok with the specified sig into the user session keyring.
:~/tmp/home/.ecryptfs/<redacted>$ sudo ecryptfs-add-passphrase --fnek
Passphrase:
Inserted auth tok with sig [<redacted>] into the user session keyring
Inserted auth tok with sig [<redacted>] into the user session keyring
This seems good. With this information, I should now be able to mount it...
:~/tmp/home/.ecryptfs/<redacted>$ sudo mount -t ecryptfs .Private ~/ecryptfs
Passphrase:
Select cipher:
1) aes: blocksize = 16; min keysize = 16; max keysize = 32
2) blowfish: blocksize = 8; min keysize = 16; max keysize = 56
3) des3_ede: blocksize = 8; min keysize = 24; max keysize = 24
4) twofish: blocksize = 16; min keysize = 16; max keysize = 32
5) cast6: blocksize = 16; min keysize = 16; max keysize = 32
6) cast5: blocksize = 8; min keysize = 5; max keysize = 16
Selection [aes]:
Select key bytes:
1) 16
2) 32
3) 24
Selection [16]:
Enable plaintext passthrough (y/n) [n]: y
Enable filename encryption (y/n) [n]: y
Filename Encryption Key (FNEK) Signature [<redacted>]: <redacted>
Attempting to mount with the following options:
ecryptfs_unlink_sigs
ecryptfs_fnek_sig=<redacted>
ecryptfs_passthrough
ecryptfs_key_bytes=16
ecryptfs_cipher=aes
ecryptfs_sig=<redacted>
WARNING: Based on the contents of [/root/.ecryptfs/sig-cache.txt],
it looks like you have never mounted with this key
before. This could mean that you have typed your
passphrase wrong.
Would you like to proceed with the mount (yes/no)? : yes
Would you like to append sig [<redacted>] to
[/root/.ecryptfs/sig-cache.txt]
in order to avoid this warning in the future (yes/no)? : no
Not adding sig to user sig cache file; continuing with mount.
Mounted eCryptfs
MOUNTED!?!?! COULD IT BE TRUE?!
:~/ecryptfs$ ls -la
total 3483220
drwx------ 17 <redacted> <redacted> 12288 <redacted> 26 2014 .
drwxr-xr-x 27 <redacted> <redacted> 4096 <redacted> 11 04:51 ..
-rwxrwxr-x 1 <redacted> <redacted> 108558863 <redacted> 15 2013 amd-driver-installer-catalyst-13.1-legacy-linux-x86.x86_64.run
-rw-rw-r-- 1 <redacted> <redacted> 106908508 <redacted> 6 2014 amd-driver-installer-catalyst-13.1-legacy-linux-x86.x86_64.zip
SUCCESS! ... for the first layer.
Now it's 6am, and I'm going to bed because I realise I have poorly named all the VMs, each of them encrypted in a similar fashion with unique passphrases, and I've no clue which one has the data I originally wanted.
Overnight I copied the VMs I wanted to network storage and went to sleep for it to complete.
Day 2
Upon waking up, I could hear a distinct loud ticking sound that was indicitive that the HDD was damaged. Great.
Checking the network storage showed that 4 our of 8 VMs had copied, 1 was probably only a partial.
This next part was much easier, I reinstalled VirtualBox on my desktop - though since the last time I've used it, the media manager no longer shows an add button to use an existing VDI. Back to RTFM and it turns out the new way is to simply create a new VM and choose an existing disk, which allows you to select a VDI from the filesystem.
At this point, had I actually wanted the whole system, I could have exported the VMs to OVA/OVF then imported them into my ESXi box, but I don't really want the enviornments, just the code on them, which I plan on checking into my local git server for future safe keeping.
Since some of the old VMs also contained potentially sensitive data, I decided it was best to shred those images rather than attempt to recover them.
I placed the disk in the freezer and let it chill out for a bit.
After returning it to the VM and restepping over the decryption, I managed to save a few more VMs. Unfortunately one that wasn't able to be saved was a demo of a system I was asked to build using Active Directory to manage eJabberd authentication, groups, contact lists and group chats (MUCs). Though that project never came to fruition, I did complete the code and would have liked to put it up on my github.
It was a super longshot, but I'm going to try and use extundelete to recover it from the encrypted partition. There was even less chance of success as the disk had to be mounted to attempt recovery, as unmounting it would leave the contents encrypted and unrecoverable.
First, I needed to find the name of the Virtualbox VMs directory in the .ecryptfs.
I touched the directory I wanted in the mounted directory, then went and sorted by timestamp with ls -ltr. The last name in the list was the encrupted name of the Virtualbox VMs directory in my encrypted home.
:~/tmp/home/.ecryptfs/<redacted>/.Private$ sudo extundelete --restore-files 'ECRYPTFS_FNEK_ENCRYPTED.FWZmvYGQrp0gcEQLJYD9B8pnts7iWw9yPCY0XiZDQ9WLRqSxeMHmwj1OQU--' /dev/mapper/<redacted>--vg-root
WARNING: Extended attributes are not restored.
WARNING: EXT3_FEATURE_INCOMPAT_RECOVER is set.
The partition should be unmounted to undelete any files without further data loss.
If the partition is not currently mounted, this message indicates
it was improperly unmounted, and you should run fsck before continuing.
If you decide to continue, extundelete may overwrite some of the deleted
files and make recovering those files impossible. You should unmount the
file system and check it with fsck before using extundelete.
Would you like to continue? (y/n)
y
Loading filesystem metadata ... 1830 groups loaded.
Loading journal descriptors ... 26383 descriptors loaded.
Writing output to directory RECOVERED_FILES/
As expected, nothing was recovered - but it was worth trying anyway. There are certainly other ways to attempt to recover lots data but it was too late...
Right then, the clicking returned and that was that. The disk had finally died.
[ 1741.402367] Read(10): 28 00 00 00 08 00 00 00 08 00
[ 1741.402371] end_request: I/O error, dev sdb, sector 2048
[ 1741.402400] Buffer I/O error on device sdb, logical block 256
Now cracks a noble heart. Goodnight, sweet prince; And flights of angels sing thee to thy rest.
All that remained of the physical medium was a series of seemingly random 1's and 0's that without extreme measures and the secrets stored in my head would never be readable again.
Now to get to my blog files.
- Import the VM to Virtualbox on my desktop
- Enter the LUKS passphrase at boot time - 3-4 tries to derive the phrase from my scheme
- Login to my account - again a few tries based on the scheme
That was significantly easier!
Finally, I had reached the Goal, my final /home was recovered, and it contained my blog and some code. It might seem like something I could have easily replicated, and it is, but if that was the case why didnt I shred the disk to begin with?
It was equally the purpose of recovering something I had bothered to keep and the need to build a new blog, and maybe a pinch of procrastination mixed with some frustration and after it all, a good excersise.