On Friday, a buddy of mine reached out to me on Discord - “hey, uh, do you happen to know how to recover data from a dead NVMe drive?” Well, no, I didn’t at the time, but challenge accepted? I figured that I could at least figure out just how bad off the drive was. I steeled both my friend and myself for the event in which we wouldn’t be able to recover anything, and we got to work.
When attempting to do data recovery off of any failed drive, the first order of business is to try and grab an image of the drive. Depending on what data is most important to recover, and your knowledge of the layout of the drive, you might want to focus on a specific part of the drive, but the overall process is the same. Create an image of the drive first, and perform your data recovery off the image. Doing it this way reduces the chance that the failing disk will corrupt further along the process, and ensure that you are able to recover the most data.
ddrescue to the, well, rescue
After some quick searching, it turns out that making a forensic image of an NVMe drive can use the same tools that I’ve used in the past for SATA drives in the past, including ddrescue. ddrescue, much like dd, is a tool that copies raw blocks from a disk drive, however ddrescue uses a multi-step process to attempt to copy as much data from the disk as possible on a failed disk. This meant that I could stand on the shoulders of the data recover giants that I was already used to. We set out to create a bootable Linux flash drive with SystemRescue (fka SystemRescueCD), which has ddrescue as well as read-write NTFS drivers (via ntfs-3g) and pre-installed smartmontools to check the drive health metrics. Once booting into SystemRescue, we checked to see if the drive showed up, and lo and behold it did. Things looking good. Next, we checked the NVMe drive’s SMART status. I was concerned that perhaps the disk failure might have been part of the recent Windows 11 update woes, and if so, it wasn’t actually dead. No such luck unfortunately. The drive controller reported that it had run out of free blocks, and had gone into emergency read-only mode as a result. Not great, but as far as failure states to be in, this seemed to be the best case scenario.
Alright, with a drive in a failed read-only state and an external drive with more capacity than the failed drive to record an image to, we were ready to begin the recovery operation. I didn’t use any of ddrescue’s more advanced options, and let it go with its default process. After about 7 hours, it completed the recovery operation with only 32 MB out of the 2 TB drive unrecoverable. Not terribly surprising on a TLC SSD that the unrecoverable chunk of data was so large, and statistically unlikely that any data we recover will be in the block since the failure happened while my friend was away from his computer. This gave us a disk image for us to perform recovery on.
Recovering data from the image
Huzzah, we have a disk image! Now to mount it up and pull data out of it. On Linux and Unix, it’s possible to treat files as block devices by creating a loop device, so I used losetup to configure our recovered disk image as such. From there, we could mount the Windows partition and start to copy data over. While we couldn’t verify the integrity of every file we recovered, we had a fully functional filesystem table, so we didn’t need to delve into the world of testrec or other file recovery / undelete tools that attempt to reconstruct the data heuristically. This meant that the data we recovered was statistically pretty likely of being uncorrupted.
Doing the data curly shuffle
Before starting the recovery operation, I had my friend check the filesystem of his external drive. The vendor pre-formatted the external disk as ExFAT, which is great if you’re moving data between different OSes, but not great if you’re on a system with no UPS and want to ensure your image is being written to a journaled filesystem. We could have used ntfs-3g to format the disk as NTFS, which would have let Windows read the drive after we recovered the data, but I felt like it would be safer to use a filesystem that Linux had native support for. I decided that we’d format the disk to ext4 in this case. xfs was another option, and might have been more performant, but performance wasn’t necessarily the characteristic I was optimizing for. ext4 was more than sufficient for this case.
However, since ext4 is not a natively supported filesystem on Windows, this meant that getting our extracted data back to the new drive was going to be a bit of a process. We could have used WSL with some fancy CLI args to pass the external NVMe drive to the Linux VM and mounted it there, but I figured that ntfs-3g inside of SystemRescue was going to be sufficient. After shutting the system down, swapping the failed NVMe drive for the replacement, and booting back into SystemRescue, we copied the recovered data to the root of the fresh Windows partition. I suggested we do that so that ntfs-3g wouldn’t have to try and deal with NTFS security, instead letting Windows handle that when my friend copied that data from the staging location to its final location in his User Profile.
Great success!
My friend has confirmed that all of his data was successfully recovered, which he was thankful for since he didn’t have any real backups (casual reminder to backup your important data!) This boiled down to pure luck, and the good engineering done by the SSD manufacturer in their firmware to put the drive in a read-only state when it exhausted all of it’s free cells. I’m a little frustrated, as the SMART data didn’t report that the SSD had reached anywhere near its lifecycle for TB’s written before failure, which to me suggests that the wear leveling algorithm might not have worked appropriately. That said, I didn’t get a full write count, so it is possible that the cells were exhausted from a high count of writes as well. All things being equal though, we couldn’t have been more stoked about the outcome.