This blog post is from Troy Larson, a true leader in the field of digital forensics and engineer on the Microsoft Security Response Center’s Azure team. Troy is focused on building and measuring forensic capabilities in the Azure platform, and executing advanced security investigations. Troy has been on the front lines of critical cases for Microsoft for over 10 years, creator of the Windows Forensic Environment toolkit and is a frequent speaker on Windows and Office incident response and forensics. Troy received his undergraduate and law degrees from the University of California at Berkeley, and has been working in the field of digital forensics since the late 90s.
Some of the very things that make the cloud appealing to enterprise IT may worry the security responder, especially with respect to digital forensics. In the cloud, computers are virtual and potentially spread across datacenters located in geographically dispersed locations. Where enterprise IT sees availability and scalability, the security responder may see a computer environment that is intangible and untouchable. For the security responder, intangible and untouchable raise a fundamental question: Is digital forensics possible when the enterprise moves into Azure?
The answer is emphatically yes. Digital forensics not only survives when enterprise IT moves to Azure, it can thrive. The challenges of computer forensics in Azure are much the same as those of traditional on-premises enterprise forensics. Over the past decade, enterprise level forensics has adapted to world wide networks, rapid shifts in technology, and computers in distant offices or data centers. Azure makes few, if any, additional technical demands. What currently works for the enterprise should transfer to Azure.
What can the security responder do with a “computer” in Azure? A virtual machine in Azure presents essentially the same technical issues around evidence collection as a computer in geographically remote data center or office. It has been a common and standard practice for some time in the enterprise to collect evidence from remote systems, while they are running, over the network. Those same tools and procedures should work for Azure. The computers may be virtual, as are the hard drives, but the file systems, event logs, data files and everything the security responder normally would want to collect and examine, remain the same. If one can do forensics on the enterprises’ Windows or Linux systems now, one can do forensics on those systems when they move into Azure.
Azure virtual machines run on top of Microsoft Hyper-V technology. The actual Host OS-Guest VM architecture in Azure, while interesting, is not critical to forensics. Each Azure VM sees itself as an independent computer with defined boundaries between itself and the Host OS. Although a VM may share the same physical hardware as several other VMs, the VMs are not aware of each other or of the Host OS. The security responder, working with, or collecting from, a VM has no access to the Host OS or the other VMs.
Azure compute comes in two distinct service offerings. Azure Platform as a Service, or PaaS, provides prebuilt Windows servers that can run customer supplied applications. Azure Infrastructure as a Service, or IaaS, provides the hardware and virtualization environment on which the customer can run its own Windows or Linux systems and applications. Both PaaS and IaaS VMs use Microsoft virtual hard drives, or VHDs, for VM local storage.
There are some insignificant differences in how the VHDs are arranged between PaaS and IaaS, with the OS disk mounted as the C: drive in IaaS, but mounted as the D: drive in PaaS. PaaS systems typically involve three drives, an OS, Application, and “Resource” drive. The OS drive contains an OS installation prebuilt and configured by Microsoft, and the Application drive will contain the customer’s application code and settings. The Resource drive is used for system temporary storage and is where the page file will be located for Windows VMs. In PaaS, all VHDs reside on the local hardware with the VM.
IaaS VMs will have an OS and a Resource drive. The resource drive in IaaS has the same function as in PaaS, temporary system storage. IaaS VMs may also have one or more Data drives. Data drives are VHDs that a customer can attach to a VM as needed for data storage.
IaaS is notably different from PaaS in that both the OS and any Data drives of IaaS VMs are replicated to Azure blob storage. As we shall see, this is something that the security responder can leverage to capture drive images.
Collecting Evidence from Azure IaaS VMs
Data in Execution
Forensics begins with evidence collection. Security responders typically want to collect at least some amount of “live data” from suspect computers for triage — for example, live data about current processes, logged on users, network activity, among other things. Depending on nature of the incident and the capability of the security response team, security responders will often also want to collect complete copies of the hard drives and a dump of the physical memory. The good news is that any data that can be collected from Windows machines in a physical environment can be similarly collected from a Windows system in Azure. There is no reason to think that this situation will be any different for Linux VMs hosted in Azure.
One of the capabilities the security responder has in Azure, at least for IaaS VMs currently, is the ability to attach Data drives as needed to a running VM, without rebooting. This allows the security responder to launch tools from, and copy output files to, the newly attached Data drive, minimizing the impact of evidence collection on the existing VHDs. These operations may be performed through the Azure management portal, or programmatically via the Azure API. In the screen shot below, for example, we have attached a new data drive from which we are running LiveKD to create a full memory dump of the VM. Of course, we are writing the dump file to the newly attached Data drive.
Attaching a Data disk works well for collecting a variety of live data. In the screen shot below we have run a rather comprehensive data gathering script, using a number of Windows command line tools, as well as other standard tools of the trade, such as utilities from Sysinternals and NirSoft. All of these utilities work as they would on a similarly configured physical system.
Data at Rest
Finally, whole drives can be copied from within the VM, just as they might be collected from a remote physical machine. In the screenshot below, we are imaging the C: (OS) drive of a running IaaS VM to a Data drive that we attached to the system to capture disk images of the existing drives.
Although we have not yet tested remote imaging tools, such as F-Response, in Azure ourselves, these sorts of collection tools could be used for remote drive or memory capture from Azure VMs. This approach to evidence collection is fairly common in the enterprise, but it would require the VMs to have been configured with the proper networking endpoints to allow access for the remote collection software. Copying memory or existing drives to a special Data drive attached for evidence collection, as described above, may allow for the fastest evidence collection in most situations, but the security responder has options. As with enterprise level forensics, in general, there are many ways to accomplish any task in Azure.
IaaS VMs provide yet another drive capture capability—one that physical systems cannot provide. As mentioned above, the IaaS OS and Data drives are replicated to Azure blob storage. They can also be copied directly from Azure blob storage for use as evidence. While there are a number of ways to copy items from Azure blob storage, the capability to copy the OS or Data drives of an IaaS VM is even provided in the Azure Portal, as shown below.
Currently, only IaaS OS and Data drive can be copied from Azure Storage. IaaS Resource drives, and all PaaS drives, must be copied from the VM.
Capturing copies of drives from storage can provide the security responder the OS and Data VHDs for a VM without touching or impacting the VM in any way. However, stopping the VM before copying its VHDs from Azure blob storage will more likely render a copy of a VHD with its file system or files in a consistent state (since the VM will continue to write to a its drives while it remains running.) While imperfections in the file system will usually present no problem for common forensics tools in examining a VHD, a VHD with a less than perfect file system may not boot, and may not mount properly as a disk, if that is a goal.
Azure VMs and Host OS’s have TRIM enabled by default, as do all current version of Windows. TRIM is used to notify storage media when files have been deleted, so that space allocated to the deleted files in the media can be more efficiently freed and reused. Originally designed to allow solid state media to optimize available free space, TRIM can cause deleted files to disappear rapidly from media—any media. (In practice, forensics investigators nonetheless often find a good amount of deleted material on media despite TRIM, even on solid state drives.) In Azure, TRIM appears to be very effective in quickly removing unallocated data from the VHDs stored in Azure blob storage. VHDs captured from blob storage will not likely contain much, if any, deleted data in the clusters of files that have been deleted. Metadata for deleted files, however, will still be retained in the file system, just as it would be for file systems on physical hard drives.
The effect of TRIM is clearly evident in the screen shot below. The file system metadata remains, providing evidence of the names, sizes, and dates of deleted files. The actual file data, however, has been removed from the VHD by virtue of TRIM, leaving nothing but zeros in the clusters that had been allocated to the files before they were deleted.
In our experience, the VHDs stored locally with the VM appear to contain the usually expected amount of deleted material even though the Azure blob stored versions of the VHDs do not. In other words, drives copied from the VM will have deleted material that will not exist in a VHD copied from blob storage. If deleted data are important, turn off TRIM before there is an incident, or collect copies of the drives from the VM.
Rather than proceeding in strict order of volatility prescribed by RFC 3227, Guidelines for Evidence Collection and Archiving, for example, the security responder in Azure can preserve copies of an IaaS VM’s OS and Data drives from Azure Storage before running any programs on the VM to capture memory or other dynamic or live data. Collecting copies of disks early in a security incident can be quite important, since some data on drives can be more volatile than some memory or other live data. For example, the metadata of deleted files can be easily lost—every new file created on an NTFS volume causes the loss of file system metadata for a deleted file.
After collecting memory or other live data, the security responder can still copy the drives from the VM. Collecting the VHDs from Azure blob storage before running any collection procedure on the VM will preserve the file system metadata of deleted files, as well as the state of files that might be altered by other collecting activity (such as registry or event log files). Collecting the same drives from the VM later will provide access to any deleted data that TRIM might have removed from the Azure blob stored VHDs.
Examining Evidence from Azure VMs
Many common forensics tools will natively parse VHDs just as they will traditional “forensic” disk images such as dd or the ubiquitous “Encase” image file format (*.e**). In other words, VHDs can serve as forensic disk images, directly, without the need for format conversion or additional processing. Like traditional forensic disk images, VHDs will contain whatever is on disk, including deleted data (TRIM permitting, of course). The screen capture below, for example, shows an IaaS OS VHD as parsed and displayed by an industry standard forensics tool. But for the obvious “WindowsAzure” directory at the root of the drive and some special Azure only binaries and event logs, this disk image would be indistinguishable from a disk image from a similarly configured Windows machine running on physical hardware.
As mentioned at the start of this post, once the security responder gets to the point of examining memory dumps, volumes, file systems, or files, the distinction between virtual systems and physical systems disappears. The standard tools and procedures for Windows or Linux forensics, be it disk, memory, or file analysis, will be equally effective for data coming from virtual or physical machines. Sound forensics investigations will involve examining the same sort of evidentiary artifacts, in the same way, with the same tools and procedures, to resolve the same sorts of issues, regardless of the source of the evidence.
If there is any challenge at all to working within the cloud, it is in collecting evidence. We hope we have been able to briefly show you that collecting evidence from Azure VMs is not significantly different or more difficult from current standard practices for collecting evidence from remote systems in an enterprise IT environment. We’ll surely see some tremendous innovation in this space as the cloud evolves, and forensic practices follow. If you can do it in the enterprise, you can do it in Azure.