Best Practices to protect your Azure deployment against “cloud drive-by” attacks

  • Article

As a follow-up to David Cross‘s "Cloud Security as a Shared Responsibility" blog, I’d like to share the perspective of the Microsoft Security Response Center (MSRC). The MSRC executes security response activities for threats that target Azure’s internal infrastructure and customer deployments. This experience provides a unique perspective on how cloud providers and customers are targeted by attackers, and in the spirit of transparency and continuous improvement, I would like to share some details about the incidents we've seen and best practices for Azure customers.

In this blog post, I’ll describe incidents that arose from a style of attack that we refer to as “cloud drive by attacks.”  Take note that we are coining a new type of attack – don’t confused this with the category of attacks referred to as “drive-by downloads.”

Aligning to Bruce Schneier’s idea that attackers are categorized "along two axes: skill and focus," Azure is subject to a high degree of "Internet radiation" produced by these “cloud drive by” attacks that originate from low skilled, unfocused attackers.  These attacks target IP ranges associated with cloud providers and probe for well-known software and configuration vulnerabilities. These attacks continue because they are successful and cheap to execute. 

Azure is designed to be flexible and powerful. Microsoft is continually improving the preventive, detective, and response capabilities “of the cloud", and as an Azure customer, it’s important to understand how to maintain the security of your resources "in the cloud." 

Let’s take a look at a few examples of  actual incidents. For each attack, I’ll suggest some best practices and considerations to harden your Azure deployments against “cloud drive-by” attacks.

Learning from Azure Security Incidents

For each of the following incidents, I’m going to use the fictional company “Contoso”, and refer to the administrators as “Ben” and “Jeff” to protect the identities of the organizations that were victims of each breach. 

These incidents were detected due to activity that originated from a Virtual Machine (VM) after it was compromised by attackers.  For each incident, the customer and Microsoft worked jointly to investigate the scope of compromise and take the appropriate steps to triage, contain, and mitigate the attack.  If you are an Azure customer and need assistance, you can always contact Microsoft Support directly from the Azure Management Portal. 

“Cloud drive-by” attacks are not unique to Azure, and all cloud providers (especially those that offer IaaS services) experience similar activity. We know this because Azure is frequently attacked by virtual machines hosted by other cloud providers who were likely compromised by “cloud drive-by” attacks. 

As I discuss the corrective actions that we take, remember that our work is governed by the Azure Terms of Service, and we ensure that the actions that are taken align to our privacy and service commitments.

The case of the "Fast and Furious SSH Attacks"

Contoso’s administrator, Ben, deployed new Linux based IaaS VMs in Azure  and set a strong initial password on the “root” accounts in his VMs. He also enabled SSH on the default port (22/tcp) as an endpoint to enable remote access.

A few weeks later, Jeff (who is responsible for infrastructure monitoring and new to the cloud) installs Nagios to enable monitoring of the infrastructure. During the install process, Jeff specifies a predictable username of “nagios” with a weak default password of “p@ssword123!” 

Jeff felt comfortable setting an easy to remember password, since he felt Nagios was isolated to trusted networks.  What Jeff did not realize is that the “nagios” user is now a valid user account on the local machine that has remote access privilege, and therefore permitting login rights through SSH. 

Approximately 8 hours later, an attacker discovers the open endpoint and launches a "brute force attack."  The attacker successfully authenticates after trying approximately 200 username & password combinations. With this foothold, the attacker rapidly installed malware that provided the capability to remotely launch DDoS attacks.

Fortunately, the nagios account did not offer administrative root access, and the attacker was not able to escalate privileges to root (despite trying) before the attack was uncovered. Microsoft detected a DDoS attack that originated from the VM and escalated the findings to the customer, who swiftly investigated the breach and took corrective action.

Given enough time, the attackers would have figured out a way to elevated their privilege on the compromised machine, and explore opportunities to move laterally and attack other virtual machines. During the forensic evaluation, secrets were observed in the .bash_history file of the root user and secrets and keys that would have enabled the attacker to pivot to the customer’s internal network.

Best Practices:

  • Reduce the attack surface - SSH was exposed and not locked down.
    • Use SSH Key Authentication: Key based authentication hardens the SSH attack surface.
    • Access Control Lists (ACL) or Network Security Group (NSG) rules on the SSH Endpoint: ACLs or NSGs can be defined within Azure or within the Host Firewall.
    • Limit who can establish remote connections: the sshd_config file supports two important verbs (Allowusers and Denyusers). In this case, Ben (who built the IaaS VMs) can explicitly scope who can connect (and reject others) by defining "AllowUsers root").
  • Assume every password is critical, and use commercial or open-source solutions to securely generate, secure, and share passwords.

The Case of "Too Much Information"

Contoso decided to migrate services hosted on end-of-life, on-premise server hardware to Azure as IaaS roles, and Ben assists in the migration by bridging Contoso’s on-premises network to the cloud.  This includes migrating IIS Intranet sites hosted on Azure IaaS VMs (and enabling SSO) and an HTTPS endpoint that is accessible from the Internet. During the mitigation, Ben was juggling many tasks and did not complete the configuration of the IIS server to expose it on an Internet-accessible FQDN with proper authentication.

During his response to an unrelated application issue, Ben’s colleague Jeff noticed unexpected page requests from unusual IP entries in the IIS logs of the servers hosting the IIS intranet site.  Jeff quickly understood the external exposure:  attackers were connecting to the IIS endpoint by using common reconnaissance tools.  These tools generically scanned the environment by creating HTTPS requests with the IP addresses they were targeting (e.g. https://10.0.0.1/admin), rather than the host-headers (e.g. https://contosointranet/home.aspx) which in this case were required to access the site. Therefore, the approach that the attackers executed was unable to successfully access data -- but, make no mistake -- Contoso got lucky.

Misconfigurations can happen, and proper detection should spot the unusual access patterns that result when they are discovered by attackers.  In cloud migration scenarios, it‘s best practice to:

The Case of “Searching for a Foothold"

Contoso has deployed a large number of VMs hosting a popular open-source software package that enables programmatic search.  Shortly thereafter, a significant vulnerability was discovered in the package that enabled attackers to perform unauthenticated, remote command execution. Given the power of the vulnerability, it was quickly weaponized by attackers to compromise virtual machines.

This type of vulnerability isn’t exclusive to open-source projects. The Microsoft Security Response Center has responded to similar vulnerabilities within Microsoft products, and IT professionals have accepted the reality that all software has security vulnerabilities that need to be periodically updated.

The cloud is not immune to the need for updates. However, the responsibilities have shifted depending on the service. For example, Microsoft patches software-as-a-service (SaaS) products such as Exchange and SharePoint if you are an Office 365 customer. For platform-as-a-service products, Microsoft patches and maintains SQL Azure.

In this particular case, the open-source software package was deployed on infrastructure-as-a-service (IaaS) VMs that were built by Contoso. As a IaaS deployment, Microsoft does not apply updates. Instead, Contoso‘s team is responsible for the maintenance of the VM and all software installed on it.  They were not yet scanning their cloud hosted resources with the same rigor that they’ve grown accustomed to with their on-premises servers.

Attackers scanned Azure’s public IP address space, probed hosts for the vulnerability, and obtained a foothold on Contoso’s virtual machine. Shortly thereafter, the attacker uploaded malware to enable remote attacks against other hosts on the Internet. The attacker triggered the server to start an attack, and the compromised VM was quickly detected and mitigated.

Best practices:

  • Scan the resources you own with vulnerability management solutions such as Tinfoil. Our partners in this space are rapidly innovating new features that enable visibility to cloud-based services. One note of caution: Please respect your cloud neighbors -- tune your policies carefully, and only scan your own deployments!
  • Monitor vulnerabilities for software that is deployed on VMs, and prioritize patching software that can be directly or indirectly accessed from the Internet. Unless you invest time to understand the vulnerability, these is often difficult to distinguish how dependencies (such as bash and Java) are called across the system. The safest path is to install all patches quickly as they are made available by deploying Microsoft Update or using technologies from the Azure marketplace such as Shavlik Protect.
  • Continue to be vigilant - practice secure development and apply the same level of focus to common OWASP vulnerabilities on your cloud infrastructure as you would your on-premise deployments.

Getting Better

As the industry moves to the Internet of Things, the scale of the cloud will grow in response to the larger client base. The Microsoft Security Response Center will continue to adapt as the cloud landscape and threats evolve. However, the best defense is a strong offense and these tips will help you protect, detect, and respond to “cloud drive-by“ attacks.

Craig Nelson @ath0nocarrier
Principal Software Engineer, Microsoft Security Response Center | Azure