Azure IaaS Audit Solution
I have been working with a customer for the last two years that has their eCommerce platform deployed within Azure. Because of that, their application has to be PCI compliant which requires that they perform an annual audit of the system. That along with my many years working with customers within the Federal Government, brought up a problem that I did not see as easily solvable within the scope of the Azure Portal today and that is how best to gather information about a particular system within Azure for the purposes of an Audit.
The customer that I mentioned at the top actually keeps a manually entered Excel spreadsheet of all components of their IaaS architecture and their relevant metadata. This is fine if you only have one system to worry about and not a lot of changes are being made to it. However, most customers do not fall into that category, they have many systems, each with their own architectures and many of them may need to be audited on a regular basis.
What is the Azure IaaS Audit Solution?
So with the above as a backdrop, I had an idea. Could I put together some kind of solution that customers could easily consume that would produce the audit data that they are looking for in an automated way? That was actually a pretty easy question to answer, because there are many ways to get the data, it is just a question of understanding all of the relationships within Azure so that the correct metadata is associated with the right tier of the system's architecture.
The harder part of the question was actually deciding on what kind of solution I should build that would not require a tremendous cost burden on the customer but still give them what they are looking for. I will go into more detail about how I built and you can deploy the solution in the next section.
What I have built so far and plan to evolving over time is a solution that can be deployed within Azure and gather all of the relevant data. The solution will perform the following steps:
Login to the Azure Framework using an Azure SDK
Loop through all of the Resource Groups within a given Subscription
Retrieve all of the relevant pieces of information for a given architecture and place them into a JSON document.
-
- VMs
- Networks
- NICs
- Disks
- Load Balancers
- NSGs
ASSUMPTION: This does assume that a given system is self-contained within a resource group. As this is one of the recommended ways to organize applications and resources within Azure, I thought that this would be a good place to start.
Once the data is available, there are many ways to format it in a readable way that can be used as part of an audit. I hope to be providing a basic UI that can sit on top of the JSON data in the near future.
How is the Solution Put Together and Deployed?
As I mentioned above, I do plan on building onto what I have already developed, but for now, I have put together a set of C# based Azure Functions which utilize the Azure Fluent SDK. Links to information about each of these things can be found below as well as a link to my solution on Github.
A walk-through of the solution is below, in addition there is a tremendous amount of code commenting to help anyone understand what I did:
1.) Both functions were written using the new Azure Functions extension for Visual Studio 2017 Preview 3. This allowed me to include all of the needed external libraries like the Azure Fluent SDK as well as the Code into a Compiled DLL. This DLL can then be deployed to Azure through Visual Studio or other CD means.
2.) These functions do require a valid Subscription user, so I have provided a link to documentation on how to create a Service Principal User that can be assigned to the Reader role within the Subscription.
3.) In addition to the requirement of the Service Principal, the Azure Fluent SDK will need Subscription ID and AAD Tenant ID, these items combined with the Service Principal App ID and App Key(Password) are what is used to authenticate properly.
4.) Each of the pieces mentioned above will need to be created as App Settings within the App Service being used to host your Azure Functions. These App Settings will then be pulled by both Functions and use the information inside. Information about the name for each App Setting can be found within the README on the Github repo.
5.) Function 1 (IaaSAudit-RGtoQueue-Timer) is a simple Timer based function that loops through each Resource Group within the provided Subscription and puts the name of the Resource Group on a Queue. The Storage Account Connection string for the Queue is also stored as an App Setting within the App Service for the Functions.
6.) Function 2 (IaaAudit-AuditReport-QueueTrigger) is where the majority of the work is being done. For each Resource Group name that is added to the Queue in the first Function, this function will build a JSON document to represent all of the IaaS resources within.
7.) Once the JSON document has been fully created, the Function pushes the document to a configured Cosmos DB database and collection.
8.) I tested everything locally using the local.settings.json file which can take the place of the App Settings within the App Service
In addition to the explanation above, you will also find a great amount of documentation in the form of code commenting throughout both functions.
What's to Come?
As I mentioned at the very top, this is just a starting point. It does provide a tremendous amount of information in one single JSON document per Resource Group, but it is up to each customer at the moment to provide their own interface into that document. So with that, I am hoping to do the following in the upcoming months:
Provide the same set of Functions in a different language, specifically Node JS
Add a separate document containing all of the RBAC assignments for the same subscription
A Web application which can do the following:
-
- Show the available documents in a Tree format
- Provide an Excel export of each document
- Provide for more than one Subscription to be applied to both the Functions and the UI
Within the Functions, add additional other potential IaaS/PaaS/Hybrid services
-
- App Service Environment
- Azure SQL
- Azure SQL DW
- HD Insight
One thing to keep in mind, I do expect that the Azure product teams may be providing some kind of similar feature in the future, but in the meantime, this is a sure way to get the information that you are looking for and do with it whatever you may need. If anyone should have any suggestions or find any issues, please don't hesitate to file an issue with the Github Repo. As I create new projects, I will either update this Blog post or create a new one to announce it and describe it.