Provisioning the Azure Kubernetes infrastructure with a declarative template

This article is the third one about a solution we have built with my friend Hervé Leclerc (Alter Way CTO) in order to automate a Kubernetes Cluster deployment on Azure:

·      Kubernetes and Microsoft Azure

· Programmatic scripting of Kubernetes deployment

· Provisioning the Azure Kubernetes infrastructure with a declarative template

· Configuring the Azure Kubernetes infrastructure with Bash scripts and Ansible tasks

· Automating the Kubernetes UI dashboard configuration

· Using Kubernetes…

So first let’s explain the way we have automated Kubernetes Azure provisioning through ARM Templates.

Overview of the solution

Kubernetes infrastructure is made up of several components – virtual machine, storage account, and virtual network which are the related and interdependent parts of a single entity.

The targeted topology (generated with https://armviz.io) should be the following one:

Azure Resource Manager enables to deploy, manage these resources in a single, coordinated operation and monitor them as a group rather than handling them individually. With Azure Resource Manager, it’s possible to define dependencies between these resources so they are deployed in the correct order. Moreover, Azure Resource Manager provides tagging features to help manage these resources after deployment, auditing and security: Role-Based Access Control (RBAC) is natively integrated into the management platform.

Azure Resource Manager Template

Azure Resource Manager offers the ability to build templates and easily leverage them to deploy and manage resources on Azure. It’s a very efficient way to deploy a cluster of several virtual machines.

An Azure Resource Manager Template enables to deploy and manage these resources together by using a JSON description of the resources and associated deployment parameters. The template consists of JSON and expressions which you can use to construct values for your deployment. The template is organized in three sections:

· In the parameters section of the template, you specify which values you can input when deploying the resources. These parameter values enable you to customize the deployment by providing values that are tailored for your particular environment.

· In the variables section, you construct values that can be used throughout your template. Typically, these variables will be based on values provided from the parameters. The objective is to simplify your template by reducing complex expressions.

· In the resources section, you define the resources to deploy or update using types defined in resource providers. When you deploy a template, you must specify an API version to use for creating each resource. The API version corresponds to a version of REST API operations that are released by the resource provider. As a resource provider enables new features, it will release a new version of the REST API. Therefore, the version of the API you specify in your template affects which properties you can specify in the template. In general, you will want to select the most recent API version when creating new templates. For existing templates, you can decide whether you want to continue using an earlier API version, or update your template for the latest version to take advantage of new features.

For example, the following extract corresponds to the provisioning of the network resource.

Linked ARM template

From within one Azure Resource Manager template, you can link to another template which enables you to decompose your deployment into a set of targeted, purpose-specific templates. This facilitates the reading and maintenance of the template.

In our solution, provisioning of a Kubernetes cluster built upon a configurable number of etcd nodes and Kubernetes nodes (master and minions) is implemented with a linked Azure Resource Manager Template “azuredeploy.json” combined with its parameters file “azuredeploy.parameters.json”.

Note : In the current armviz.io version, the previously displayed diagram is the one related to the global “azuredeploy.json” template [https://github.com/DXFrance/AzureKubernetes/blob/master/Kubernetes-Ansible-Centos-Azure/azuredeploy.json], not the linked one, which will be the one presented in this article [https://github.com/DXFrance/AzureKubernetes/blob/master/AzureKubernetes/Templates/VM-Cluster/azuredeploy.json].

The declaration for the usage of a linked template is similar to the declaration of a native Azure resource:

The Resource Manager service must be able to access the linked templates, so we have simply used GitHub as a repository for our linked templates : https://raw.githubusercontent.com/DXFrance/AzureKubernetes/master/AzureKubernetes/Templates/VM-Cluster

The main template “azuredeploy.json” integrates seven resources which are also json templates. Three of these resources are shared by all the nodes and are related to the provisioning of availability sets and storage (“SharedAvailabilitySetAndStorage”), network (“SharedNetwork”, “SharedPIPandLoadBalancer”). The network interfaces, the virtual machines and their extension are specifically deployed and configured by type of cluster nodes (“EtcdNodes”,”KubeMasterNodes”, “KubeMinionNodes”, “ConfigAnsibleController”).

ARM template extension

Configuration is requested from the ARM Template and this is done through the provisioning of an extension. For example, the following is an extract from the “KubeMasterNodes.json” resource template.

It uses the CustomScriptForLinux extension in order to configure the master nodes with a bash script “config-node.sh” which will also be executed on the other minion or etcd nodes. I will present more precisely how this script is implemented in my next article Configuring the Azure Kubernetes infrastructure with Bash scripts and Ansible tasks.

Implementing the template

A good JSON editor, such as Visual Studio Code can simplify the task of implementing templates, but the more efficient one would certainly be Visual Studio which brings several complementary features. First, it enables the developer to have a better view of his parameters, variables and resources by presenting the different sections of the template and enabling the navigation between the elements declared in these sections. Moreover, it includes a wizard that offers the ability to directly add resource to the current template, which enables you to generate the json script on the fly.

Another useful tool is Azure Resource Explorer, a site where you can discover the Azure Resource Management APIs, get API documentation and make actual API calls directly in your own subscription. It’s all done directly at the JSON level, so it always stays very close to the wire, and you can control and see everything that it’s doing.

Launching and monitoring the infrastructure deployment

The template may be directly deployed from within Visual Studio 2015 with a simple click…

You can monitor the provision deployment logs in the output of Visual Studio.

Another way to deploy it and monitor the deployment is using a PowerShell or azure-cli scripts. Here is the PowerShell script: AzureKubernetes/Scripts/ Deploy-Kubernetes-Ansible-Centos-Azure.ps1 that may be used for this purpose.

You may also provision the infrastructure by directly editing the template from the portal.

And of course, independently of the way you deploy the template you may monitor the deployment directly from the portal. And that’s it. The following screen capture displays the result of the deployment of Kubernetes cluster resources through the linked template. It takes about 30 minutes to have the solution fully deployed.

Next…

Provisioning of the whole topology is not so difficult when you get used to it. What’s may be more complex is how to automate the configuration through extension, especially when there are dependencies between them and that’s what I will explain in my next article.

Stay tuned!