Creating, deployment & customizing Linux VMs with Python & Chef – Part 2

  • Article

 

Through this two part series we will look at using Python to setup and create Linux virtual machines on Azure and customing the machines through Custom Script Extensions (Part 1) and Chef VM Extensions (Part 2).

This is part 2 series where we will use the same script (available on github) and deploy a MySQL Percona cluster using the Linux Chef VM Extension.

In part 1, the Python script creates a VM, and installs a Custom script extension. This extension downloads the custom Linux scripts from a blob storage and executes them. Logging the execution details. We created a script that downloads and installs the Chef client and then runs a MySQL Percona runbook.

The Linux Chef extension is a very specific extension, which automatically installs the chef client and runs a single or multiple run books for provisioning the server.

Please follow all the steps explained in Part 1, as everything remains the same except the script “python percona_cluster_setup.py” . Please use the “percona_cluster_setup_linuxchefclient.py” script instead.

The script is largely the same except that the Get_Resource_Extension() function returns a LinuxChefClient resource extension details instead of Linux VM Extension explained in Part 1.

 

Following is the function.

def Get_Res_Ext_List():

    chef_server_url = "https://api.opscode.com/organizations/paigeinc"

    validation_client_name = "paigeinc-validator"

    run_list = "role[percona]"

    # For multiple cookbooks : run_list = "recipe[cookbook1::recipe1],role[test_role1],recipe[cookbook2::recipe2],role[test_role2]
    ##Note- update following path with your local

    validation_key_file_path = "validation.pem"

 

    res_ext_ref = []

    res = ResourceExtensionReference()

    res.reference_name="LinuxChefClient"

    res.name = "LinuxChefClient"

    res.publisher = "Chef.Bootstrap.WindowsAzure"

    res.version = "11.16"

    res.label = "LinuxChefClient"

 

    # Sets the public configuration for extension

    res_chef_client_rb = ResourceExtensionParameterValue()

    res_chef_client_rb.key = "PublicParams"

    pub_config = {}

    pub_config['client_rb'] = "chef_server_url \t %s\nvalidation_client_name\t%s" % (json.dumps(chef_server_url), json.dumps(validation_client_name))

    pub_config['runlist'] = json.dumps(run_list)

    res_chef_client_rb.value = _encode_base64(json.dumps(pub_config))

    res_chef_client_rb.type = "Public"

 

    # Sets the Private configuration for extension

    res_chef_client_validation = ResourceExtensionParameterValue()

    res_chef_client_validation.key = "PrivateParams"

    validation_key_file = open(validation_key_file_path, "r+")

    pri_config = {}

    pri_config['validation_key'] = validation_key_file.read()

    res_chef_client_validation.value = _encode_base64(json.dumps(pri_config))

    res_chef_client_validation.type = "Private"

 

    res.resource_extension_parameter_values.resource_extension_parameter_values.append(res_chef_client_rb)

    res.resource_extension_parameter_values.resource_extension_parameter_values.append(res_chef_client_validation)

 

    res_ext_ref.append(res)

    return res_ext_ref

 

First we set the LinuxChefClient resource extension details, you can find this information if you use the Get-AzureVMExtension command on powershell/Azure CLI or the REST API. If you just need to get details of a specific extension you could use this link.  

Back to our Chef Client extension, following is the latest version at the time of writing of this blog.

    res.reference_name="LinuxChefClient"

    res.name = "LinuxChefClient"

    res.publisher = "Chef.Bootstrap.WindowsAzure"

    res.version = "11.16"

    res.label = "LinuxChefClient"

 

Since this is a Chef extension, all we need is to provide the rb & .pem files and the list of runbooks that need to be run.  

The client.rb file and the runbook list goes as “public” parameters, and the validation key goes as private. The parameters are passed as JSON in the format shown below:

 

pub_config = {}

    pub_config['client_rb'] = "chef_server_url \t %s\nvalidation_client_name\t%s" % (json.dumps(chef_server_url), json.dumps(validation_client_name))

    pub_config['runlist'] = json.dumps(run_list)

    res_chef_client_rb.value = _encode_base64(json.dumps(pub_config))

    res_chef_client_rb.type = "Public"

pri_config = {}

    pri_config['validation_key'] = validation_key_file.read()

    res_chef_client_validation.value = _encode_base64(json.dumps(pri_config))

    res_chef_client_validation.type = "Private"

 

Please ensure that the validation key file contents are base64 encoded. I have used the Azure Python SDK’s existing function _encode_bas64() , it could be a custom function that you would like to write.

Once, you run the script as explained in Part 1, give it some time till the VM’s provisioning is done. You will notice the “Linux Chef Client” extension installed in the extensions installed section of the portal’s dashboard.

 

Troubleshooting

While the vm_getstatus.py script can be used to get status of the request submission, sometimes the startup script might fail to run during the provisioning state. You can monitor the VM creation progress on the portal too.

Once the VM reaches the running status, you can logon to the VM and view the following logs in case of any errors in VM provisioning.

Following log file logs all the VM provisioning activity by Azure VM agent.

/var/log/waagent.log

 

Following log logs running of the extension script.

 

 /var/log/azure/Microsoft.OSTCExtensions.LinuxChefClient /1.1/extension.log