Please note that this blog was updated on 2/8/2019 at 8:30 PM PST. This was because we identified that the update was hitting all Linux clusters, regardless of whether or not new nodes had been added. As a result, if you have a Linux cluster up today, you must update your ARM template as detailed below. If you had previously updated your ARM template based on the prior iteration of this post, please update it again with script FileURI seen below.
On 2/6, a security update to the Ubuntu xenial rssh package was made available via public Ubuntu package sources. The updated version (2.3.4-4+deb8u1build0.16.04.1) introduced a fault in Service Fabric's ImageBuilder process. This process is used when provisioning applications and the cluster runtime in Service Fabric. Ubuntu based Service Fabric nodes will be unable to do core management operations unless the dependent package is downgraded to the previous version (2.3.4-4).
This will impact all Azure Service Fabric Linux clusters .
Here is a script to help mitigate the issue that needs to be applied as a Custom Script Extension to your ARM (Azure Resource Manager) template, followed by an upgrade to apply the change to your cluster. Instructions for doing this are noted below. You will need to do this for existing as well as new clusters until we work to fix the issue on our end. Please watch this space for future updates on this.
Instructions for applying the fix:
Update your ARM template
In your ARM template that represents the latest configuration of your deployment, edit the extensions section to include a ‘CustomScriptExtension’ that contains the script we are providing to help mitigate the issue. Right below the definition of the ‘ServiceFabricVMExtension,’ add the following JSON snippet:
Update your cluster deployment
Once your ARM template is ready, deploy the template using the corresponding PowerShell or Azure CLI command -
az group deployment create -g MyResourceGroup --template-file azuredeploy.json --parameters azuredeploy.parameters.json
New-AzureRmResourceGroupDeployment -ResourceGroupName MyResourceGroup -TemplateFile azuredeploy.json -TemplateParameterFile azuredeploy.parameters.json
Additional information for customers leveraging POA
Customers leveraging POA on Linux should update their ApprovedPatches list with 'rssh = 2.3-4.4' to hold the package back. https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-patch-orchestration-application-linux#configure-the-app
Please reach out to us if you need specific assistance with this issue through Azure Portal Help. In addition, here are your general support options for Service Fabric: https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-support#report-production-issues-or-request-paid-support-for-azure. We will keep you appraised of any updates on this matter through the blog. Thank you!