Windows Azure : There is not enough space on the disk - description and handling error
If you have your application running on Windows Azure, it is possible that your:
1. 1. Web Role is keep Recycling after successfully running for some amount of time
2. 2. You are not getting anything out from your diagnostics log after sometime
So if you have you application based on Windows Azure SDK 1.3, you can login to your VM and investigate the issue. Once you login to your VM, I would suggest looking:
1. 1. System UpTime from Task Manager Window - This will tell you how long does not VM running so once you look for event logs, you can get an idea when this error started to check how fast your Local Storage is getting filled
2. 2, Application Event Log - For your IIS and ASP.NET based application, this is the first place to start your investigation
In the event log you may see the following Error:
Log Name: Application Source: ASP.NET 2.0.50727.0 Date: 12/20/2010 9:57:12 PM Event ID: 1334 Task Category: None Level: Error Keywords: Classic User: N/A Computer: RD00155D322252 Description: An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/1/ROOT Process ID: 1184 Exception: System.InvalidOperationException
Message: There was an error generating the XML document.
InnerException: System.IO.IOException
Message: There is not enough space on the disk. |
Now when you explore your VM you will see all 3 drives on VM have significant space on it. On Windows Azure VM you will see 3 drive as C: D: and E: and here is a little explanation for these 3 drives:
1. Drive E: -> this drive holds your Windows Azure Service package which you have uploaded or published to Azure Portal. When your VM starts the package is decrypted and added a drive E:\ to the VM
2. Drive D: -> This drive holds the Windows Server 2008 R2 OS.
3. Drive C: -> This drive can be considered as the working drive where most of the Windows Azure run time data is available.
So a few of you, may have some confusion about the above error because all 3 drives will have lots of space but event log will still show the error. So here is the explanation:
When you define Local Storage in your Windows Azure Application, a new folder will be added as below in your drive C: on Azure VM:
C:\Resources\directory\[Your_deploymentID].[your_webrolename]_[Role_Number]\<LocalStorage_name_Dir>
This is the folder which will have a size cap on it.
The error you are getting is coming out from your “Local Storage” folder size limitation. So above error is directly related with your Local Storage folder. Windows Azure OS limits the size of your “Local Storage” folder depend on the size you define in your ServiceDefinition file.
For example the local Storage name is “TomcatLocation” and the size defined for “TomcatLocation” in the below CSDEF file is 2GB:
<?xml version="1.0" encoding="utf-8"?> <ServiceDefinition name="TomcatRole" xmlns="https://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition"> <WorkerRole name="TomcatWorKerRole" enableNativeCodeExecution="true"> <ConfigurationSettings> <Setting name="DiagnosticsConnectionString" /> </ConfigurationSettings> <Endpoints> <InputEndpoint name="Tomcat" port="80" protocol="tcp" /> </Endpoints> <LocalResources> <LocalStorage cleanOnRoleRecycle="false" name="TomcatLocation" sizeInMB="2048" /> </LocalResources> </WorkerRole> </ServiceDefinition> |
So when the folder will hit the defined limit (i.e. 2048MB based on above CSDEF) the above described error start occurring.
To solve this problem either you increase the size of “Local Storage” in the CSDEF file or write file system API based code to monitor the “Local Storage” folder and delete it when it is reaching a certain threshold. We do not provide any mechanism or API within Windows Azure SDK to maintain the “Local Storage” setting. So if you want to keep using it then please either increase the size of add more code in your Azure service using file system API to keep checking the folder size and delete it when it hits the maximum.
Once thing to remember is that when role starts, the local Storage folder deleted and then recreated for your service so that is the only time it is deleted by the system and any other time, it is service responsibility to manage it.