Windows Azure: Roles Repetitively Starved of Ports


You just upgraded the OS family or in case you have auto upgrade on for Windows Azure there is a chance that you might see the following errors in case you are using Storage Client and SQL Azure .Net APIs. The other symptom could be sluggishness/slowness/delay in any storage calls like blobs/queues/tables or even SQL Azure calls in case the application uses both Windows storage and SQL Azure. It could also afflict you in case Windows Azure diagnostics are being run as Windows Azure Diagnostics write to Storage.   

Rebooting the VM resolves the issue only for it to come back after a certain time.

Storage 

Server Error in ‘/’ Application.

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified

Stack Trace:

[SocketException (0x2747): An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full]

Systen. Net. Sockets. Socket. DOBi nd(EndPoint endPoi ntSnapshot, SocketAddress socketAddress) t7985914

System. Net. Sockets. Socket. Internal Bi nd(EndPoint localEP) +243

Systen.Net.Sockets.Socket.BeginConnectEx(EndPoint remoteEP, Boolean flowcontext, AsyncCallback callback, Object state) +307

System. Net. Sockets. Socket. Unsafeßeqi nConnect(EndPoint remoteEP, AsyncCal lback callback, Object state) t71

System.Net.ServicePoint.ConnectSocketlnternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, lAsyncResult asyncResult, In

[WebException: Unable to connect to the remote server]

Microsoft.WindowsAzure.StorageClient.Tasks.Task’1.get_Result() 

Microsoft.WindowsAzure.StorageClient.Tasks.Task’l.ExecuteAndwait() 

Microsoft.WindowsAzure. StorageClient. <LazytnumerateSegmented>d_01.MoveNext()

System.Linq.Enumerable.Count(IEnumerable’1 source) 

 

SQL Server

Server Error in ‘I’ Application. 
A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where d originated in the code.
Exception Details: System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection SQL Server)
Source Error:
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.
Stack Trace:
[SqlException (0x80131904): A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server 
System.Data.ProviderBase.DbConnectionPool .GetConnection(DbConnection owningObject) +706
System.Data.ProviderBase.DbConnectionFactory.CetConnection(DbConnection owningConnection) +89
System.Data.Providerßase.DbConnectionClosed.OpenConnection(DbConnection outerConnection, DbConnectionFactory connectionFactory) +6425518
System.Data.SqlClient.SqlConnection.Open() +300
System.Data.EntityClient.EntityConnection.OpenStoreConnectionlf(Boolean openCondition, DbConnection storeConnectionToOpen, DbConnection originalConnection, String exceptionCode, String attemptedOperation, Boolean& closeStoreConnectic
[EntityException: The underlyin9 provider failed on Open.]
System.Data.EntityClient.EntityConnection.OpenStoreConnectionlf(Boolean opencondition, DbConnection storeConnectionToOpen, DbConnection originalconnection, String exceptioncode, String attemptedoperation, Boolean& closeStoreConnectic
System.Data.EntityClient.EntityConnection.Open() +142
System.Data.Objects.ObjectContext.EnsureConnection() +97
System.Data.Ob1ects.Ob1ectQuery’1.GetResults(Nullabl&1 forMergeoption) +66
System.Data.Objects.ObjectQuery 1.System.Collections .Generic.IEnumerable<T>.GetEnumeratorO ÷47
System.Linq.Enumerable.SingleOrDefault(IEnumerable 1 source) +271 
System.Linq.Queryable.SingleOrDefault(IQueryabl&1 source) +383 
System.Data.Entity.Internal .Linq.InternalSet’1.FindlnStore(WrappedEntityKey key, String keyValuesParamName) +779 
System.Data.Entity.Internal .Linq.InternalSet’1.Find(Object[] keyvalues) +182 
System.Web.Providers.DefaultSessionStateProvider .SetAndReleaseltemExclusive(HttpContext context, String id, SessionStateStoreData item, Object lockld, Boolean newltem) +259 
System.Web.SessionState.SessionStateModule.OnReleaseState(Object source, EventArgs eventArgs) +929 
System.Web.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +270

 

Open a command prompt in elevated mode and type netstat – anob

If you see a pattern as follows where a lot of ports are held by a single PID as below 

Active Connections

Proto         Local Address                 Foreign Address           State        PID

[MonAgentHost.exe]

 TCP         [::1]:80                       [::1]:52108            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52109            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52110            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52111            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52112            ESTABLISHED     4   Can not obtain ownership information  

 TCP         [::1]:80                       [::1]:52108            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52109            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52110            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52111            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52112            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52108            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52109            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52110            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52111            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52112            ESTABLISHED     4   Can not obtain ownership information 

 TCP         [::1]:80                       [::1]:52108            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52109            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52110            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52111            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:80                       [::1]:52112            ESTABLISHED     4   Can not obtain ownership information

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188 [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 TCP         [::1]:52108                 [::1]:80                 ESTABLISHED     3188  [WaIISHost.exe]

 

As we can see above that a single process with ID 4 or ID 3188 is holding numerous connections open to the server.

Do an RDP to the instance and please check if KB2750149 or KB2805227 is installed on the machine. The actual cause and resolution steps can be found at http://blogs.msdn.com/b/windowsazurestorage/archive/2013/08/08/net-clients-encountering-port-exhaustion-after-installing-kb2750149-or-kb2805227.aspx

 

Comments (0)

Skip to main content