Skip to main content

This content has been archived and is no longer being updated.

Links may not function; however, this content may be relevant to outdated versions of the product.

Troubleshooting: SOAP Connection errors with many TCP sockets in a CLOSE_WAIT state (Red Hat Enterprise Linux)

Suggest edit Updated on October 13, 2017

When running activities in production on Red Hat Enterprise Linux, you see errors in the Clipboard tool and in the Pega log. Examining the details of the connections from the client machine to the SOAP service endpoints reveals tens of thousands of TCP sockets in a Close_Wait state. Further analysis determines that this situation is the result of Axis2 connection pooling implemented by HFix-6324 and HFix-6912.

Errors

In the Clipboard tool

'java.net.BindException'

'ResourceUnavailableException'

'SOAP Fault Reason: Cannot assign requested address'

In the Pega log

'SOAP Fault Reason: Cannot assign requested address'

'java.io.IOException: Invalid keystore format'

'java.io.EOFException'

Explanation

Configuration of the Axis2 connection pooling is set too high by default, and, in the case of this operating system (Red Hat Enterprise Linux), the TCP sockets are not reclaimed after the specified timeout. This observed behavior is dependent on the client-side operating system where PRPC is running.

Consequently, the operating system is overrun with TCP sockets that cannot be reused and are allocated to the connection pool managed by Axis2. All endpoints are SOAP services in use by the PRPC application.

Suggested approach

To resolve the issue, specify the following Dynamic System Settings:

Pega-IntegrationEngine axis2_max_connections 1000

Pega-IntegrationEngine axis2_max_hostconnections 100

The values that you specify depend on your application, but the values suggested here are reasonable starting points in most cases.

This limits the overall allocation of any TCP sockets to the axis2_max_connections. Any one host is allowed 100 active connections.

When the connection is closed, the TCP socket enters a CLOSE_WAIT state in response from the endpoint sending a FIN in response to the CLOSE. This persists until a new connection is required. Then the socket is released by Axis2 by sending a LAST_ACK, and the socket is immediately reused for the new connection request. On some operating systems, the LAST_ACK is sent or the TCP socket is reclaimed.

Reclaiming of the TCP sockets is not guaranteed and, in the scenario described here, the sockets are not reclaimed, nor were they reclaimed during testing.

Axis2 HTTP Connection Pooling

Following is a summary of how Axis2 HTTP Connection Pooling works. It assumes that the connection uses an HttpConnection object.

  1. Initially, the number of connections in the pool is 0.
  2. When a request for a connection is made, a connection is returned following this sequence:
    1. First, the connection pool specific to the host (hostPool) is checked for any free connections.
      1. If available, a free connection is returned from this pool. Otherwise, the next condition is evaluated.
      2. If the number of connections in hostPool is less than MaxHostConnections and number of connections in the global connection pool (connectionPool) is less than MaxTotalConnections, a new connection is created and added to hostPool and connectionPool. Otherwise, the next condition is evaluated.
      3. If the number of connections in hostPool is less than MaxHostConnections and connectionPool has free connections but has reached the MaxTotalConnections, the least used connection is deleted from the pool and a new connection is created and added to hostPool and connectionPool. Otherwise, enter wait state.
    2. Enter waiting state.
      • The request waits as long as specified by the timeout value. The default is 30 seconds.
      • When the timeout occurs, an exception is thrown.
  3. A connection has an associated socket.

The Response Timeout configured in the Connect SOAP rule is also used as the Socket Timeout.

  1. At the end of the request, ServiceClient.cleanupTransport() is called.

This releases the connection back to the pool. The TCP Socket associated with the connection is not closed. It remains open as long as the platform does not reclaim it, or the corresponding connection is reestablished, whichever occurs first.

If the connection from the pool is reestablished, the underlying socket is checked for staleness.

If the operating system does not see the socket in CLOSE_WAIT state, the socket is reused.

  1. On Linux, you can observe that a socket remains in ESTABLISHED state for approximately two minutes after the corresponding HTTP connection is released to the pool.

At the end of two minutes, the socket timeout occurs, and its state is changed to CLOSE_WAIT. It remains in CLOSE_WAIT state for another 2 minutes before it is reclaimed by the operating system.

  1. On UNIX and Linux, the maximum number of open sockets allowed per process at any given point is 1024.

You can customize this setting, but doing so is risky, not a best practice.

  1. Therefore, the best practice is to reduce the Dynamic System Setting values for Axis2_Max_HostConnections to 100 and for Axis2_Max_Connections to 1000.

See the References section for more information.

Additional information

How to debug SOAP Connect failure, ResourceUnavailableException or RemoteApplicationException

References

Securing and Optimizing Linux RedHat Edition v1.3, Linux General Optimization, Better manage your TCP/IP resources

Apache JavaDoc, Class DefaultCommonsHTTPClientProperties

Did you find this content helpful? YesNo

100% found this useful

Have a question? Get answers now.

Visit the Collaboration Center to ask questions, engage in discussions, share ideas, and help others.

Ready to crush complexity?

Experience the benefits of Pega Community when you log in.

We'd prefer it if you saw us at our best.

Pega.com is not optimized for Internet Explorer. For the optimal experience, please use:

Close Deprecation Notice
Contact us