How to Setup a Dedicated SAP Application Server to DB Server Network

What is a Dedicated SAP Application Server to DB Network & Do I Need One?

A dedicated SAP Network is a network interface that is separated from the normal “Public” user network and carries only network traffic between the SAP application servers, the (A)SCS and the SQL Server database. This blog discusses why and how to set this up and provides a step by step procedure for doing this on a cluster system.

Larger SAP systems that are now common on Win/SQL with more than 6-15 million Dialog Steps/Day, > 650,000 Dialog Steps/hour and more than a few thousand users will greatly benefit from isolating the “intra-SAP” network traffic from the “Public” server network. There are also some security reasons for doing this in addition to SSL. 

Recently I setup a dedicated 10 Gigabit Ethernet Network for a customer migrating a ECC 6.0 9.5TB Database from HPUX/Oracle to Windows/SQL Server. The customer moved to Win/SQL and achieved the following:

1. DB Size reduced from 9.5TB to 2.1TB due to SQL Server Compression
2. Reduced Backup Size– SQL Server Backup Compression reduced 2.1TB on SQL Server to less than 1TB. 
3. Reduced Backup Time– SQL Server Backup Compression reduced backup runtime from >6 hours to 1 hour 4 min.
4. The CHECK DATABASE time reduced dramatically to just 3 hours
5. Dramatic increase in performance on new high clock speed Intel Westmere x5675 2 socket commodity servers with 96GB RAM compared to Itanium application servers
6. Dramatic reduction in TCO – acquisition, operation & maintenance costs

During the project testing high network utilization was detected. Why was high network utilization observed? Why was the network utilization on HPUX/Oracle not so high?

Modern Intel servers are extremely powerful yet are very low cost/economic to purchase. The SAPS/thread of modern Intel Servers such as Intel SandyBridge servers is higher than UNIX platforms and the highest value ever recorded on any SAP SD Benchmark ever (on any OS/DB combination). This has created a problem : Network is a bottleneck when using built in 1 Gigabit network cards. The CPU on Intel servers is so powerful network can become a bottleneck.

Why does this bottleneck not show up on older UNIX servers? Simple – these servers do not have enough SAPS / processing power to ever come close to saturating a 1 Gigabit connection. The slowest part of the Itanium SAP application server is the CPU which has a relatively low clock speed which leads to very poor performance on single threaded applications like SAP. 

Ethernet Networks have evolved over recent years not only in terms of bandwidth but also features:

1. 10 Megabit/sec Ethernet – 10BASE-T
2. 100 Megabit/sec Ethernet – 100BASE-T
3. 1,000 Megabit/sec Ethernet (called “1Gig”) – 1000BASE-T (most often uses Copper UTP cable)
4. 10,000 Megabit/sec Ethernet (called “10Gig”) – 10GBASE-T (most often uses Fibre Optical cable)

As I discussed in my recent blog on RSS, there are a number of new technologies and features that are needed to allow very fast networks to scale. Windows 2008 R2 has advanced feature support. Older versions of Windows generally do not have the ability to fully leverage new Network technologies. 10 Gigabit Network Cards cannot be fully leveraged with older versions of Windows such as Windows 2003.

A list of these technologies (not complete) is below:

1. TCP and UDP Offload for IPv4 and IPv6
2. Large Receive Offload (called Receive Side Coalescing), Large Send Offload
3. Receive Side Scaling
4. Jumbo Frame / Packet Size
5. Settings to localize the Device Driver to a particular NUMA Node, K-Group or Logical Processor
6. Flow Control
7. Interrupt Moderation
8. CHECKSUM Offload

At the time of writing the Emulex and Intel 10Gig network cards are tested and recommended. Network cards from other vendors are still being deployed and tested as at April 2012.

The QLogic 10G Network Card (sometimes sold as NC523) has not been validated for use as at April 2012.

The Windows operating system does not set or default most of these values. The Network Card vendor (Intel or Emulex for example) will set the correct default values with their device driver. It is therefore very important to have up to date firmware and device drivers for network cards. 

The technologies listed above are in some cases relatively new. Therefore I will create another blog shortly that will discuss how to test network throughput and how to debug network problems. This will include how to use Microsoft Network Monitor (free to download) to capture network packets.  

1. Diagram of Modern Consolidated SAP Infrastructure

Multi-SID SAP deployments have been common for > 5 years, although some Hardware vendors still promote a 2 node Active/Passive cluster for each and every SAP component as they are seemingly unaware of Multi-SID clusters. The majority of new SAP deployments or upgrades today use Active/Active Multi-SID clustering. Active/Passive clusters for each component have poor resource utilization. The example below shows a typical configuration deployed and operating at customer sites such as the system that runs Infosys’s 140,000 users worldwide. 

The overwhelming majority of SAP customers have very large and busy ECC, BW and SCM systems, but other systems such as EP, GRC and Solution Manager are typically not very busy yet they need to be highly available. The solution below meets the customer’s requirements for performance and availability, yet does *not* restrict the customer from an operation/upgrade/maintenance point of view. The details of how this is achieved is in the blog about Mountpoints and Multi-SID

Figure 1. SQL Server & SAP ASCS/SCS is clustered on 2 or 4 socket Intel servers.  

Additional DB SAPS can be added fully online by adding cluster nodes. Windows 2008 R2 supports up to 16 nodes and Windows "8" Server supports 64 cluster nodes. SAP application servers are a non-SPOF components and show best price/performance on 2 socket commodity Intel servers. If additional SAPS is required or addition SAP components required more application servers (or VMs) can be added online.   

2. Diagram of “Public”, “SAP” and “Heartbeat” Network

The configuration of “Multi-homed” servers is considerably more complex than servers with just one network card. Incorrect configuration can impact performance and stability of the SAP application, therefore this blog will step through all required steps.

Network High Availability using two network switches should be implemented. This ensures that even if one switch was to fail this will not impact the availability of the platform. 

Network Teaming is required to complete the HA solution for the network. Note: 10 Gigabit is enough bandwidth for any known customer. It is technically possible to use Teaming to aggregate multiple 10G ports (example: 2 x 10G = 20G) but this should seldom if ever be necessary. If port aggregation of 10G connections is used, be sure to benchmark network performance with and without port aggregation. Teaming configuration for “Network Fault Tolerance” only is commonly deployed - only one port is Active and the other port is in Standby mode. 

Most vendors like HP, Cisco, Dell and IBM ship their own teaming software. Windows 8 Server will have network teaming build into the operating system as standard. 

Figure 2. Fully Redundant Teamed Network Topology for modern consolidated Win/SQL deployment using 10Gigabit network for SAP internal traffic and 1Gigabit for SAPGUI & user traffic.

3. Network Diagram Including IP Addressing Scheme

The diagram shows a sample IP address schema:

1. SAP Public Network 1 Gigabit : Subnet = 10.10.100.x
2. SAP Private Network 10 Gigabit : Subnet = 172.10.10.x
3. Cluster Heartbeat Network : Subnet = 192.168.0.x

In addition to the IP addresses depicted below one additional IP address is required for the “Cluster Core Resources”. The IP address for the Cluster Core Resources must be on the Public network or another network with full access to Active Directory.

  Figure 3. Diagram showing IP addressing schema for 10 Gigabit Private SAP network, Cluster Heartbeat and “Public” network. Virtual IP addresses for the cluster resources are also shown

4. Network Requirements & Settings

Heartbeat, Public and SAP Private Network have different Requirements & Settings.

4.1 SAP Public 1 Gigabit Network : SAPGUI, RFC and other user traffic must be first in the Bindings Order. The Public network must have “Allow cluster network communication” and “Allow clients to connect”. This network must have connectivity to Active Directory to register the A and AAAA records in DNS. SAP require that IP -> Hostname must resolve to this interface. More information is provided at the very end of this blog on RSS

4.2 SAP Private 10 Gigabit Network : Only traffic from the SAP application server to/from SQL Server will use this network. This network should be second in the Bindings Order. The SAP private network must have “Allow cluster network communication” and “Allow clients to connect”. This network does not need any connectivity to AD.

4.3 Heartbeat Network : Heartbeat is used by the cluster itself to send/receive is alive notifications and should be third in the Bindings Order. This dedicated cluster interface is also used to send cluster database configuration updates to all cluster nodes. The Heartbeat network must have “Allow cluster network communication”. Do not allow clients to connect. The cluster use should be “Internal”. This network does not need any connectivity to AD.

Any additional networks such as those for management or backups should follow after the Heartbeat network in the bindings order.

Figure 4. Network Settings for Different Network Interfaces

4.4 DNS & Name Resolution: Windows Active Directory DNS should be used in all cases. Modern SAP releases do not require a /etc/hosts file in general. One exception to this is when the SAP Private 10G Network is disconnected & isolated from the main server network and therefore is unable to register the hostname in AD. In this case the Network Name Cluster Resource will start and will come online, but will have a comment that name resolution is not possible. This issue can be resolved by adding the hostname and IP address in the hosts file on the cluster nodes.  

4.5 Advanced TCPIP Settings: If the Private SAP 10G network is not connected to the main server network and cannot communicate to Active Directory then change the properties on the network settings as illustrated in Figure 5 below.

Figure 5. Advanced Network Settings for SAP Private 10G networks that are not connected to the main server network and/or cannot access Active Directory

5. SAP Configuration & Settings

SAP ABAP server will attempt to connect to the database host that is specified in the default.pfl profile parameter dbs/mss/server = instance\hostname. The values that are in the registry keys and the Windows Environment MSSQL_SERVER = instance\hostname are overridden by the values. The registry and environment variables are important for "external" processes such as R3Trans.exe etc

To avoid any name resolution issues it is possible to specify the IP address of the SQL Server 2008 cluster virtual hostname:

  default.pfl profile parameter dbs/mss/server = tcp:172.10.100.200

The tcp: prefix will force SAP to always connect via TCPIP (rather than named pipes or shared memory)

SQL 2012 Always On or Multi-subnet clustering (which is supported on Win2008 R2) requires a hostname as the IP address may change. 

Therefore it is necessary to test name resolution via ping and nslookup. If an old value for the SQL Server vHostname is returned by nslookup use the DNS admin tool to change to the correct ip address. This scenario can happen when a private 10G network is added to an existing system and the vHostname for SQL Server is moved from the public network to a private network. 

6. Security Settings

One benefit of a Private SAP network that is not connected the main server network is to completely block any access to SQL Server other than from the SAP application servers. SQL Server will have the Private SAP network IP 172.10.100.200 address bound in cluster admin, therefore there will no way to access SQL Server from any device on the Public network.

It is generally recommended isolate the Private SAP Network from the Public network. 

It is possible to bind two or more IP addresses to SQL Server if access onto the Public network is required. To do this Add a Resource –> More Resources –> Add IP Address. Select the appropriate Subnet and then start the IP address. Pay careful attention to the dependencies. 

7. How to Validate that Hostname Resolution is Correct?

Please review this previous blog on Network Settings, Network Teaming, Receive Side Scaling (RSS) & Unbalanced CPU Load.

Run transaction SMLG and click on Message Server Status.

In addition please check the dev_ms tracefile. The virtual hostname of the ASCS should resolve to the vIP on the SAP Public 1G network.

Another blog will be released soon discussing how to stress test and validate the network configuration, how to trouble shoot performance issues with teaming software and how to use Network Monitor to analyse network issues.