Azure Quickstart Templates for SAP

As part of the announcement during the keynote at SAPPHIRE NOW to support SAP on Linux on Azure we released 5 Azure Quickstart Templates on github and 4 PowerShell cmdlets. For a description of the PowerShell cmdlets, read the blog New Azure PowerShell cmdlets for Azure Enhanced Monitoring

Azure Quickstart Templates

The quickstart templates can be used to create a 2-tier or 3-tier setup of one or multiple virtual machines that can be used to run SAP NetWeaver. You can select from four different sizes for each architecture which create setups ranging from 2000 SAPS to 250000 SAPS. The new virtual machine(s) can be connected to a new virtual network or to an existing virtual network which can be connected to your on-premises network via VPN or ExpressRoute. The property that determines whether the new virtual machines are connected to an existing VNET or a new VNET is created is NEWOREXISTINGSUBNET.

  • Productive setting with VPN (the subnet must already exist)
    NEWOREXISTINGSUBNET: existing
    SUBNETID: enter the subnet id. You can find the subnet id by e.g. using the PowerShell cmdlet
    Get-AzureRmVirtualNetwork -ResourceGroupName <resource group name> | select -ExpandProperty Subnets | select Id
  • Azure only (non production) – Azure only scenario
    NEWOREXISTINGSUBNET: Select new
    SUBNETID: Leave empty

The templates can be used to install SAP NetWeaver ABAP, SAP NetWeaver JAVA or a SAP NetWeaver Dual Stack System.

2-tier templates

We released three 2-tier templates which can be used to create one virtual machine using an Azure Marketplace image, an image that you created or an OS disk.

For more details about the deployed resources, see

The template creates two software RAID configuration. One of them should be used to store the database data files, the other one to store the database log files.

Please follow the SAP NetWeaver installation guides to install your SAP system. The installation for a 2-tier system does not differ from an installation on-premises.

3-tier templates

We release two 3-tier templates which can be used to create a 3-tier setup using an Azure marketplace image or an image that you created.

For more details about the deployed resources, see

The template does not deploy a dedicated virtual machine that can be used to host an Active Directory. If you need to join the virtual machines to a domain (e.g. if you want to use Windows Failover Cluster) and you do not already have a domain controller, add an additional virtual machine to the resource group.

The template creates two software RAID configuration on the database server(s). One of them should be used to store the database data files, the other one to store the database log files. The (A)SCS and Application servers have one volume that can be used to install SAP NetWeaver.

Please follow the SAP NetWeaver installation guides to install your SAP system if you deployed a non-HA configuration. The installation for a 3-tier non-HA system setup does not differ from an installation on-premises.

Installing SAP NetWeaver on a 3-tier HA setup

For a HA setup, the template creates two load balancers. The first load balancer is used for the database servers and the second load balancer is used for the SAP Central Services (ASCS or SCS).

Database Load Balancer

The Database Load Balancer uses probe port 62400 to decide to which virtual machine the traffic is routed. You need to make sure that the virtual machine hosting the active database instance listens on port 62400. For more information, read chapter Probe Port below.

The Database Load Balancer create a rule for port 1433 that can be used to setup a SQL Server AlwaysOn Availability Group Listener. You can adapt this rule to the needs of your database software.

(A)SCS Load Balancer

The (A)SCS Load Balancer uses probe port 62300 to decide to which virtual machine the traffic is routed. You need to make sure that the virtual machine hosting the (A)SCS listens on port 62300. For more information, read chapter Probe Port below.

The (A)SCS Load Balancer creates rules for the following ports:

445 SMB File Server (used for sapmnt)
3200 Enqueue Server of ASCS instance
3201 Enqueue Server of SCS instance
3301 Gateway Server of SCS instance
3600 ABAP Message Server
3900 Internal ABAP Message Server
3901 Java Message Server
5985 WinRM
8100 ASCS Message Server HTTP
8101 SCS Message Server HTTP
50013 ASCS SAP Start Service SCS HTTP
50113 SCS SAP Start Service SCS HTTP
51013 SAP Start Service ASCS ERS HTTP
51113 SAP Start Service SCS ERS HTTP
50014 ASCS SAP Start Service SCS HTTPS
50114 SCS SAP Start Service SCS HTTPS
51014 SAP Start Service ASCS ERS HTTPS
51114 SAP Start Service SCS ERS HTTPS
50016 ASCS Enqueue Replication
50116 SCS Enqueue Replication

Install the SAP System with the following instance numbers:

ASCS Instance Number 00
ERS Instance Number for ASCS 10
SCS Instance Number 01
ERS Instance Number for SCS 11

Probe Port

The probe port is used to decide which virtual machines are currently part of the load balancing. For a HA SAP installation, only one virtual machine should be active at any time. For Windows Failover Cluster, this would be the node that where the (A)SCS is running on. For more information on how to configure it for Windows Failover Cluster, see the Step-by-Step chapter below.

Step by Step installation of SAP NetWeaver 7.5 ABAP on SQL Server using Windows Failover Cluster

This step by step guide assumes that you create a new virtual network as part of the deployment and that you do not already have an Active Directory Domain Controller.

Required Software

  1. SQL4SAP SQL Server 2012 or higher DVD
  2. SAP Software Provisioning Manager (latest version recommended)
  3. sapcar to unzip SAR files
  4. SAP DC Kernel 7.45 Windows Server on x64 64bit
  5. NW 7.5 Installation Export
  6. SAP GUI installation

Installation Steps

Deploy resources

Open https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fsap-3-tier-marketplace-image%2Fazuredeploy.json in the browser enter the following parameters:

  • SAPSYSTEMID: Enter the SAP System ID that you want to use. For this step-by-step guide we assume that you entered AZR.
  • OSTYPE: Select Windows Server 2012 R2 Datacenter
  • SAPSYSTEMSIZE: Select the size you want to use. For more information about the resources that get deployed, see https://github.com/Azure/azure-quickstart-templates/tree/master/sap-3-tier-marketplace-image
  • SYSTEMAVAILABILITY: Select HA
  • ADMINUSERNAME: Enter a username
  • ADMINPASSWORD: Enter the password for the new user with administrative privileges
  • NEWOREXISTINGSUBNET: Select new
  • SUBNETID: Leave empty

Click on OK

Enter the name of a new resource group accept the Legal Terms and then click on Create

Once the deployment is finished, the resource group contains the following resources:

Name ResourceType
azr-avset-ascs Availability Sets for ASCS machine(s)
azr-avset-db Availability Sets for Database server(s)
azr-avset-di Availability Sets for SAP Application server(s)
azr-ascs-x Virtual Machine(s) hosting SAP (A)SCS
azr-db-x Virtual Machine(s) hosting database server(s)
azr-di-x Virtual Machine(s) hosting SAP Application server(s)
azr-lb-ascs Load Balancer for (A)SCS server(s)
azr-lb-db Load Balancer for database server(s)
azr-nic-ascs-x Network Interface of (A)SCS server(s)
azr-nic-db-x Network Interface of database server(s)
azr-nic-di-x Network Interface of SAP Application server(s)
azr-nsg-ascs-x Network Security Group of (A)SCS server(s)
azr-pip-ascs-x Public IP Address of (A)SCS server(s)
azr-vnet Virtual Network
ascsazr<random>x Storage Account(s) for (A)SCS server(s)
dbazr<random> Storage Account for database server(s)
diazr<random>x Storage Account(s) for (A)SCS server(s)

Install Active Directory

  • Create a new virtual machine in the resource group where you deployed the template and use the virtual network. In this example, the virtual machine is called azr-dc
  • Open a remote desktop to azr-dc either via a public ip address that you created for this virtual machine or open a remote desktop on azr-ascs-0 via its public ip address and then open a remote desktop on azr-dc.
  • Open a PowerShell and execute


Install-WindowsFeature AD-Domain-Services,RSAT-AD-Tools
Install-ADDSForest -DatabasePath "C:\Windows\NTDS" -DomainName "azr.local" -DomainNetbiosName "AZR" -InstallDns -LogPath "C:\Windows\NTDS" -NoRebootOnCompletion -SysvolPath "C:\Windows\SYSVOL"

  • Run ipconfig and note down the IP address of azr-dc (for example 10.0.0.11)
  • Reboot the server

Change Virtual Network configuration

  • Open portal.azure.com, open Virtual Networks and select azr-vnet. Open the settings of the virtual network, and select DNS Servers. Switch to Custom DNS and enter the IP address of azr-dc as the primary DNS server.
  • Note down the IP addresses of azr-nic-di-0, azr-nic-db-0, azr-nic-db-1, azr-nic-ascs-0 and azr-nic-ascs-1 that are listed in Connected Devices
  • Restart all virtual machines to ensure that the new network configuration is applied. You can use the Azure Portal or the following command to restart all virtual machines in a resource group


Get-AzureRmVM -ResourceGroupName <resource group name> | Restart-AzureRmVM

Join all virtual machines to domain

  • Open a remote desktop on azr-ascs-0 and execute the following command in PowerShell for the IP address of azr-nic-di-0, azr-nic-db-0, azr-nic-db-1 and azr-nic-ascs-1 in this order (do not execute the commands on azr-ascs-0). You will be logged out if you execute the command for azr-ascs-0 since the virtual machine will be restarted.


Set-Item WSMan:\localhost\Client\TrustedHosts <ip address> -Concatenate –Force
Invoke-Command -ComputerName 10.0.0.8 -ScriptBlock { Enable-NetFirewallRule "WMI-WINMGMT-In-TCP";Enable-NetFirewallRule "WMI-RPCSS-In-TCP"} -Credential (get-credential <admin user>)
Add-Computer -DomainName azr.local -ComputerName <ip address> -LocalCredential (get-credential <admin user>) -Credential (Get-Credential azr\<admin user>)
Restart-Computer -ComputerName <ip address> -Force

  • On azr-ascs-0 execute the following commands


Add-Computer -DomainName azr.local -Credential (Get-Credential azr\<admin user>)
Restart-Computer -Force

Create database cluster

  • Open a remote desktop on azr-ascs-0 and login as AZR\<admin user>
  • Open a PowerShell and execute


Install-WindowsFeature Failover-Clustering,RSAT-Clustering, NET-Framework-Core -ComputerName azr-db-0
Install-WindowsFeature Failover-Clustering,RSAT-Clustering, NET-Framework-Core -ComputerName azr-db-1
Install-WindowsFeature RSAT-Clustering
New-Cluster -Name AZRCLSQL -Node azr-db-0,azr-db-1 -StaticAddress 10.0.0.100 –NoStorage

Install SQL Server

  • Open remote desktop on azr-db-0 and login with AZR\<admin user>. Execute the following command


<path to sql4sap dvd>\SQL4SAP.bat -d -u azr\<admin user>

  • Open remote desktop on azr-db-1 and execute the following command


<path to sql4sap dvd>\SQL4SAP.bat -d -u azr\<admin user>

  • On azr-db-0 open a PowerShell and execute the following commands:


Invoke-Command -ComputerName azr-db-1 { New-NetFirewallRule -Name SQLAZR -DisplayName "Rule for AZR SQL Server" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 1433 }
Invoke-Command -ComputerName azr-db-1 -ScriptBlock { New-NetFirewallRule -Name SQLAZRAG -DisplayName "Rule for AZR SQL Server AG" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 5022 }
Invoke-Command -ComputerName azr-db-1 -ScriptBlock { New-NetFirewallRule -Name SQLPROBE -DisplayName "Rule for AZR SQL Server AG Probe Port" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 62400 }
New-NetFirewallRule -Name SQLAZR -DisplayName "Rule for AZR SQL Server" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 1433
New-NetFirewallRule -Name SQLAZRAG -DisplayName "Rule for AZR SQL Server AG" -Direction Inbound -ActionAllow -Protocol TCP -LocalPort 5022
New-NetFirewallRule -Name SQLPROBE -DisplayName "Rule for AZR SQL Server AG Probe Port" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 62400
Import-Module SQLPS
Invoke-Sqlcmd -ServerInstance azr-db-0 -Query "CREATE DATABASE AZR"
New-Item c:\sapcd -Type Directory –Force
Backup-SqlDatabase -Database AZR -Path ("SQLSERVER:\SQL\azr-db-0\DEFAULT") -BackupFile "c:\sapcd\hadrbackup.sql"
Invoke-Command -ComputerName azr-db-1 -ScriptBlock { New-Item c:\sapcd -Type Directory -Force }
xcopy /Y c:\sapcd\hadrbackup.sql \\azr-db-1\c$\sapcd
Restore-SqlDatabase -Database AZR -Path ("SQLSERVER:\SQL\azr-db-1\DEFAULT") -BackupFile "c:\sapcd\hadrbackup.sql" –NoRecovery
Enable-SqlAlwaysOn -Path SQLSERVER:\SQL\azr-db-0\DEFAULT –Force
Invoke-Command -ComputerName azr-db-1 { Enable-SqlAlwaysOn -Path SQLSERVER:\SQL\azr-db-1\DEFAULT –Force }
Invoke-Sqlcmd -ServerInstance azr-db-0 -Query ("CREATE LOGIN [azr\azr-db-1`$] FROM WINDOWS WITH DEFAULT_DATABASE=[master]")
Invoke-Sqlcmd -ServerInstance azr-db-0 -Query ("ALTER SERVER ROLE [sysadmin] ADD MEMBER [azr\azr-db-1`$]")
Invoke-Sqlcmd -ServerInstance azr-db-1 -Query ("CREATE LOGIN [azr\azr-db-0`$] FROM WINDOWS WITH DEFAULT_DATABASE=[master]")
Invoke-Sqlcmd -ServerInstance azr-db-1 -Query ("ALTER SERVER ROLE [sysadmin] ADD MEMBER [azr\azr-db-0`$]")
New-SqlHadrEndpoint Hadr_endpoint -Port 5022 -Path ("SQLSERVER:\SQL\azr-db-0\DEFAULT") | Set-SqlHadrEndpoint -State "Started"
New-SqlHadrEndpoint Hadr_endpoint -Port 5022 -Path ("SQLSERVER:\SQL\azr-db-1\DEFAULT") | Set-SqlHadrEndpoint -State "Started"
$templdb0 = New-SqlAvailabilityReplica -Name azr-db-0 -EndpointURL ("TCP://azr-db-0:5022") -AvailabilityMode "SynchronousCommit" -FailoverMode "Automatic" -Version 11 -AsTemplate
$templdb1 = New-SqlAvailabilityReplica -Name azr-db-1 -EndpointURL ("TCP://azr-db-1:5022") -AvailabilityMode "SynchronousCommit" -FailoverMode "Automatic" -Version 11 –AsTemplate
New-SqlAvailabilityGroup -Name "AZRAG" -Path ("SQLSERVER:\SQL\azr-db-0\DEFAULT") -AvailabilityReplica $templdb0,$templdb1 -Database AZR
Join-SqlAvailabilityGroup -Path ("SQLSERVER:\SQL\azr-db-1\DEFAULT") -Name "AZRAG"
Add-SqlAvailabilityDatabase -Path ("SQLSERVER:\SQL\azr-db-1\DEFAULT\AvailabilityGroups\AZRAG") -Database AZR
Resume-SqlAvailabilityDatabase -Path ("SQLSERVER:\SQL\azr-db-1\DEFAULT\AvailabilityGroups\AZRAG\AvailabilityDatabases\AZR")
New-SqlAvailabilityGroupListener -Path ("SQLSERVER:\SQL\azr-db-0\DEFAULT\AvailabilityGroups\AZRAG") -Port 1433 -Name AZRDB -StaticIp 10.0.0.5/255.255.255.0
$resource = Get-ClusterResource | where ResourceType -eq "IP Address" | where Name -like "*AZR*"
$resource | Set-ClusterParameter -Name ProbePort -Value 62400
$resource | Stop-ClusterResource
Start-ClusterGroup -Name "AZRAG"

Create (A)SCS Windows Failover Cluster

  • Open a remote desktop on azr-ascs-0 and login as AZR\<admin user>
  • Open a PowerShell and execute


Install-WindowsFeature Failover-Clustering,RSAT-Clustering, RSAT-DNS-Server -ComputerName azr-ascs-0
Install-WindowsFeature Failover-Clustering,RSAT-Clustering -ComputerName azr-ascs-1
Install-WindowsFeature RSAT-Clustering
New-Cluster -Name AZRCLSAP -Node azr-ascs-0,azr-ascs-1 -StaticAddress 10.0.0.200 –NoStorage
Add-DnsServerPrimaryZone -ReplicationScope Domain -DynamicUpdate Secure -Name 0.0.10.in-addr.arpa -DirectoryPartitionName DomainDnsZones.azr.local -ComputerName azr-dc
Add-DnsServerResourceRecordA -Name AZRSAP -IPv4Address 10.0.0.6 -ZoneName azr.local -CreatePtr -ComputerName azr-dc
Invoke-Command -ComputerName azr-ascs-1 -ScriptBlock { New-NetFirewallRule -Name ASCSPROBE -DisplayName "Load Balancer Probe Port for ASCS" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 62300 }
New-NetFirewallRule -Name ASCSPROBE -DisplayName "Load Balancer Probe Port for ASCS" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 62300
@(3200,3201,3301,3600,3900,3901,5985,8100,8101,50013,50113,51013,51113,50014,50114,51014,51114,50016,50116) | % { New-NetFirewallRule -Name "SAP$_" -DisplayName "SAP Port $_" -Direction Inbound -Action Allow -Protocol TCP -LocalPort $_ }
Invoke-Command -ComputerName azr-ascs-1 -ScriptBlock { @(3200,3201,3301,3600,3900,3901,5985,8100,8101,50013,50113,51013,51113,50014,50114,51014,51114,50016,50116) | % { New-NetFirewallRule -Name "SAP$_" -DisplayName "SAP Port $_" -Direction Inbound -Action Allow -Protocol TCP -LocalPort $_ } }
$setting = Get-WmiObject Win32_PageFileSetting
$setting.InitialSize = 35000
$setting.MaximumSize = 35000
$setting.Put()
Invoke-Command -ComputerName azr-ascs-1 -ScriptBlock {
$setting = Get-WmiObject Win32_PageFileSetting
$setting.InitialSize = 35000
$setting.MaximumSize = 35000
$setting.Put()
}

Install SAP (A)SCS on first node

  • Open remote desktop on azr-ascs-0 and login with AZR\<admin user>. Start SWPM with option SAP NetWeaver 7.5 -> MS SQL Server -> MS SQL Server -> Application Server ABAP -> High-Availability System -> First Cluster Node
    • a. Use drive S: for local instances
    • Use AZRSAP as Network Name (SAP Virtual Instance Host)
    • Use Instance Number 00 for ASCS instance
    • Keep defaults for ABAP Message Server ports
    • Use Instance Number 10 for ERS instance
  • Open a PowerShell and set the Probe port of the cluster resource with the following command


$resource = Get-ClusterResource | where ResourceType -eq "IP Address" | where Name -like "*SAP AZR IP*"
$resource | Set-ClusterParameter -Name ProbePort -Value 62300
$resource | Stop-ClusterResource
Start-ClusterGroup -Name "SAP AZR"

Install SAP Database instance

  • Open a remote desktop on azr-db-0 and login with AZR\<admin user>. Open a PowerShell and execute the following command


if ((Get-Item SQLSERVER:\SQL\azr-db-0\DEFAULT\AvailabilityGroups\AZRAG).PrimaryReplicaServerName -ne "azr-db-0") { Switch-SqlAvailabilityGroup SQLSERVER:\SQL\azr-db-0\DEFAULT\AvailabilityGroups\AZRAG }

  • Start SWPM with option SAP NetWeaver 7.5 -> MS SQL Server -> MS SQL Server -> Application Server ABAP -> High-Availability System -> Database Instance
    • Use \\azrsap.azr.local\sapmnt\AZR\SYS\profile as Profile Directory
    • Select AZRDB as Database Instance
    • MS SQL Server Tempdb Configuration, place tempdb data file on S:\TEMPDB\tempdev.mdf and tempdb log file on L:\TEMPDB\templog.ldf
    • SQL Server Memory Configuation. Since the database servers are only used for the database, change to Custom Memory Configuration and enter ~ 90% of available memory e.g. 50000 for a server with 56GB for min and max server memory

Setup second database node


Msg 3701, Level 11, State 5, Line 1
Cannot drop the procedure 'sp_hexadecimal', because it does not exist or you do not have permission.
Msg 3701, Level 11, State 5, Line 33
Cannot drop the procedure 'sap_help_revlogin', because it does not exist or you do not have permission.

  • Execute


execute sap_help_revlogin

  • Copy the output of the procedure, log on to azr-db-1 and execute it using SQL Server Management Studio. You can ignore errors about AZR-DB-0\SAP_AZR_LocalAdmin not being found

Add second (A)SCS node

  • Open remote desktop on azr-ascs-1 and login with AZR\<admin user>. Start SWPM with option SAP NetWeaver 7.5 -> MS SQL Server -> MS SQL Server -> Application Server ABAP -> High-Availability System -> Additional Cluster Node
    • Select Cluster Group SAP AZR and use drive S: for local instances
    • Use Instance Number 10 for ERS instance

Setup cluster quorums

This chapter describes how to configure a quorum for the database cluster using a file share of the ASCS cluster. To configure the quorum for the ASCS cluster you could add a shared disk to the database cluster and use it as a file share witness for the ASCS cluster.

  • Open a remote desktop on azr-ascs-0 and login as AZR\<admin user>
  • Open a PowerShell and execute


Move-ClusterGroup -Node azr-ascs-0 -Name "SAP AZR"
New-Item -Path X:\witness -Type Directory
New-SmbShare -ScopeName AZRSAP -Path "X:\witness" -FullAccess azr\sapadmin -Name Witness

  • Open a remote desktop on azr-db-0 and login as AZR\<admin user>
  • Open a PowerShell and execute


Set-ClusterQuorum -FileShareWitness \\AZRSAP\Witness

Install SAP Primary Application Server Instance

  • Open remote desktop on azr-di-0 and login with AZR\<admin user>.
  • Open a PowerShell and execute


Install-WindowsFeature NET-Framework-Core
$setting = Get-WmiObject Win32_PageFileSetting
$setting.InitialSize = 35000
$setting.MaximumSize = 35000
$setting.Put()

  • Open a command prompt and execute


<path to sql4sap dvd>\SQL4SAP.bat -c

  • Start SWPM with option SAP NetWeaver 7.5 -> MS SQL Server -> MS SQL Server -> Application Server ABAP -> High-Availability System -> Primary Application Server Instance
  • Install SAP GUI
  • Otional: Install additional application servers

You need to install additional SAP application servers to protect the SAP system against a restart of the DI server (azr-di-0). You can either create additional virtual machines or install additional application servers on the ASCS servers (azr-ascs-0 and azr-ascs-1) as described in chapter “Install SAP Primary Application Server Instance” of this blog.

Test failover

Failover database
  • Open a remote desktop on azr-db-0 and login with AZR\<admin user>. Open a PowerShell and execute the following command


Import-Module SQLPS
Switch-SqlAvailabilityGroup -Path SQLSERVER:\SQL\azr-db-1\DEFAULT\AvailabilityGroups\AZRAG

  • If azr-db-1 is already the primary replica, you get an error that the failover cannot be done. You can ignore this error.
  • Open a remote desktop on azr-di-0 and login with AZR\<admin user> and logon to the system using SAP GUI and
    • Connection Type: Group / Server Selection
    • System ID: AZR
    • Message Server: AZRSAP
    • Group/Server: SPACE
  • Open transaction ST04 and check if the DB server is azr-db-1
  • Open a PowerShell and execute


Import-Module SQLPS
Switch-SqlAvailabilityGroup -Path SQLSERVER:\SQL\azr-db-0\DEFAULT\AvailabilityGroups\AZRAG

  • Go back to the start screen in SAP GUI and reopen transaction ST04. The DB Server should now be azr-db-0.
Failover (A)SCS
  • Open a remote desktop on azr-ascs-0 and login with AZR\<admin user>. Open a PowerShell and execute the following command


Move-ClusterGroup -Name "SAP AZR"

  • Open a remote desktop on azr-di-0 and login with AZR\<admin user> and logon to the system using SAP GUI and
    • Connection Type: Group / Server Selection
    • System ID: AZR
    • Message Server: AZRSAP
    • Group/Server: SPACE
  • Test that the login to the SAP System is working