You may experience “'Configuration file is not well-formed XML” error while using DFSR to synchronize the IIS configuration files
Background
Recently we have several customers reported that they are using DFSR (Distributed File System Replication) to replicate IIS related configuration files between member servers.
However, sometimes once they make a change in IIS on the primary node, they found that the application pools failed to be started as expected on the member nodes and they are receiving the errors like follows in event logs at mean time:
Source: Microsoft-Windows-IIS-W3SVC-WP
Event ID: 2297
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Description:
The worker process for application pool 'FooAppPool' encountered an error 'Configuration file is not well-formed XML
' trying to read global module configuration data from file '\\?\C:\inetpub\temp\apppools\FooAppPool\FooAppPool.config', line number '3'. Worker process startup aborted.
It looks like IIS is continually looking at the applicationhosts.config file in the DFS share. And if a change is made, IIS will load the incomplete configuration file immediately which causes the above error.
Though the problem could be resolved by restarting IIS service, customer still prefers a proper solution as IIS access would be interrupted before IIS being restarted.
The problem occurred on IIS7, IIS7.5 and IIS8.
Root Cause
When DFS updates a replicated file, it deletes the original, then drops in the new file. In this setup, there will legitimately be situations where IIS can’t access its config files because they are locked while the files are copied. It is all dependent on timing. So it is possible that only part of the IIS config files being replicated to the member node while the change is being made on the primary node, and the member node synchronize the files immediately even only part of the config files being replicated successfully.
If we happened to capture the Process Monitor logs while the problem occurred, we can clarify that the DFSRs.exe would Open (CreateFile) & Delete (OpenForDelete) & Close (CloseFile) ApplicationHost.config file frequently during synchronization. And WAS (Windows Process Activation Service), which hosted in svchost.exe with process ID 2684, would be alerted to read refreshed ApplicationHost.config frequently by the File Change Notification. And the File Change Notifications were triggered by DFSR as it would open & close file frequently during file synchronization as well.
The call stack of the WAS process (svchost.exe 2684) shed more light on this: The function “ParseNewAppHostConfigFile” would be called to read the new version of the applicationHost.config file after receiving the File Change Notifications:
Solution
As a short term solution, we can add Registry Key “ConfigPollMilliSeconds” on the web servers as a workaround:
- HKLM\System\CurrentControlSet\Services\W3SVC\Parameters\ConfigPollMilliSeconds (REG_DWORD)
The default value for this registry key is 0. When this value is set to 0, the ConfigPollMilliSeconds parameter is disabled. The configuration system relies on change notifications to track changes to configuration files. A positive value for this key indicates that the configuration system checks the last modified time of the configuration file for every N milliseconds. The configuration system does not use the directory monitors.
See https://support.microsoft.com/kb/954864/en-us for detailed information.
We could configure IIS in order to research webConfig file every 5 minutes (or shorter based on your requirement) for example. To do that we have to set the following registry key as mentioned below:
HKLM\System\CurrentControlSet\Services\W3SVC\Parameters\ConfigPollMilliSeconds = 300000
As a long term solution, we can use “Web Deploy” tool to synchronize the IIS configurations (applications) to the member servers:
Web Deploy allows you to efficiently synchronize sites, applications or servers across your IIS server farm by detecting differences between the source and destination content and transferring only those changes which need synchronization. The tool simplifies the synchronization process by automatically determining the configuration, content and certificates to be synchronized for a specific site. In addition to the default behavior, you still have the option to specify additional artifacts for the synchronization, including databases, COM objects, GAC assemblies and registry settings.
We can use the following command to synchronize the websites easily:
msdeploy -verb:sync -source:apphostconfig="Default Web Site" -dest:apphostconfig="Default Web Site",computername=Server1 > msdeploysync.log
For more information about using “Web Deploy” to synchronize files between different IIS servers, please reference the following article:
https://www.iis.net/learn/publish/using-web-deploy/synchronize-iis
Regards,
YongKang Chen from DSI Team