Deploy Visual Studio Test Agent Task – Test Machine Rebooting Scenarios

Deploy Visual Studio Test Agent (DTA) task in TFS/VSTS, is used for setting up Test Agent (TA) on multiple machines to run functional tests. Image below describes the functionality of the task and present reboot cases:

 

DTA Task & need of Auto-logon

To run UI tests that interact with desktop, test machine must have a valid user interactive session. If such valid session is not present, then DTA task relies on auto-logon to create such session and therefore reboots the test machine. Windows doesn’t provide any mechanism other than auto-logon, to login and create an interactive user session on the system without user intervention. More details on auto-logon and its setup can be found at How to turn on automatic logon in windows.

In case auto-logon is locally disabled on test machine, DTA enables it by doing changes to respective registry keys. But, if Group Policy has been enforced to disable auto-logon, then one need to consult their IT department to enable it on test machines.

There are features in windows logon (winlogon) viz. Legal notice/logon banner, that can cause auto-logon to fail.

DTA task does not disable legal notice/logon prompt on test machine, but if its already enabled then it does show a warning message. Legal Notice is a property of winlogon and can be set either as a policy or by modifying normal registry key. Refer to this Link for more details on enabling/disabling legal notice/logon message.

Running DTA task with auto-logon disabled due to any security/IT policy, could lead to infinite reboots of test machine in older versions of TFS. This issue has been addressed in VSTS, TFS 2015 Update 4 & TFS 2017 Update 2 onwards. Now, we keep a cap of maximum 3 reboots, i.e. DTA fails and exits task execution, if it is not able to create interactive session after trying thrice, with following error message:

“Stopping test machine setup as it exceeded maximum number of reboots. If you are running test agent in interactive mode, please make sure that auto-logon is enabled and no legal notice is displayed on logon in test machines.”

Also, VSTS / TFS 2015 Update 4 & TFS 2017 Update 2 onwards , DTA task provides warning messages to user in task build/release logs, if it finds that certain policies or registry entries may hamper auto-logon.

  1. Warning will be shown by DTA task if Legal Notice policy is found to be enabled (Refer screenshot above)

    Warning Message:
    Show legal notice at logon policy is enabled on test machine <machine-name>,  test agent configuration will not be able to auto-logon to run test agent interactively and UI tests will not be able to run. If such a policy is enabled, consider disabling it or use workgroup machines where group policies do not apply. 
  2. Warning will be shown by DTA task if Legal Notice registry key in winlogon is found to be enabled on test machine

    Warning Message:

    Show legal notice at logon is enabled on test machine <machine-name>, test agent configuration will not be able to auto-logon to run agent interactively due to this, causing test agent configuration to fail. Please disable showing of any logon message. 


  3. Warning will be shown by DTA task in case auto logon registry key is found to be disabled

    Warning Message:
    Auto-logon is disabled on test machine <machine-name>. Test agent configuration will attempt to enable auto-logon and will reboot for auto-logon to take effect. Any group policies that prevent auto-logon will prevent test agent from running interactively and UI tests will not be able to run. If such a policy is enabled, consider disabling it or use workgroup machines where group policies do not apply.

Commonly Asked Questions:

    1. What are the cases when DTA task can cause Test agent machine to reboot?

      • .NET Framework 4.6 is not installed on Test agent machine
      • TA is set to run UI test, and while configuring test agent on test machine, there does not exist an active session of test user i.e. either no user is logged in or users other than test user are logged in. (Test user is the user with which test agent needs to run and value for which is specified in DTA task definition) 
    2. What should be done if I cannot enable auto-logon on my test machine? 

      Auto-logon is a necessary configuration that DTA/RFT depends on, for running UI Tests, with no other alternatives. Its recommended in such a case:

        1. If you are not running UI tests, then uncheck the “Run UI Tests” (“Interactive process”) option in task definition
        2. Or ask your IT admin to provide exemption from domain policies disabling auto-logon in your test machines
    3. Test run having UI tests is successful when test user was logged in on test machine, but the DTA task timeouts when test user is not logged in?

      When test user is already logged in, DTA can find an active interactive session, hence it succeeds. But in case test user is not logged in, DTA tries to use auto-logon to create the session. If auto-logon is not successful, then it gets stuck in infinite reboot and hence task execution timeouts in older versions of TFS (TFS 2015 Update 3 or earlier & TFS 2017 Update 1 or earlier). 
    4. While DTA task is running, why test machine is rebooting repeatedly? 

      Possible cause is auto-logon is not able to complete. DTA task should error out after 3 reboots but in older versions of TFS (TFS 2015 Update 3 or earlier & TFS 2017 Update 1 or earlier) it can reboot infinitely until test user logs into test machine.
      To resolve it in older TFS versions, try to check for auto-logon policies or registry keys on TA machine so that auto-logon works correctly. Or try to consult your IT department for enabling auto-logon on your test machines. 

2