Error configuring SSO service BizTalk 2016 in a SQL Server availability group

I got this error while configuring SSO using the BizTalk Server Configuration tool:

 

[DBNETLIB][ConnectionOpen (SECCreateCredentials()).]SSL Security Error error(ConfigHelper)

 

and this is the configuration log specific information:  

11:20:49 AM Info SSOServerCfg] (ssoconfigobj2.cpp:5164) Returning error description: <Exception Message="There was an error in the SSO configuration module. See the configuration log for further details." Source="SSO" HelpID=""><Exception Message="(0x80070433) The dependency service does not exist or has been marked for deletion. " Source="Win32" HelpID=""/></Exception>

[11:20:49 AM Warning SSOServerCfg] (ssoconfigobj2.cpp:1657) CSSOServerCfg::ConfigureFeature: 0x80004005 (-2147467259)

[2017-10-23 11:20:49:0655 Warning Configuration Framework]Feature failed to configure: SSOServer,Engine.

[2017-10-23 11:21:10:0994 Info Configuration Framework]Configuration Summaries:

[2017-10-23 11:21:10:0995 Error Configuration Framework]Feature: [Enterprise SSO] Failed to configure with error message [<Exception Message="There was an error in the SSO configuration module. See the configuration log for further details." Source="SSO" HelpID=""><Exception

Version:1.0 StartHTML:000000227 EndHTML:000002812 StartFragment:000002682 EndFragment:000002776 StartSelection:000002682 EndSelection:000002776 SourceURL:https://blogs.msdn.microsoft.com/amantaras/wp-admin/post-new.php .mce-content-body div.mce-resizehandle {position: absolute;border: 1px solid black;box-sizing: box-sizing;background: #FFF;width: 7px;height: 7px;z-index: 10000}.mce-content-body .mce-resizehandle:hover {background: #000}.mce-content-body img[data-mce-selected],.mce-content-body hr[data-mce-selected] {outline: 1px solid black;resize: none}.mce-content-body .mce-clonedresizable {position: absolute;outline: 1px dashed black;opacity: .5;filter: alpha(opacity=50);z-index: 10000}.mce-content-body .mce-resize-helper {background: #555;background: rgba(0,0,0,0.75);border-radius: 3px;border: 1px;color: white;display: none;font-family: sans-serif;font-size: 12px;white-space: nowrap;line-height: 14px;margin: 5px 10px;padding: 5px;position: absolute;z-index: 10001}
.mce-visual-caret {position: absolute;background-color: black;background-color: currentcolor;}.mce-visual-caret-hidden {display: none;}*[data-mce-caret] {position: absolute;left: -1000px;right: auto;top: 0;margin: 0;padding: 0;}
.mce-content-body .mce-offscreen-selection {position: absolute;left: -9999999999px;max-width: 1000000px;}.mce-content-body *[contentEditable=false] {cursor: default;}.mce-content-body *[contentEditable=true] {cursor: text;}
body {min-height: 150px}

 

There are few things you need to check and fix if you are having this issue:

  1. The Entsso.exe needs inbound & outbound access in order to be configured.  

    I have seen an issue similar to that in the past and for me it was a firewall issueNew-NetFirewallRule -DisplayName “Allow Inbound ESSO� -Direction Inbound -Program "C:\Program Files\Common Files\Enterprise Single Sign-On\ENTSSO.exe" -RemoteAddress LocalSubnet -Action Allow

    New-NetFirewallRule -DisplayName “Allow Outbound ESSO� -Direction Outbound -Program "C:\Program Files\Common Files\Enterprise Single Sign-On\ENTSSO.exe" -RemoteAddress LocalSubnet -Action Allow

     

  2. Check the availability group Listener properties and set the port to 1433 (or any other)

  3. Enable TSL 1.0 and 1.1  client and server on both the Biztalk server and the SQL server and reboot servers.
    BY default,  on the BizTalk server and the SQL server , only TLS 1.2 protocol is enabled for  under:

    HKey_Local_Machine\System\CurrentControlSet\Control\SecurityProviders \SCHANNEL\Protocols