Exchange Server Management - Concepts
Windows Servers have a remote management interface called the WinRM service that listens for connections. Remote PowerShell connections to servers connect to the WinRM service on the target server. The default WinRM port is not always available from all networks and is probably not optimal for some use cases on Exchange Server. Instead, you can connect to the Exchange Server via web services that proxy commands to the WinRM service. This web service is a virtual application named PowerShell that is hosted alongside Outlook Web Access (OWA) and other Exchange web applications in the same Internet Information Server website (IIS). When troubleshooting connections to Exchange Server there are several things to consider beyond the pre-requisites in the File Access Manager documentation:
- http/https and the port
- IIS authentication
- Certificates
- Exchange Server architecture
- Exchange Server problems
How does File Access Manager connect to Exchange?
The entitlement collector (EC) and activity monitor get information from Exchange by sending web service requests to 2 separate virtual applications on the Exchange web site.
/PowerShell Virtual Application - This virtual application is setup as part of the Exchange Server installation. The Crawler and EC connect to this.
/WBXPowerShell Virtual Application - This virtual application is a copy of the /PowerShell application made by the WBXPowerShell_Installer_x64.msi. At this writing the BAM connects to this virtual application, though this may change in future versions.
Screenshot - IIS Manager on an Exchange Server
Exchange Architecture
Load Balancer
In some cases the Exchange architecture has 2 or more Exchange servers behind a load balancer. File Access Manager does not support connections through a load balancer, so you will have to connect directly to one of the Exchange Servers. In this architecture users connect to mail.abc.com and each Exchange server has a certificate with mail.abc.com for the common name rather than hostname.abc.com, this may cause trust issues when making SSL connections. The work around will be to edit the host file on the File Access Manager server that is making the connection to the Exchange Server.
Exchange Front-End Servers
File Access Manager must connect to a Client Access Servers (CAS) server. In some cases the CAS server is front-end servers only that has no mailboxes and only proxy traffic to backend Exchange servers. A back-end server has mailboxes and is configured to communicate with a front-end server with HTTP, POP3 or IMAP. If File Access Manager is connection to a front-end CAS server then the pre-requisites must apply to the back-end server(s) as well.
HTTP vs. HTTPS Connections
When File Access Manager connects to Exchange server using HTTP it also uses Kerberos authentication. You can rest assured that the credentials are protected so you can probably avoid the extra configuration and troubleshooting effort required for HTTPS connections. If the Exchange administrators are using HTTP for their PowerShell connections then use HTTP for File Access Manager also.
Using PowerShell to Test Connectivity to Exchange Server
If you are unable to connect to an Exchange server you can troubleshoot by trying to connect using PowerShell. Remember that the EC connects to the /PowerShell virtual application and the BAM connects to /WBXPowerShell so be sure to update the examples below as needed.
Example 1
protocol: http
port: 80
authentication: Kerberos
$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri http://ex01.dom1.loc/PowerShell -Authentication Kerberos -Credential $credsPS
Import-PSSession $session
Example 2
protocol: https
port: 443
authentication: Basic
If Kerberos authentication does not work over SSL, see the troubleshooting section of this document.
$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ex01.dom1.loc/PowerShell -Authentication Basic -Credential $credsPS
Import-PSSession $session
To workaround certificate trust issues you can use the -SessionOptions switch with the connection:
$SessionOptions = New-PSSessionOption –SkipCACheck –SkipCNCheck –SkipRevocationCheck
error: WARNING: Proxy creation has been skipped
You may get this error when running the above commands: "WARNING: Proxy creation has been skipped for the following command: 'Add-DistributionGroupMember…" You can remedy this by running these commands:
Enter-PSSession $SessionPS
Import-PSSession $session
Close the PowerShell Connection
Exchange Online allows a small number of concurrent PowerShell connections so be sure to close you test connections. If they are left open and the maximum sessions are reached then you must wait for a session to expire before you can connect again. Close the PowerShell session close it with this command:
Remove-PSSession $session
or this:
Exit-PSSession
Remove-PSSession -ID $Session.ID
Screenshot – Opening a PowerShell session
Exchange Online
PowerShell connections to the O365 Exchange Online service are throttled on the server side after a short amount of time. Expect that when a crawl and EC task begins it will make steady progress for about 5 minutes, then may slow down considerably.
Connecting to Exchange Server - Common Errors
Untrusted Certificate Error
Self-signed certificates in Exchange test environments prevent a successful connection to Exchange Servers. Export the certificate from the Exchange server website and import it into Trusted Root Certification Authority (Local Computer) on the server where the EC is installed. Check the IIS bindings to get the display name of the certificate that should be exported for the EC.
In this case the Exchange application has been added to File Access Manager and is configured to connect port 443 using SSL and the EC and BAM work when using non-SSL connections to Exchange on port 80.
entitlement collector log sample
The EC log will show this error when there is a certificate trust issue.
2016-05-04 14:16:08,764,40,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Could not open PowerShell session using Authentication: 'Default' User name: 'DOM1\SIQ_EX' Server: 'https://ex01/Powershell' Schema: 'http://schemas.microsoft.com/powershell/Microsoft.Exchange'
System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server ex01 failed with the following error message : The WinRM client received an HTTP bad request status (400), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic.
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)
at WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent.openRunspacePool(RunspacePool runspacePool)
2016-05-04 14:16:08,764,40,DEBUG,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,initRunspacePool,Falling back to Basic authentication
2016-05-04 14:16:08,764,40,DEBUG,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Opening runspace pool
2016-05-04 14:16:08,826,40,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Could not open PowerShell session using Authentication: 'Basic' User name: 'DOM1\SIQ_EX' Server: 'https://ex01/PowerShell-LiveID?PSVersion=4.0 ' Schema: 'http://schemas.microsoft.com/powershell/Microsoft.Exchange'
System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server ex01 failed with the following error message : The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more information, see the about_Remote_Troubleshooting Help topic.
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)
at WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent.openRunspacePool(RunspacePool runspacePool)
2016-05-04 14:16:08,826,40,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ExchangeCrawlerEngine,getBamConfiguration,Could not connect to Exchange server https://EX01:443/Powershell with DOM1\SIQ_EX via PowerShell
PowerShell session sample
Similarly, if you try to connect with PowerShell using this command...
$session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ex01.dom1.loc/PowerShell -Authentication Kerberos -Credential $creds
...you may see this error:
New-PSSession : [ex01.dom1.loc] Connecting to remote server ex01.dom1.loc failed with the following error message :
The server certificate on the destination computer (ex01.dom1.loc:443) has the following errors:
The SSL certificate is signed by an unknown certificate authority. For more information, see the
about_Remote_Troubleshooting Help topic.
At line:1 char:12
+ $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri ht ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotin
gTransportException
+ FullyQualifiedErrorId : 12175,PSSessionOpenFailed
Solution to the untrusted certificate problem
Exchange servers using self-signed certificates can cause this problem, which is very common in test environments. Export the certificate from the Exchange website and import it into Trusted Root Certification Authority (Local Computer) on the server where the EC is installed. Check the IIS bindings to get the display name of the certificate that should be exported for the EC.
Could not open PowerShell session using Authentication: 'Default'
In some configurations an Exchange Server might require basic authentication for connections.
- Exchange application has been added to File Access Manager and is configured to connect port 443 using SSL.
- The EC and BAM work when using non-SSL connections to Exchange on port 80.
- The Exchange server certificate is in the Trusted Root certification Authority, but now a different error is occurring.
Sample error from EC log
2016-05-06 09:13:38,654,7,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Could not open PowerShell session using Authentication: 'Default' User name: 'DOM1\SIQ_EX' Server: 'https://ex01/Powershell' Schema: 'http://schemas.microsoft.com/powershell/Microsoft.Exchange'
System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server ex01 failed with the following error message : The WinRM client received an HTTP bad request status (400), but the remote service did not include any other information about the cause of the failure. For more information, see the about_Remote_Troubleshooting Help topic.
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)
at WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent.openRunspacePool(RunspacePool runspacePool)
2016-05-06 09:13:38,680,7,DEBUG,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,initRunspacePool,Falling back to Basic authentication
2016-05-06 09:13:38,681,7,DEBUG,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Opening runspace pool
2016-05-06 09:13:38,789,7,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent,openRunspacePool,Could not open PowerShell session using Authentication: 'Basic' User name: 'DOM1\SIQ_EX' Server: 'https://ex01/PowerShell-LiveID?PSVersion=4.0 ' Schema: 'http://schemas.microsoft.com/powershell/Microsoft.Exchange'
System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server ex01 failed with the following error message : The WinRM client sent a request to an HTTP server and got a response saying the requested HTTP URL was not available. This is usually returned by a HTTP server that does not support the WS-Management protocol. For more information, see the about_Remote_Troubleshooting Help topic.
at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)
at WBX.whiteOPS.Server.RoleAnalytics.ConcurrentPowerShellComponent.openRunspacePool(RunspacePool runspacePool)
2016-05-06 09:13:38,792,7,ERROR,WBX.whiteOPS.Server.RoleAnalytics.ExchangeCrawlerEngine,getBamConfiguration,Could not connect to Exchange server https://EX01:443/Powershell with DOM1\SIQ_EX via PowerShell
Sample Error from PowerShell
Likewise, attempts to Exchnage using PowerShell with SSL and Kerberos authentication result in this output. It is misleading because the username and password in use is correct as it works on port 80/non-SSL connections:
New-PSSession : [ex01.dom1.loc] Connecting to remote server ex01.dom1.loc failed with the following error message :
The user name or password is incorrect. For more information, see the about_Remote_Troubleshooting Help topic.
At line:1 char:12
+ $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri ht ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (System.Manageme....RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException
+ FullyQualifiedErrorId
Solution
Turn on Basic authentication on the /PowerShell website. If you do not do this the PowerShell session will run using the either the Windows service account or the application pool account and will fail
/WBXPowerShell web application will not start
The EC and BAM connection to /PowerShell or /WBXPowerShell which are IIS application on the Exchange server. These applications start when the IIS gets the first request for a resource in the application. If for some reason the IIS application cannot start the EC or BAM will not get the type of message it expects.
• There is not a straightforward message in the EC or BAM log files to tell you this.
• Also the PowerShell session will fail and the messages it returns will not help you identify the problem.
• This can happen no matter what port, protocol, or credentials are in use.
EC log file or PowerShell session error message
System.Management.Automation.Remoting.PSRemotingTransportException: Connecting to remote server ex01 failed with the following error message : The WinRM client cannot process the request. It cannot determine the content type of the HTTP response from the destination computer. The content type is absent or invalid. For more information, see the about_Remote_Troubleshooting Help topic.
Windows Log File
You might see these events in the Windows event log file
ERROR 3008
Event code: 3008
Event message: A configuration error has occurred.
Event time: 3/24/2016 5:47:02 PM
Event time (UTC): 3/24/2016 9:47:02 PM
Event ID: 6ee0ff193e454be39c65230e4adf562a
Event sequence: 1
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/1/ROOT/WBXPowerShell-4-131033296219817913
Trust level: Full
Application Virtual Path: /WBXPowerShell
Application Path: E:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\WBXPowerShell\
Machine name: MYSERVERNAME
Process information:
Process ID: 7484
Process name: w3wp.exe
Account name: NT AUTHORITY\SYSTEM
Exception information:
Exception type: ConfigurationErrorsException
Exception message: Could not load file or assembly 'Hitech.SomeProduct.Exchange.SomeLibrary' or one of its dependencies. The system cannot find the file specified.
Request information:
Request URL: https://MyServerName.is.customer.com:443/wbxPowerShell
Request path: /wbxPowerShell
User host address: 10.122.0.136
User:
Is authenticated: False
Authentication Type:
Thread account name: NT AUTHORITY\SYSTEM
Thread information:
Thread ID: 1
Thread account name: NT AUTHORITY\SYSTEM
Is impersonating: False
Stack trace: at System.Web.Configuration.ConfigUtil.GetType(String typeName, String propertyName, ConfigurationElement configElement, XmlNode node, Boolean checkAptcaBit, Boolean ignoreCase)
at System.Web.Configuration.Common.ModulesEntry..ctor(String name, String typeName, String propertyName, ConfigurationElement configElement)
at System.Web.HttpApplication.BuildIntegratedModuleCollection(List`1 moduleList)
at System.Web.HttpApplication.GetModuleCollection(IntPtr appContext)
at System.Web.HttpApplication.RegisterEventSubscriptionsWithIIS(IntPtr appContext, HttpContext context, MethodInfo[] handlers)
at System.Web.HttpApplication.InitSpecial(HttpApplicationState state, MethodInfo[] handlers, IntPtr appContext, HttpContext context)
at System.Web.HttpApplicationFactory.GetSpecialApplicationInstance(IntPtr appContext, HttpContext context)
at System.Web.Hosting.PipelineRuntime.InitializeApplication(IntPtr appContext)
Solution
One reason an IIS application fails to start is that it is missing a dependency. In the case above, the web.config file of the /WBXPowerShell application has a reference to Hitech.SomeProduct.Exchange.SomeLibrary that it cannot resolve. This is not needed by WBXPowerShell and can be commented in the web.config file in the root folder of the virtual application. The path to the web.config file and the line that needed to be commented out is shown below.
Program Files\Microsoft\ExchangeServer\V14\ClientAccess\WBXPowerShell\web.config
<add name="PowerShellMonitor" type="Hitech.SomeProduct.Exchange.PowerShell.HttpHandler, Hitech.SomeProduct.Exchange.SomeLibrary"/>
Turn on Basic authentication on the /PowerShell website. If you do not do this the PowerShell session will run using the either the Windows service account or the application pool account and will fail
Error: O365 / Too many sessions
O365 allows a limited number of connections and if they are exceeded the error below is raised.
2016-05-06 10:42:26,624,23,ERROR,WBX.Common.Utilities.PowerShellUtilities,runCommands,Error: IncorrectProtocolVersion,PSSessionOpenFailed
System.Management.Automation.Remoting.PSRemotingTransportException: Processing data from remote server ps.outlook.com failed with the following error message: [ClientAccessServer=SN1PR0501CA0001,BackEndServer=dm2pr0401mb0960.namprd04.prod.outlook.com,RequestId=39a8e990-0f0d-4260-8e3c-8b20ec93d790,TimeStamp=5/6/2016 3:42:26 PM] [AuthZRequestId=f32902d1-42ed-4946-8fac-13e17f4aa5cb][FailureCategory=AuthZ-AuthorizationException] Fail to create a runspace because you have exceeded the maximum number of connections allowed : 3 for the policy party : MaxConcurrency. Please close existing runspace and try again.
Policy: CN=GlobalThrottlingPolicy_16dcaf50-eef2-42a9-a8e2-4255a75e3095,CN=Global Settings,CN=ExchangeLabs,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=namprd04,DC=prod,DC=outlook,DC=com;
Snapshot: Owner: Sid~S-1-5-21-3835325838-1622933477-1555390567-34503953~WSMan~false
BudgetType: WSMan
ActiveRunspaces: 3/3
Balance: 600000/1800000/-3000000
PowerShellCmdletsLeft: 200/200
ExchangeCmdletsLeft: 9223372036854775807/Unlimited
CmdletTimePeriod: 5
DestructiveCmdletsLeft: 9223372036854775807/Unlimited
DestructiveCmdletTimePeriod: Unlimited
QueueDepth: Unlimited
MaxRunspacesTimePeriod: 60
RunSpacesRemaining: 2/5
LastTimeFrameUpdate: 5/6/2016 3:42:26 PM
LastTimeFrameUpdateDestructiveCmdlets: 5/6/2016 3:42:17 PM
LastTimeFrameUpdateMaxRunspaces: 5/6/2016 3:42:17 PM
Locked: False
LockRemaining: 00:00:00
For more information, see the about_Remote_Troubleshooting Help topic.
Unable to retrieve mailbox folder statistics for mailbox
The permission collector is working, but is not able to collect the permissions on come mailboxes due to this error, and PowerShell commands executed from the command line give a similar result.
System.Management.Automation.RemoteException: Unable to retrieve mailbox folder statistics for mailbox SOME-MAILBOX@somedomain.com. Failure: Error code -2146233088 occurred with message The mailbox of user SOME-MAILBOX@somedomain.com that is located on a server that is running version 14 can't be opened on a server that is running version 15..
Solution
This is a problem with Exchange mailboxes that have not been properly migrated to from one version of Exchange to another. The Exchange server administrator should use the New-MoveRequest command or other method to resolve this problem.
No Exchange Server Activities in the File Access Manager Client
In this scenario there are no activities for Exchange Server showing in the File Access Manager Client and:
- You have installed the Exchange Server activity monitor and verified the configuration is correct.
- There are no errors in the File Access Manager log files.
- The debug log shows that File Access Manager is successfully running PowerShell commands to collect the audit data.
- Manually running PS commands also does not return any audit results.
This can occur if the Exchange Server indexes are corrupt and Exchange Server does not return audit results. This problem can occur in new or existing installations of Exchange server and different versions. Run the Get-MailboxDatabaseCopyStatus command and check the ContentIndexState.
screenshot - output of Get-MailboxDatabaseCopyStatus with an index problem
Follow these instructions to fix the indexes.
screenshot - output of Get-MailboxDatabaseCopyStatus with a healthy index
