Friday, August 18, 2017

Exchange 2010 / Online - cross organization permissions and access - PART 1

The subject of this blog post is cross organization permissions and access in an Exchange 2010 / Exchange Online hybrid environment. 

For my tests, I am using Exchange 2010 SP3 RU18 and Outlook 2010 SP2. I have the Exchange Online 1 Plan and a working hybrid environment (working to the extent that the hybrid configuration wizard completes successfully each time I run it). 

But first, why use Exchange 2010, a version that is approaching end of life? Because some organizations still use this version and I want to observe how Exchange Online and "Exchange Onsite" interacts in those circumstances.

Moreover, I am not using cached Exchange mode on the Outlook client. Please note: this is not recommended for access to mailboxes migrated to Exchange Online. However, many organizations implement virtual desktops which make the use of cached mode difficult (because of the potentially large .ost files), and especially if the desktops are "non-persistent". One of my objectives (not necessarily addressed in the present text) is to evaluate differences in access time with and without cached mode. I would not think this setting affects permissions in any way but switching to cached mode may be considered.


First scenarios: migrated user attempts to access on-premises elements.

  • Access mailbox of another user (still on-premises)
  • Access shared mailbox (on-premises)
  • Access calendar (on-premises)

We have a user named Aisha Bhari whose mailbox has already been migrated to Office 365. At this point, this is the only mailbox on Office 365. Every other mailbox remains on-premises.

We will test these scenarios:

Grant Aisha Bhari Full Permission access to on-premises mailbox of Alannah Shaw

Result: Aisha can add the mailbox but cannot expand the folders:

Note: "Use Cached Exchange Mode" is unchecked.

I attempted the following for troubleshooting:

  • Remove and re-add the mailbox? Still fails.
  • Wait a couple days for permissions to take effect? Still fails.

Grant Aisha Bhari access to the Finance shared mailbox through membership in a security group granted Full Access.

Can she access the shared mailbox?


What if we grant her permissions directly (bypassing the security group).

The attempt still fails.

Can Aisha Bhari access the calendar of Alannah Shaw?

She can open the calendar but only after 2-3 attempts.

Usually, on the first attempt, there is a delay when "connecting" and the attempt fails. We have to try until the connection is made, apparently within the timeout limit. This most likely is related to the greater latency when directly accessing mailboxes (and associated calendars) online.

Moreover, we can only see the indication "Busy" on the test meeting (but not even limited details):

Granting more permission has no effect.

Send calendar invite to Aisha Bhari

This fails. I cannot even send the invitation:

So something is not quite right.

I take some further troubleshooting steps.

Various sources indicate that the following updates for Outlook are necessary but it looks like they are already installed or not pertinent:

  • KB2956191 - already installed according to the installer
  • KB2965295 - "There are no products affected by this package installed on the system" according to the installer.
  • KB3114409 - "There are no products affected by this package installed on the system" according to the installer.

I used the Exchange Remote Connectivity Analyzer (EXRCA) which first failed because Outlook Anywhere was not enabled (it had been disabled for testing MAPI/HTTP with Outlook 2010). Once Outlook Anywhere was (re)enabled, the following EXRCA tests passed:
  • Outlook Connectivity
  • Outlook Autodiscover
  • Exchange Web Services

What effect did enabling Outlook Anywhere have on the cross organization permissions and access tested earlier?

At first nothing. Nothing changed and Aisha Bhari could still not access the resources in question.

Some time later, I retried and Aisha Bhari is now able to expand the mailbox of Alannah Shaw in her Outlook profile (Aisha Bhari's profile). In the past, I have observed that new Exchange permissions may need some time (even days) before taking effect. So I thought I would grant Aisha access to another mailbox (that of Alan Reid) and see if she could add it to her profile immediately (within minutes) and that was possible.

Moreover, Aisha is now able to see the meeting details of appointments in Alannah Shaw's calendar.

On the other hand, she still cannot add a shared mailbox or calendar to her profile (or even locate them in the GAL - only users appear).

Also, other users still cannot send her an invitation to view their calendar. Note: she belongs to a group that has "Publishing Editor" rights.

After some additional reflection, I realize the nature of the problem.

I use Azure AD Connect to synchronize users and groups to Office 365. I can see the synchronized users in the Office 365 Admin Center:


Because I filtered the objects to be synchronized to Office 365 by organizational unit (OU), only the users in the "ExchangeUsers" OU have been synchronized:

If I select the OU that contains the accounts associated with the shared mailboxes (no screenshot) and force synchronization, the shared mailboxes (and calendars) are now represented in Office 365:

The results are significant.

Aisha Bhari can not only access the mailbox of Alannah Shaw, as already stated...

But she can also open the Finance shared mailbox (now that we have synchronized the associated accounts to Office 365):

Moreover, Aisha can now locate and add the MktCal calendar...

She cannot open this appointment...

But that is due to lack of sufficient permissions rather than a faulty configuration of our hybrid environment:

When Aisha does have "Full Permission", she can view meeting details (in the calendar of Alannah Shaw)...

And in the FinCal calendar:


After some adjustments, our migrated user can access another user's mailbox, a shared mailbox and a calendar, all still located on-premises (and provided they have appropriate access rights to these resources of course). There is still much to test. We've looked at "Full Permission" but what about "Send As" and "Send on behalf of"? Also, access was granted after Aisha Bhari was migrated to Office 365. What about an on-premises user who already has access to other mailboxes and calendars? Will the permissions be retained after the migration of their mailbox? Lastly, what about users on-premises that attempt to access resources in the Cloud? These are scenarios that I plan to examine (not neccesarily in that order) in future blog posts.


Understanding Hybrid Deployment Permissions with Exchange 2010 SP3

Exchange Hybrid Cross-Premises Mailbox Permissions Demystified (Part 1)

Outlook Anywhere must be enabled

Run Microsoft Exchange Hybrid for the long haul

At 1:14:00 in the video

Monday, July 31, 2017

Exchange 2010 (SP3) - troubleshooting POP3 SSL

POP3 is not the most commonly used client access protocol with Exchange but one we could still encounter as an Exchange administrator. I recently had to troubleshoot an application that used POP3 to access an Exchange mailbox with the option to secure the connection with SSL (so port 995 rather than port 110, used for unencrypted connections). In the end, we discovered that the problem had nothing to do with Exchange or the certificate. However, to come to that concluson, we had to demonstrate that connections to port 995 could be made by other methods, notably OpenSSL and Thunderbird (which can function as a POP3 client). In this blog post, I want to present how we established connections from a Windows 7 client with OpenSSL and enabled the logging of these connection on the Exchange side.


If you want to reproduce this in a test environment, you may have to enable POP3 and then configure it to use SSL, so:
  1. Enable the POP3 service.
  2. Associate POP3 with a certificate for secure communication with SSL.
I will not present these operations (above) step by step. I would refer you to some other articles by Paul Cunningham (Exchange MVP) here:

The certificate I use in my test network has several subject alternative names ("mail", "autodiscover") but not "pop", so I will use "mail" instead, although in practice we would probably use "pop" for clarity. I notice that when I select the option "Secure Logon" for the POP connection the correct certificate was located automatically.

We can test a POP3 connection without SSL using Telnet (on port 110) but what if we want to test POP3 with SSL (on port 995)?

I downloaded and installed OpenSSL for Windows.

Several download option are presented here:

Open SSL binaries

I then attempt to connect on port 995 with this command:

openssl s_client -crlf -connect

As we can see in the screenshots below, the connection is successful (despite the warning "unable to get local issuer certificate"):

If we need to troubleshoot POP3, we can enable logging on the Exchange server with this command:

Set-PopSettings -ProtocolLogEnabled $true

At first, there is no POP3 subfolder in the Logging folder:

We must restart the POP3 service:

Once we attempt to connect, a file is created in the POP3 folder with content similar to this:

It may be useful also to increase the level of logging in the Event Viewer for POP3. We do this in the properties of the Exchange server. The default value is shown below:

We could place the setting at "Expert", for example:

Note the PowerShell equivalent:

In practice, we should be prudent with the logging level because a very high level (like "Expert") could flood the logs with a plethora of events and even have adverse effects on performance.

When finished troubleshooting, we must remember to turn off logging or set it at an acceptable level for our environment. Available disk space would be one aspect to consider.

Tuesday, July 25, 2017

Exchange 2010 SP3 - Search-AdminAuditLog (2)

During my previous research on the Search-AdminAuditLog cmdlet, I read some claims that only changes made in the EMC (Exchange Management Console) and EMS (Exchange Management Shell) are audited. If we open PowerShell itself, import the Exchange snap-ins and then execute the command in question, it will not appear in the logs.

"Unfortunately, this does not address the issue that the admin audit logs do not actually record everything. If you open Windows PowerShell and load the Exchange module nothing you do in or to Exchange will be in the admin log. [...] if the purpose of the log is to help you monitor what someone has done it seems to be a big hole."

Apparently the same person makes the claim again here:

"Unfortunately, not everything done by an Admin is logged. If you open a Windows PowerShell window and load the Exchange module nothing is recorded in the audit logs. Only actions performed in the EMS are recorded."

This seems like a surprising oversight and if exact would make auditing almost useless. Any administrator aware of this shortcoming could execute their commands in PowerShell (with the Exchange snap-ins imported) and leave no tracks for auditing.

But is the claim exact?

What does Microsoft have to say?

"Cmdlets that are run directly in the Exchange Management Shell are audited. In addition, operations that are performed by using the Exchange Management Console (EMC) and the Exchange Web management interface are also logged because those operations run cmdlets in the background."

Nothing is mentioned about cmdlets executed directly in PowerShell (after the Exchange snap-in is loaded).

If we open PowerShell and load the Exchange module, rather than using the EMS, this apparently bypasses RBAC (although I'm not sure if that has any effect as far as auditing is concerned).

This is also stated in Paul Cunningham's article (and ensuing discussion).

In any event, this is something worth testing.


First, I'm testing with Exchange 2010 SP3 RU 15. I may or may not test with other versions of Exchange (possibly 2016). My PowerShell version is 2:

We can enable admin auditing with this command...

Set-AdminAuditLogConfig -AdminAuditLogEnabled $True

But since Exchange 2010 SP1, it is enabled by default.

Indeed, that is the case on my test server:

So let's open PowerShell and load the Exchange module:

In our experiment, the administrator "XADMIN" will grant himself permissions to the mailbox of user Alan Reid.

Right now, these are the permissions on the mailbox:

XADMIN grants himself Full Access to the mailbox with this command:

Before testing the Search-AdminAuditLog cmdlet, I want to see if the action is simply visible in the Event Viewer (MSExchange Management log). I open Find and enter the cmdlet I am seeking (Add-MailboxPermission). The only results are an entry for the Search-AdminAuditLog cmdlet (produced in an earlier experiment) and...

... the entry created during that earlier experiment when XADMIN granted himself Full Access to the mailbox of Alex Heyne (but not Alan Reid):

There is no other entry in the Event Viewer for the Add-MailboxPermission cmdlet:

Note: could this be due to a simple delay? First, there is nothing else happening on my test network (no one else connected, no incoming or outgoing messages to process). Second, I performed the search a second time five days later (when I started the composition of this blog post) and there was still no entry for the access granted to the mailbox of Alan Reid.

Yet, when I execute the Search-AdminAuditLog cmdlet, there is an entry for the action (as well as the earlier event concerning the mailbox of Alex Heyne):

So we do have a trace of XADMIN granting himself Full Access to the mailbox of Alan Reid (and earlier to Alex Heyne's mailbox) but if it cannot be found in the Event Viewer... then where is it coming from?

The admin audit logs are stored in a folder of a system mailbox that is not displayed in the EMC.

We can see the system mailboxes with this command:

Get-Mailbox -Arbitration

However, there are two system mailboxes. Which one holds the admin audit logs?

I thought these properties might distingush the mailbox in question...

But not really...

I thought we could use the Get-MailboxFolderStatistics cmdlet to view the different folders of each of the system mailboxes - but first I want to set a variable to represent the mailbox (and avoid having to deal with the long name).

$SM1 = Get-Mailbox -Arbitration "SystemMailbox{1f05a927-dd92-45c6-8e7e-3ee6d8fdb1e4}"

$SM2 = Get-Mailbox -Arbitration "SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}"

Note: the cmdlet did not work if I did not specify -Arbitration

There is nothing about admin audit logs in the first of the two system mailboxes.

$SM1 | Get-MailboxFolderStatistics | fl name

I have better luck with the second system mailbox:

$SM2 | Get-MailboxFolderStatistics | fl name

There is probably a better method to reach the same result: there were a lot of unnecessary folders listed with the commands above, so many that I've opted not to post the excessive output.

Knowing that the pertinent folders contain the word "audit", I'll attempt the following:

[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -contains "audit"}
[PS] C:\> [Note: no results here]
[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -match "audit"}

The last command above displays the folders in question but with each and every property.

This is much neater:

[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -match "audit"} | fl Name

Name : AdminAuditLogSearch

Name : MailboxAuditLogSearch

Name : AdminAuditLogs

In any event, we have located where the admin audit logs are located.


Another question arises in my mind: what if we purge the MSExchange Management event log?

I would predict this would NOT delete the admin audit logs since they are not stored there in the first place. Unless by some mechanism (?) the command to purge the MSExchange Management log also acts on the Exchange admin audit logs in the system mailbox.

First, I want to provide these additional details on the system mailbox:

I then purge the log on EX13-2:

There is no change. There are still 49 items in the folder and I can still find examples of the Add-MailboxPermission cmdlet (using the Search-AdminAuditLog cmdlet).

Remember that the system mailbox in question resides on EX13-1 (at least for the time being, since the mailbox is part of a DAG and could become active on EX13-2).

What if I purge the MSExchange Management log on EX13-1? As far as the audit logs are concerned, nothing. There are still 49 items in the folder and we can still retrieve examples of use of the Add-MailboxPermission cmdlet with Search-AdminAuditLog.

We should also remember that if a log is cleared, that too creates an entry (in the System log):


So we have answered two questions. First, Exchange cmdlets executed in PowerShell (after importing the Exchange module) are not logged in the MSExchange Management log but are logged in the admin audit logs. Second, the admin audit logs are stored in the folder of a system mailbox. Even if the MSExchange Management log is cleared, the admin audit logs remain.

Sunday, July 16, 2017

Exchange 2010 SP3 - Search-AdminAuditLog (1)

After my previous blog posts on Exchange 2016 (in French), this post marks both a return to English (probably brief) and a return to Exchange 2010. I still manage Exchange 2010 servers and recently needed to examine the auditing of changes to mailbox permissions made by administrators (a subject that I've covered in the past as well). The context? Troubleshooting a third-party auditing tool that uses the PowerShell cmdlet Search-AdminAuditLog. Pending a solution to the problem in question, I wanted to see to what extent the same results could be achieved with native Exchange tools.


First, I'll present the scenario.

The mission is to create a report that will list changes made to mailbox permissions with the Add-MailboxPermission cmdlet (and optionally the Remove-MailboxPermission cmdlet). We should remember that even if we make changes in the Exchange Management Console (EMC), or Exchange Admin Center (EAC) in later versions of Exchange, these interfaces use PowerShell cmdlets behind the scenes to execute the desired action. I will also point out that by default administrators are denied access to user mailboxes - unless they execute the Add-MailboxPermission cmdlet that we will audit. Once created, the report should be sent to a designated auditor.

Next, I will execute the command and observe how Exchange records the action in the Event Viewer logs.

For example, the administrator "xadmin" grants himself Full Access to the mailbox of Alex Heyne:

Note: yes, you can click on the image to enlarge it.

Such an action is recorded in the MSExchange Management log of the Event Viewer :

Almost all actions performed by administrators are recorded in this log so we might have to use the "Find" tool to locate the actions that interest us, for example:

The action is only recorded on the server where it was executed (EX13-2 in this case). If we search for the action on EX13-1, there will be no result:

However, our objective is to discover any use of the Add-MailboxPermission cmdlet rather than locating single examples of the event in the MSExchange Management log. For this, we need to use another cmdlet: Search-AdminAuditLog.

I can execute the command with the expected results on EX13-2:

I also notice that the Search-AdminAuditLog cmdlet apparently examines the logs of other servers, since I obtain the same results on EX13-1 (note that the "OriginatingServer is still EX13-2):

We still do not have our report but we are on the right track. Indeed, the reporting function of the third-party software does use the Search-AdminAuditLog to retrieve the actions to be audited.


We are on the right track but I already notice something missing in the results of the Search-AdminAuditLog cmdlet. We see ...

  • Object modified (Alex.Heyne)
  • CmdletName (Add-MailboxPermission)
  • Caller (XADMIN)

So XADMIN executed the command Add-MailboxPermission against the mailbox of Alex.Heyne.

But what permissions were granted and to whom? Himself? Someone else?

We should keep in mind that the entry in the Event Viewer (shown in one of the screenshots above) does indicate these details.

After some research, I discovered that the information we want is present but not displayed but default. We are supposed to observe the property "CmdletParameters" with (in braces) "Identity", "User" and "Access rights" and realize this is a hash-table containing "name-value" pairs. We need to extract these using an array.

I used this article as a reference but was not able to obtain the same results:

Next, I attempted to expand the CmdletParameters property (successfully) but the output was not what I would have liked. I will not present all the variations attempted but in the end, this was the most readable:

So, for a single action (XADMIN grants himself Full Access to the mailbox of Alex Heyne), we have three separate entries. The auditor would have to be able to see the relationship between the three items that (by default) are not united in a single entity (unlike the presentation in the Event Viewer, once again).

While I could not create an array as described in the article cited earlier...

Parsing the Admin Audit Logs with PowerShell

It did include a script by Matthew Byrd that is supposed to format the output of the Search-AdminAuditLog so it is easier to interpret. I thought I would give this a try.


First, we download the script from the TechNet Gallery (Script Center):

Get-SimpleAuditLogReport script

Second, we declare a variable and assign it the output of the Search-AdminAuditLog as shown in the screenshot below.

Third, we pipeline the result to the Get-SimpleAdminAuditLog.ps1 script:

Note: click to enlarge.

This is much better! In the FullCommand field, we see the exact command that was executed. The auditor still needs to understand what that command accomplishes but no longer has to determine the relationship between three separate items as before.

But can we export the content to a file?

Yes we can:

And can we send the file to the auditor? Yes! In my case, I had to create a script for this (simple text to be saved and then executed as a .ps1 file):

The operation is successful and the auditor (we'll pretend user Alan Reid holds this role) does indeed receive the .csv attachment:

However, some may consider that the presentation is not the best:


I thought that HTML might allow for a more attractive and perhaps even readable presentation.

First, this is what I tried:

You might want to ignore what is inside the orange frame. Without further formatting, the output is really no better than our .csv file. With the -head parameter (please observe the details in the screenshot above), we can create a document like this:

We can combine all the elements so the file is sent to the auditor:

Note: we can also configure the Task Scheduler to run the script at the interval of our choice.


With some instruction, our auditor should be able to make sense of the HTML file. We can add other events as desired after the -Cmdlets section of the script. All in all, it seems like an acceptable solution if we do not have a third-party tool (or if our third-party tool stops functioning after an insufficiently tested update...).

And giving credit where credit is due...

Besides the text already cited twice above, I also used the following resources to construct my script(s):

Windows PowerShell Tip of the Week - ConvertTo-Html

Trying to pipe Export-Csv to Send-MailMessage