Monday, July 31, 2017

Exchange 2010 (SP3) - troubleshooting POP3 SSL

POP3 is not the most commonly used client access protocol with Exchange but one we could still encounter as an Exchange administrator. I recently had to troubleshoot an application that used POP3 to access an Exchange mailbox with the option to secure the connection with SSL (so port 995 rather than port 110, used for unencrypted connections). In the end, we discovered that the problem had nothing to do with Exchange or the certificate. However, to come to that concluson, we had to demonstrate that connections to port 995 could be made by other methods, notably OpenSSL and Thunderbird (which can function as a POP3 client). In this blog post, I want to present how we established connections from a Windows 7 client with OpenSSL and enabled the logging of these connection on the Exchange side.


***


If you want to reproduce this in a test environment, you may have to enable POP3 and then configure it to use SSL, so:
  1. Enable the POP3 service.
  2. Associate POP3 with a certificate for secure communication with SSL.
I will not present these operations (above) step by step. I would refer you to some other articles by Paul Cunningham (Exchange MVP) here:




The certificate I use in my test network has several subject alternative names ("mail", "autodiscover") but not "pop", so I will use "mail" instead, although in practice we would probably use "pop" for clarity. I notice that when I select the option "Secure Logon" for the POP connection the correct certificate was located automatically.




We can test a POP3 connection without SSL using Telnet (on port 110) but what if we want to test POP3 with SSL (on port 995)?

I downloaded and installed OpenSSL for Windows.

Several download option are presented here:

Open SSL binaries


I then attempt to connect on port 995 with this command:

openssl s_client -crlf -connect mail.mitserv.net:995


As we can see in the screenshots below, the connection is successful (despite the warning "unable to get local issuer certificate"):






If we need to troubleshoot POP3, we can enable logging on the Exchange server with this command:

Set-PopSettings -ProtocolLogEnabled $true



At first, there is no POP3 subfolder in the Logging folder:




We must restart the POP3 service:




Once we attempt to connect, a file is created in the POP3 folder with content similar to this:




It may be useful also to increase the level of logging in the Event Viewer for POP3. We do this in the properties of the Exchange server. The default value is shown below:



We could place the setting at "Expert", for example:



Note the PowerShell equivalent:




In practice, we should be prudent with the logging level because a very high level (like "Expert") could flood the logs with a plethora of events and even have adverse effects on performance.

When finished troubleshooting, we must remember to turn off logging or set it at an acceptable level for our environment. Available disk space would be one aspect to consider.

Tuesday, July 25, 2017

Exchange 2010 SP3 - Search-AdminAuditLog (2)

During my previous research on the Search-AdminAuditLog cmdlet, I read some claims that only changes made in the EMC (Exchange Management Console) and EMS (Exchange Management Shell) are audited. If we open PowerShell itself, import the Exchange snap-ins and then execute the command in question, it will not appear in the logs.


"Unfortunately, this does not address the issue that the admin audit logs do not actually record everything. If you open Windows PowerShell and load the Exchange module nothing you do in or to Exchange will be in the admin log. [...] if the purpose of the log is to help you monitor what someone has done it seems to be a big hole."



Apparently the same person makes the claim again here:

"Unfortunately, not everything done by an Admin is logged. If you open a Windows PowerShell window and load the Exchange module nothing is recorded in the audit logs. Only actions performed in the EMS are recorded."



This seems like a surprising oversight and if exact would make auditing almost useless. Any administrator aware of this shortcoming could execute their commands in PowerShell (with the Exchange snap-ins imported) and leave no tracks for auditing.


But is the claim exact?

What does Microsoft have to say?

"Cmdlets that are run directly in the Exchange Management Shell are audited. In addition, operations that are performed by using the Exchange Management Console (EMC) and the Exchange Web management interface are also logged because those operations run cmdlets in the background."

Nothing is mentioned about cmdlets executed directly in PowerShell (after the Exchange snap-in is loaded).

If we open PowerShell and load the Exchange module, rather than using the EMS, this apparently bypasses RBAC (although I'm not sure if that has any effect as far as auditing is concerned).


This is also stated in Paul Cunningham's article (and ensuing discussion).

In any event, this is something worth testing.


***


First, I'm testing with Exchange 2010 SP3 RU 15. I may or may not test with other versions of Exchange (possibly 2016). My PowerShell version is 2:




We can enable admin auditing with this command...


Set-AdminAuditLogConfig -AdminAuditLogEnabled $True


But since Exchange 2010 SP1, it is enabled by default.

Indeed, that is the case on my test server:



So let's open PowerShell and load the Exchange module:



In our experiment, the administrator "XADMIN" will grant himself permissions to the mailbox of user Alan Reid.

Right now, these are the permissions on the mailbox:



XADMIN grants himself Full Access to the mailbox with this command:



Before testing the Search-AdminAuditLog cmdlet, I want to see if the action is simply visible in the Event Viewer (MSExchange Management log). I open Find and enter the cmdlet I am seeking (Add-MailboxPermission). The only results are an entry for the Search-AdminAuditLog cmdlet (produced in an earlier experiment) and...



... the entry created during that earlier experiment when XADMIN granted himself Full Access to the mailbox of Alex Heyne (but not Alan Reid):



There is no other entry in the Event Viewer for the Add-MailboxPermission cmdlet:



Note: could this be due to a simple delay? First, there is nothing else happening on my test network (no one else connected, no incoming or outgoing messages to process). Second, I performed the search a second time five days later (when I started the composition of this blog post) and there was still no entry for the access granted to the mailbox of Alan Reid.


Yet, when I execute the Search-AdminAuditLog cmdlet, there is an entry for the action (as well as the earlier event concerning the mailbox of Alex Heyne):



So we do have a trace of XADMIN granting himself Full Access to the mailbox of Alan Reid (and earlier to Alex Heyne's mailbox) but if it cannot be found in the Event Viewer... then where is it coming from?

The admin audit logs are stored in a folder of a system mailbox that is not displayed in the EMC.

We can see the system mailboxes with this command:

Get-Mailbox -Arbitration



However, there are two system mailboxes. Which one holds the admin audit logs?

I thought these properties might distingush the mailbox in question...



But not really...

I thought we could use the Get-MailboxFolderStatistics cmdlet to view the different folders of each of the system mailboxes - but first I want to set a variable to represent the mailbox (and avoid having to deal with the long name).

$SM1 = Get-Mailbox -Arbitration "SystemMailbox{1f05a927-dd92-45c6-8e7e-3ee6d8fdb1e4}"

$SM2 = Get-Mailbox -Arbitration "SystemMailbox{e0dc1c29-89c3-4034-b678-e6c29d823ed9}"

Note: the cmdlet did not work if I did not specify -Arbitration

There is nothing about admin audit logs in the first of the two system mailboxes.

$SM1 | Get-MailboxFolderStatistics | fl name

I have better luck with the second system mailbox:

$SM2 | Get-MailboxFolderStatistics | fl name

There is probably a better method to reach the same result: there were a lot of unnecessary folders listed with the commands above, so many that I've opted not to post the excessive output.

Knowing that the pertinent folders contain the word "audit", I'll attempt the following:

[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -contains "audit"}
[PS] C:\> [Note: no results here]
[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -match "audit"}

The last command above displays the folders in question but with each and every property.

This is much neater:

[PS] C:\>$SM2 | Get-MailboxFolderStatistics | where {$_.Name -match "audit"} | fl Name

Name : AdminAuditLogSearch

Name : MailboxAuditLogSearch

Name : AdminAuditLogs



In any event, we have located where the admin audit logs are located.


***


Another question arises in my mind: what if we purge the MSExchange Management event log?

I would predict this would NOT delete the admin audit logs since they are not stored there in the first place. Unless by some mechanism (?) the command to purge the MSExchange Management log also acts on the Exchange admin audit logs in the system mailbox.

First, I want to provide these additional details on the system mailbox:



I then purge the log on EX13-2:






There is no change. There are still 49 items in the folder and I can still find examples of the Add-MailboxPermission cmdlet (using the Search-AdminAuditLog cmdlet).

Remember that the system mailbox in question resides on EX13-1 (at least for the time being, since the mailbox is part of a DAG and could become active on EX13-2).

What if I purge the MSExchange Management log on EX13-1? As far as the audit logs are concerned, nothing. There are still 49 items in the folder and we can still retrieve examples of use of the Add-MailboxPermission cmdlet with Search-AdminAuditLog.

We should also remember that if a log is cleared, that too creates an entry (in the System log):



***


So we have answered two questions. First, Exchange cmdlets executed in PowerShell (after importing the Exchange module) are not logged in the MSExchange Management log but are logged in the admin audit logs. Second, the admin audit logs are stored in the folder of a system mailbox. Even if the MSExchange Management log is cleared, the admin audit logs remain.


Sunday, July 16, 2017

Exchange 2010 SP3 - Search-AdminAuditLog (1)

After my previous blog posts on Exchange 2016 (in French), this post marks both a return to English (probably brief) and a return to Exchange 2010. I still manage Exchange 2010 servers and recently needed to examine the auditing of changes to mailbox permissions made by administrators (a subject that I've covered in the past as well). The context? Troubleshooting a third-party auditing tool that uses the PowerShell cmdlet Search-AdminAuditLog. Pending a solution to the problem in question, I wanted to see to what extent the same results could be achieved with native Exchange tools.


***

First, I'll present the scenario.

The mission is to create a report that will list changes made to mailbox permissions with the Add-MailboxPermission cmdlet (and optionally the Remove-MailboxPermission cmdlet). We should remember that even if we make changes in the Exchange Management Console (EMC), or Exchange Admin Center (EAC) in later versions of Exchange, these interfaces use PowerShell cmdlets behind the scenes to execute the desired action. I will also point out that by default administrators are denied access to user mailboxes - unless they execute the Add-MailboxPermission cmdlet that we will audit. Once created, the report should be sent to a designated auditor.

Next, I will execute the command and observe how Exchange records the action in the Event Viewer logs.

For example, the administrator "xadmin" grants himself Full Access to the mailbox of Alex Heyne:


Note: yes, you can click on the image to enlarge it.



Such an action is recorded in the MSExchange Management log of the Event Viewer :




Almost all actions performed by administrators are recorded in this log so we might have to use the "Find" tool to locate the actions that interest us, for example:





The action is only recorded on the server where it was executed (EX13-2 in this case). If we search for the action on EX13-1, there will be no result:




However, our objective is to discover any use of the Add-MailboxPermission cmdlet rather than locating single examples of the event in the MSExchange Management log. For this, we need to use another cmdlet: Search-AdminAuditLog.

I can execute the command with the expected results on EX13-2:



I also notice that the Search-AdminAuditLog cmdlet apparently examines the logs of other servers, since I obtain the same results on EX13-1 (note that the "OriginatingServer is still EX13-2):




We still do not have our report but we are on the right track. Indeed, the reporting function of the third-party software does use the Search-AdminAuditLog to retrieve the actions to be audited.



***


We are on the right track but I already notice something missing in the results of the Search-AdminAuditLog cmdlet. We see ...

  • Object modified (Alex.Heyne)
  • CmdletName (Add-MailboxPermission)
  • Caller (XADMIN)

So XADMIN executed the command Add-MailboxPermission against the mailbox of Alex.Heyne.

But what permissions were granted and to whom? Himself? Someone else?

We should keep in mind that the entry in the Event Viewer (shown in one of the screenshots above) does indicate these details.

After some research, I discovered that the information we want is present but not displayed but default. We are supposed to observe the property "CmdletParameters" with (in braces) "Identity", "User" and "Access rights" and realize this is a hash-table containing "name-value" pairs. We need to extract these using an array.

I used this article as a reference but was not able to obtain the same results:




Next, I attempted to expand the CmdletParameters property (successfully) but the output was not what I would have liked. I will not present all the variations attempted but in the end, this was the most readable:



So, for a single action (XADMIN grants himself Full Access to the mailbox of Alex Heyne), we have three separate entries. The auditor would have to be able to see the relationship between the three items that (by default) are not united in a single entity (unlike the presentation in the Event Viewer, once again).

While I could not create an array as described in the article cited earlier...

Parsing the Admin Audit Logs with PowerShell

It did include a script by Matthew Byrd that is supposed to format the output of the Search-AdminAuditLog so it is easier to interpret. I thought I would give this a try.


***


First, we download the script from the TechNet Gallery (Script Center):

Get-SimpleAuditLogReport script

Second, we declare a variable and assign it the output of the Search-AdminAuditLog as shown in the screenshot below.

Third, we pipeline the result to the Get-SimpleAdminAuditLog.ps1 script:


Note: click to enlarge.

This is much better! In the FullCommand field, we see the exact command that was executed. The auditor still needs to understand what that command accomplishes but no longer has to determine the relationship between three separate items as before.

But can we export the content to a file?

Yes we can:



And can we send the file to the auditor? Yes! In my case, I had to create a script for this (simple text to be saved and then executed as a .ps1 file):





The operation is successful and the auditor (we'll pretend user Alan Reid holds this role) does indeed receive the .csv attachment:



However, some may consider that the presentation is not the best:




***


I thought that HTML might allow for a more attractive and perhaps even readable presentation.

First, this is what I tried:



You might want to ignore what is inside the orange frame. Without further formatting, the output is really no better than our .csv file. With the -head parameter (please observe the details in the screenshot above), we can create a document like this:



We can combine all the elements so the file is sent to the auditor:



Note: we can also configure the Task Scheduler to run the script at the interval of our choice.


 ***


With some instruction, our auditor should be able to make sense of the HTML file. We can add other events as desired after the -Cmdlets section of the script. All in all, it seems like an acceptable solution if we do not have a third-party tool (or if our third-party tool stops functioning after an insufficiently tested update...).

And giving credit where credit is due...

Besides the text already cited twice above, I also used the following resources to construct my script(s):

Windows PowerShell Tip of the Week - ConvertTo-Html

Trying to pipe Export-Csv to Send-MailMessage







Saturday, July 8, 2017

Exchange 2016 (25) : le mode maintenance pour le DAG (2)

English summary: in my previous blog post, I outlined the procedure to place our Exchange DAG in maintenance mode. The post became rather lengthy because I not only presented the steps to follow but also verified the effect of the commands and resolved differences between the sources that I consulted. I opted to save the presentation on taking the DAG out of maintenance mode for a later blog post (here). In fact, this presentation proved to be quite short and I realize I could have just added to the previous blog post without increasing its length that much more. Finally, I decided to post the procedure here, in the few paragraphs below.


***

Dans mon texte précédent, j'ai examiné la marche à suivre pour mettre notre DAG (Database Availability Group) en mode maintenance  selon la nouvelle méthode pour les versions d'Exchange 2013 et 2016. Le texte s'est allongé parce que j'ai non seulement présenté la marche à suivre mais vérifié l'effet des commandes (et résolu des différences entre les sources que j'ai consultées pour la rédaction du texte). J'ai donc remis à plus tard (à ce texte) la marche à suivre pour sortir le DAG du mode maintenance.

Note : est-ce un abus du langage de dire "mettre le DAG en mode maintenance" ou l'en "sortir" ? Je me pose la question dans la mesure où ce n'est pas le DAG entier qui passe en mode maintenance mais seulement un des serveurs qui en est membre à la fois.

Je vois aussi qu'on se sert des termes "mise hors ligne" et "remise en ligne" du DAG (en fait, du serveur membre en question, comme nous venons de le préciser). Quand j'ai remis notre DAG en ligne, grâce à une demi-dizaine de commandes PowerShell, je me suis rendu compte qu'avec un peu de persistance j'aurai pu attacher la description de cette procédure à la fin du texte précédent, sans vraiment l'allonger davantage. Mais ce qui est fait est fait. La remise en ligne du DAG fera l'objet de ce texte, beaucoup plus court que le premier.

Pour sortir le DAG du mode maintenance, il s'agit d'exécuter les commandes suivantes (dans l'ordre), gardant à l'esprit que c'est le serveur EX16-4 que nous avons mis en mode maintenance :

Set-ServerComponentState EX16-4 -Component ServerWideOffline -State Active -Requester Maintenance

Resume-ClusterNode EX16-4

Set-MailboxServer EX16-4 -DatabaseCopyActivationDisabledAndMoveNow $False

Set-MailboxServer EX16-4 -DatabaseCopyAutoActivationPolicy Unrestricted

Set-ServerComponentState EX16-4 -Component HubTransport -State Active -Requester Maintenance

Voici à quoi cela ressemble :



Et c'est tout.

Bien entendu, nous ferions bien de vérifier l'état du DAG, de la réplication et des bases de données avec les mêmes commandes que nous avons utilisées dans le texte précédent, soit :

Get-DatabaseAvailabilityGroup DAG1 -status | format-list

Get-MailboxDatabase | Get-MailboxDatabaseCopyStatus

Test-ReplicationHealth