Windows Server 2012 R2 Cluster Node issues

I recently came across an error where nodes within a Windows Server 2012 R2 Cluster who stopped being active nodes within the cluster and continually cycled round trying to join again.

We were seeing errors:

Event 1070: Failover cluster nodes must have the ability to start the Cluster service, form a cluster (when a given node starts but no other nodes are up) and join a cluster (when a given node starts and discovers that one or more nodes are already up). This requires that certain conditions be met, for example, failover cluster nodes must run compatible versions of the operating system.

Event 1145: Cluster resource <resource> timed out. If the pending timeout is too short for this resource consider increasing the pending timeout value.

Within the event log over and over again.

We did some basic troubleshooting to check network connectivity, configuration etc.  One of our troublshooting steps included evicting a node and using the Clear-ClusterNode Powershell command to try and clear any configuration issues on the node.   When we tried to add the node back to the cluster we were presented with a new error "Event ID: 7024 The Cluster Service service terminated with the following service-specific error: Keyset does not exist".

We did some more digging and found that the permissions on the folder and files within C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys were largely missing.  This is the folder that holds the certificate keys that the cluster uses to connect.  Rather than change all the 38 files individually within the folder manually we came up with this wee script:

##This grants ownership of the folder and files below it to the administrator group.
takeown /f C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys /R /A

##This grants the System and Administrators accounts Full Access to the machinekey folder and all it's subfolders/files, and removes any inherited permissions
icacls "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys" /INHERITANCE:R /GRANT ("SYSTEM" + ':(CI)(OI)F')
icacls "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys" /INHERITANCE:R /GRANT ("Administrators" + ':(CI)(OI)F') 

Once the permissions were set as above we were able to successfully add the node back into the cluster. And all four nodes were active again.

Retention Policy Powershell Commands

I've recently been working on applying retention policies to mailboxes and we've been doing a granular roll-out to the users.  I've been using some Powershell commands to track who has the policy applied and who hasn't.

The following command will list all users who have a retention policy applied to their mailbox:

Get-Mailbox -filter 'RetentionPolicy -like "*"' | Select-Object Name, RetentionPolicy

You can export that to a CSV if need be by using:

 Get-Mailbox -filter 'RetentionPolicy -like "*"' | Select-Object Name, RetentionPolicy | Export-CSV C:\directory\file.csv

To identify which users don't have a retention policy applied to their mailbox the following command will assist:

Get-Mailbox -filter 'RetentionPolicy -eq $null' | Select-Object Name, RetentionPolicy

Again to export that to CSV you would use:

Get-Mailbox -filter 'RetentionPolicy -eq $null' | Select-Object Name, RetentionPolicy | Export-CSV C:\Directory\file.csv 

Clearing out move requests

When you perform a successful mailbox migration, either from another version of Exchange or from another mailbox database the move request is "left" behind as such and should be cleared out as part of good housekeeping.

My favourite command for doing this is:

Get-MoveRequest -MoveStatus Completed | Remove-MoveRequest

Unable to sync phone to Exchange

If you've been following my blog for the last few weeks you'll have heard me talk about the Exchange 2013 migration project I've been working on.  Well we hit another snag in the migration the other day around the users mobile phones not syncing to the new mailbox server after moving to Exchange 2013.

At first we thought it was just one or two users but it transpired that over 80 users were affected!! After investigating whether or not ActiveSync was working as expected (it was) we turned our attention to looking at an issue within the users accounts.

It turned out that the 80 users were all a member of a protected group within Active Directory and weren't getting the correct permissions to sync their phones as per Microsoft's best practices.

In order to get to that stage I used some PowerShell queries which I thought were quite interesting so I'm sharing.

I used the following command to query Active Directory for all users that had the "AdminCount" attribute set to something greater than 0.  If set to 1 it indicates the user is either a member of a protected group or has been:

Import-Module ac* 

Get-ADuser -filter {admincount -gt 0} -Properties admincount -ResultSetSize $null | export-csv c:\\onyx\document.csv

To find out which groups within the Active Directory environment I was working in were considered a Protected Group I ran the following query:

Import-Module ac*

Get-ADgroup -LDAPFilter "(admincount=1)" | select name 

From there I was able to check the groups individually to see which ones contained, if any, the users that were having issues with their phones.  All the affected users were a member of the "Print Operators" group. Mystery solved!

Get Mailbox Move Progress

I've recently been involved in an email migration from Exchange 2007 to Exchange 2013 and in order to monitor the progress of the mailbox moves I've been using Powershell commands, the two I've used are:

Get-MoveRequestStatistics -MoveRequestQueue "Mailbox Database 1"| Sort PercentComplete

This results in giving the Display Name, StatusDetail, TotalMailboxSize, TotalArchiveSize and PercentComplete of each mailbox move to that Exchange 2013 database.

The other one that I've found useful is:

Get-MoveRequestStatistics -MoveRequestQueue "Mailbox Database 1"| Where-Object {$_.StatusDetail -eq "Copying Messages"} | Sort PercentComplete

This is just a variation on the first query but it only shows the mailboxes that are currently copying messages to the Exchange 2013 environment.

Just remember a watched kettle never boils though! lol 

Powershell Query to Analyze your Mailbox Movement

I've recently been doing an Exchange 2007 to Exchange 2013 migration and wanted to find out the kind of speed the mailbox moves were taking.  I used the below Powershell script to pull out the MB transfer speed per minute:

Get-MoveRequest | Where { $_.Status -eq “Completed” } | Get-MoveRequestStatistics | Select DisplayName,TotalMailboxSize,TotalMailboxItemCount,@{n=”Speed MB/min”; e={ [int]($_.BytesTransferred.ToMB() / $_.TotalInProgressDuration.TotalMinutes) }}

How to resolve the error ‘550 5.7.1 Unable to Relay’?

I recently came across an issue at work where one of our servers wasn't able to send any email alerts out. On troubleshooting the issue we were receiving a '550 5.7.1 Unable to relay' error message.

Duly checked the receive connectors within the Exchange Management GUI and everything looked okay but still the server wasn't able to relay. On launching the Exchange Management Powershell console we ran the command:

Get-ReceiveConnector "Receive Connector" | Get-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON"

And noticed that instead of there being rights for "ms-Exch-SMTP-Accept-Any-Recipient", there were rights for "ms-Exch-SMTP-Accept-Any-Sender"

We issued the command:

Get-ReceiveConnector "Receive Connector" | Add-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON" -ExtendedRights "ms-Exch-SMTP-Accept-Any-Recipient"

And the server was able to relay messages once again. We haven't been able to determine, who or what changed the permissions on the connector but they'd been modified and the above command resolved our issue.

If you are trying to send notifications via your Exchange with an Anonymous connector ensure the following has extended rights:

NT AUTHORITY\Anonymous Logon {ms-Exch-SMTP-Submit}
NT AUTHORITY\Anonymous Logon {ms-Exch-Accept-Headers-Routing}
NT AUTHORITY\Anonymous Logon {ms-Exch-Bypass-Anti-Spam}
NT AUTHORITY\Anonymous Logon {ms-Exch-SMTP-Accept-Any-Recipient}

Powershell Tip: Search for users with an Active Sync device

The script below will search through a mailbox database and list the users that have Active Sync devices attached and list the type of device.

ForEach ($mb in (Get-Mailbox -database DATABASENAME)) { Get-ActiveSyncDeviceStatistics -Mailbox $mb | Select @{Label="Name";Expression={$mb.Name}},DeviceType,DeviceOS,LastSuccessSync}

Replace DATABASENAME with the name of the database you wish to query

This script has been test on Exchange 2010 SP1 and Exchange 2010 SP3.

Powershell Tip: How many DIMMs are in a machine and utilised

The following Powershell command will tell you the total amount of DIMM slots within a machine and what is currently installed within those slots, if anything.

Just save it as a .PS1 file and when you run it you will be asked for the name of the machine you wish to query. If you wish to query your own PC just enter "localhost" as the name.

$strComputer = Read-Host "Enter Computer Name"
$colSlots = Get-WmiObject -Class "win32_PhysicalMemoryArray" -namespace "root\CIMV2" `
-computerName $strComputer
$colRAM = Get-WmiObject -Class "win32_PhysicalMemory" -namespace "root\CIMV2" `
-computerName $strComputer

Foreach ($objSlot In $colSlots){
"Total Number of DIMM Slots: " + $objSlot.MemoryDevices
}
Foreach ($objRAM In $colRAM) {
"Memory Installed: " + $objRAM.DeviceLocator
"Memory Size: " + ($objRAM.Capacity / 1GB) + " GB"
}

Writing to Event Logs with Powershell

Was pondering how to trigger a event log entry on a server today at work in order to test SCOM monitoring, came across Powershell and the various commands that can be used for to do something like this. Found it quite interesting so thought it would be worth writing a wee post about it.

To begin you can see what Powershell commands are available in relation to the Eventlog:

get-command -Name *eventlog

The one that writes the event logs is called "Write-EventLog". The command syntax should be something as straightfoward are this:

Write-EventLog -LogName Application -Source "Test EventLog" -EntryType Information -EventID 1 -Message "This is a test message".

In this command the following parameters are required - LogName, Source, EventID.

Unforuntately this generates an error message because the source does not exist on the machine.  The cmdlet New-EventLog should help us here.

To fix the previous error we need to run the following command:

New-EventLog -LogName Application -Source "Test EventLog"

Then we can run:
Write-EventLog -LogName Application -Source "Test EventLog" -EntryType Information -EventID 1 -Message "This is a test message".

As we can see within the Event log the entry is now there:

Why not follow me on Twitter for other hints and tips - www.twitter.com/weeyinfotos