* Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab
* Hint:Please double-click the snapshots to zoom into them.
This is one of the most intriguing cases I have every dealt with: This client has Cisco Unity connection 8.6x integrated with a Microsoft Forest containing over ten Microsoft server 2003 and 2008 domains controllers spread out across multiple sites in Europe and the Americas.
The client complained that all of a sudden, when new users were created in Active directory, they would no longer show up in Cisco unity connection . I asked the client what had changed prior to the problem and I was informed that nobody had touched the Cisco servers except for the fact that their Group policies had been recently been re-written within active-directory.
I logged unto Cisco unity connection and tried do an import of users and nothing happened; no error messages were displayed on the webpage and no newly created users from Active-Directory were imported.
I made up my mind that I would have to pull out the ‘ Big-Guns’:
Stage one
1) I Installed Wireshark on a PC and started capturing packets
Stage Two
On the same PC, I also installed Softerra LDAP Browser. softerra LDAP Browser is a tool that you simple put the username, password and IP-address of your LDAP server into and it will show you everything in your Domain. This was going to be the way that I would use to simply test the username and password that Cisco unity connection was using. If you need this tool, here is the link to it: http://www.ldapbrowser.com/download.htm
Here is snapshot of the tool in action:
Stage Three
After I entered the username,password and IP-address of the LDAP server, I started seeing errors in the Wireshark. Here is a screen-shot of the errors below:
The username and password that I was using to authenticated was the same one that the Cisco unit Connection server was using so from the wireshark capture, it was clear that the reason the unity connection server was unable to import users from Active Directory was because it did not even make it pass authentication let-alone pulling user database.
If you look at the error message in the capture above, it say: ‘ In order to perform this operation, a successful bind must be completed on the connection’
The term ‘bind’ means authentication. So in ordinary English, the error message that can be seen in the LDAP message coming back from the LDAP server is basically saying: in order to search the LDAP user database, you must first authenticate successfully.
I double checked with the ‘DirSync’ service traces on unity connection and I found exactly the same error message.
Stage Four
With this evidence, I told the client to speak to his Server team; who in tern, contacted Microsoft Directly. Shortly after the Microsoft team arrived on the scene, they informed me that they did not share my view that the problem was on the Microsoft server side. They stated that since other users and servers were authenticating against LDAP without any problems, the LDAP server was fine.
Stage Five:
I knew I was talking to skilled and experienced Engineers so I was not quick to doubt them. I pulled out my wireshark again and investigated how the other non-cisco servers were managing to authenticate without any problems. what I found was that Cisco Unity Connection was using a method of authentication called ‘ Simple Bind‘ while the other servers were not. Simple bind authentication method is when usernames and passwords are sent in clear text: I will share how I got to know this later in this post.
I went back to Microsoft and informed them that they needed to permit simple bind for the Cisco Unity Connection server because that is how Unity connection operates. And that if they wanted security, they would have to use ‘ transport-Layer-security (TLS)’ because that is what Unity Connection supports.
Stage Six:
In order to test my theory, Microsoft then decided to enable Anonymous Bind for test purposes only. Anonymous bind is when a client can authenticate to the server without username or password.
Stage Seven
After we made the change above, the Cisco unity Connection still could not import users from Microsoft Active directory. Again, the Microsoft Team told me that they did not share my view . However, I reminded them that I asked for ‘simple bind’ to be allowed on the network not ‘ anonymous bind’ so they continued to work with me.
I was really starting to look really bad. I pulled out wireshark again and captured the packets being exchanged between the Cisco Unity Connection server and Microsoft LDAP Server and noticed that something had changed since anonymous bind was allowed; this is what I found:
Here is the interpretation of the logs above:
1) In the very first line, we see that Cisco Unity Connection server at Ip-address 172.16.6.33 sends an LDAP simple bind request to the LDAP server at IP-address 172.16.6.5. We know that we are using simple bind authentication because can see the word ‘ simple‘ at the very end of the line.
2) Looking at the ‘ source’ and ‘destination’ fields In the second line of the capture, we can see that the LDAP server sends a message back to the unity connection server stating that authentication had successfully completed.
3) In the third line, we see the Unity connection server trying to find all the users in the database by sending a search request to the LDAP sever and in the fourth line we see that the server responds with a result of zero; meaning that no result was provided to the Cisco unity connection server.
4) Going further down the line, we see that the unity connection server continues trying until the 20th line when the LdAP Server responds back with NameErr 0000208D, Problem 2001 (No_object)
Stage Eight
I decided to collect traces from Unity Connection so I turned on detailed traces, performed a full LDAP system sync on unity connection and collected Cisco Dir-sync service traces from the unity connection server. Here is what I found:
DAPSync(cca4b920-9361-9535-8d80-ece481332007)[getInvocationId] caught exception … [LDAP: error code 32 – 0000208D: NameErr: DSID-03151F00, problem 2001 (NO_OBJECT)
According to the following Microsoft website, LDAP error code 32 means: ‘The user has insufficient access rights: http://support.microsoft.com/kb/218185
Stage nine
I Informed the Server Admins and Microsoft team about the new error messages I was getting from their Microsoft servers. I asked the team to explain why their server was not happy with my request since the account I was using to authenticate was an admin user. In response, they informed me that the server was treating my Simple bind Request as an Anonymous bind request and since Anonymous bind request were not permitted to search the database, my request were being dropped. To remedy this, the following steps were applied to the Domain Controller.
- Open regedit.exe on a domain controller
- Go to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurityProviders
- In the right window pane double-click on SecurityProviders
- Add pwdssp.dll to the end of the Value data field which should be separated by a comma after the last entry. For example:
msapsspc.dll, schannel.dll, digest.dll, msnsspc.dll, pwdssp.dll
5. Reboot the DC: This was done on all the domain controllers on Windows 2008 and above and a reboot was required
Stage Ten
After the changes above were applied, everything started working fine lol 🙂 . . . and all along, everyone was blaming me but thanks to wireshark, I was able to prove to the remote Microsoft Server team that the problem was from them. Long live Wireshark. 🙂 and Mr Remi A @silvercleric for his contribution to this case.
The Voice Initiate 