IP Phone Agent (IPPA) cannot log into UCCX after failover.

* Hint: Please click on the pictures in order to maximize them

So there I was sitting in the NOC room with my Colleagues and all of a sudden, a ticket was pushed into my queue. . . I went through the content of the ticket and it read as follows:

‘ The service desk team  at company X are unable to log  into the call center. They are getting the error message “ Your request to the CTI Server has timed out, please try again“. ’

I logged unto the client’s infrastructure and found out that they had deployed UCCX in High availability mode. On the uccx server’s serviceability page, I checked ‘ Control Center –Network Services’  and I found out that the primary sever had become the Slave Server. So from this point, I started suspecting that something may have gone wrong during the last failover.

 

I decided to investigate further so I jumped on the UCCX cluster via RTMT and collected the IPPA logs and in the logs I saw this:

 

2013-01-07 10:35:41,525 WARN IPPA3008 Agent phone IP address <192.168.XX.43>: IPPA client encountered an error <5:IPPA server is not active.> while making call to BIPPA service.
2013-01-07 10:56:34,514 WARN IPPA3008 Agent phone IP address <192.168.XX.42>: IPPA client encountered an error <5:IPPA server is not active.> while making call to BIPPA service.

2013-01-07 10:35:41,539 DEBUG [http-8080-3|SIPPA#login:205] IP phone address <192.168.XX.43>: Caught exception.
2013-01-07 10:56:34,514 WARN IPPA3008 [http-8080-3|SIPPASvrCorba#getCtiType:205] Agent phone IP address <192.168.XX.42>: IPPA client encountered an error <5:IPPA server is not active.> while making call to BIPPA service.
2013-01-07 10:56:34,514 DEBUG [http-8080-3|SIPPA#login:205] IP phone address <192.168.XX.42>: Caught exception.

 

On seeing this trace, I became very intrigued by the word ‘ exception’. In my mind I thought, ‘ hmm. . . exception! this must imply that the server encountered a problem that was not supposed to happen. Another thing that intrigued me was the log message; ‘ server not active’.

So based on the fact that  all the agents using their Ip Phones to log into uccx could not login anyway, and the agents using CAD  did not have any browser pages integrated with their CAD, I decided to restart the  ‘ Cisco Desktop Browser and IP Phone Agent Service’

I carried out a service restart and tried to log in as an agent and this time around,  the service was available but I was now getting a new error message.

The error message I got was  ‘ internal error: invalid input’ .

This error message is normally seen when the service URL that the agent is using to log into uccx  has not been configured correctly or when the agent’s phone has not been subscribed to the service properly on Cisco unified communications manager.

Because of the above error message, I then went over to the Administration page of CUCM and went over to Device-> Device Settings->  Phone Services.   When I got there I discovered that two service url(s) had been configured for the IP Phone agent to log unto UCCX. The first was pointing at the primary CCX while the second url was pointing at the secondary server. Now since there had been a failover which resulted in the secondary server assuming the role of a Master server, I focused on the second URL because this was the only URL that was now active due to the failover.

I copied and pasted the secondary server’s url on notepad and I found this:

http://192.168.XX.140:6293/ipphone/jsp/sciphonexml/IPAgentLogin.jsp

 

 

I also checked the service url for the primary server and I found this:

 

http://192.168.XX.15:6293/ipphone/jsp/sciphonexml/IPAgentInitial.jsp

It was at this point that I spotted the problem:The difference in these two url(s) was the reason why the agents could not log into the secondary server after the failover.

Did you notice that one URL ended with Initial while the other ended with Login?

I changed the secondary url to the value below and everything started working.

http://192.168.XX.140:6293/ipphone/jsp/sciphonexml/IPAgentInitial.jsp

::::::::::::::::::::::::::::::::::::::::::::::::::

Now you may be thinking why did this change of URL work! I’ll explain.

::::::::::::::::::::::::::::::::::::::::::::::::::

When http://server-ip:6293/ipphone/jsp/sciphonexml/IPAgentInitial.jsp is configured on cucm, and the agent presses the services button and selects the service configured for UCCX login, the agent is supplied with a Form on the IP Phone’s screen which the agent must fill in. The form will ask for the agent’s login credentials and phone number.

To show this better, the XML output below is the output I got by pasting the directly-above URL into internet explorer. Notice the Agent ID, Password and Extension parameters highlighted below? They are variables or containers that store data. Basically, as soon as an agent fills out the form (which appears on the phone) and submits it, the UCCX tries to verify the agent’s credentials by extracting the data stored in those parameters or container  and doing a lookup for a match.

<!–?xml version=”1.0″ encoding=”ISO-8859-1″ ?>–>

– <CiscoIPPhoneInput>

<Title>Agent Login</Title>

<Prompt>Enter agent information.</Prompt>

<URL>http://192.168.XX.15:6293/ipphone/jsp/sciphonexml/IPAgentLogin.jsp

– <InputItem>

<DisplayName>Name</DisplayName>

<QueryStringParam>ID</QueryStringParam>

<InputFlags>A</InputFlags>

<DefaultValue />

</InputItem>

– <InputItem>

<DisplayName>Password</DisplayName>

<QueryStringParam>Pwd</QueryStringParam>

<InputFlags>AP</InputFlags>

<DefaultValue />

</InputItem>

– <InputItem>

<DisplayName>Extension</DisplayName>

<QueryStringParam>Ext</QueryStringParam>

<InputFlags>N</InputFlags>

<DefaultValue />

</InputItem>

– <SoftKeyItem>

<Name>Submit</Name>

SoftKey:Submit

<Position>1</Position>

</SoftKeyItem>

– <SoftKeyItem>

<Name><<</Name>

SoftKey:<<

<Position>2</Position>

</SoftKeyItem>

– <SoftKeyItem>

<Name>Exit</Name>

<URL>Init:Services</URL>

<Position>3</Position>

</SoftKeyItem>

</CiscoIPPhoneInput>

 

:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

This then leaves the question; ‘what is different about the other URL below?’

http://192.168.XX.15:6293/ipphone/jsp/sciphonexml/IPAgentLogin.jsp

::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::;

 

Well, when I pasted the above url on internet explorer, I got the output below.

Did you notice that the error message below is the same  error message I was getting after restarting the Cisco Desktop Browser and IP Phone Agent Service?
Well the reason why it was saying ‘invalid input’ was because this url is only used when an administrator wants to configure one-button-login. This is when an agent is not asked to fill in their credentials when trying to log unto uccx because the administrator has already filled out the agent’s login details on his or her phone configuration page. So in essence,  when the agents presses the login button, the already configured agent credentials are sent to the UCCX server for verification/authentication.  Now in the case that I was dealing with, the agents did not have their credential configured on their phones or  user device profiles and apart from that, the phone services were not even configured for one-button-login anyway. It just had the single-button-login url configured.

What this then  meant was that agents were trying to login with blank or empty credentials; hence the reason for the error message: ‘ invalid Input’ .

<!–?xml version=”1.0″ encoding=”ISO-8859-1″ ?>–>

– <CiscoIPPhoneText>

<Title>Error</Title>

<Text>Internal error: Invalid input.</Text>

<Prompt />

– <SoftKeyItem>

<Name>OK</Name>

<URL>Init:Services</URL>

<Position>1</Position>

</SoftKeyItem>

</CiscoIPPhoneText>

When one-button-login Service url  is correctly configured on Call Manager,  you will see a configuration similar  to the one below:

 

Noticed the Extension, ID and Password parameters? Did you also notice that we saw these parameters in the XML out-put (above)when we pasted the URL ending with IPAgentInitial.jsp into internet explorer? Basically, when these parameters are configured on the service, they appear as variables that can be filled in by an administrator when a specific phone or device profile is being configured for a specific agent.  These pre-configured credentials are then automatically sent to the UCCX server whenever an agent tries to log-on by selecting the service: Hence, the name; Single or one-button-login.

Well. . that’s all folks. Hope you’ve found this post intriguing and enjoyable.

Cheers!