* Hint:Please note that some of the screenshots and outputs in this post have been recreated in a lab

 * Hint: Please click on the pictures in order to maximize them

This was a case that turned out to be another unexpected ticket which had a few gatchas up it’s sleeves.  The issue started off with a client side engineer stating that he could not log into one of the nodes of his Unity Connection cluster.  As soon as I got this information, I Jumped on the cluster in order to see what was happening.

I tried logging into the Web graphical user interface but I got an error message instead. The error message said; ‘database communication error’. I logged unto the subscriber server and that was fine.  I then decided to log into the Publisher server via the command line interface.

As soon as I logged on, I saw these errors:

Command Line Interface is starting up, please wait …
java.io.FileNotFoundException: /var/log/active/platform/log/cli.bin (Read-only file system)
at java.io.RandomAccessFile.open(Native Method)

log4j:ERROR No output stream or file set for the appender named [CLI_LOG].
Welcome to the Platform Command Line Interface

The /common file system is mounted read only.
Please use Recovery Disk to check the file system using fsck.

At this point, it was clear that the system had gone into read-only-mode and would have to be recovered. The team agreed to recover this server whilst planning a full rebuild.

After the server was recovered, two days later it crashed again so a rebuild was now the only reasonable thing to do. The responsibility fell on me so I rushed to create a new VMware Virtual Unity Connection server. In approaching this case, I created a plan that ensured that I was following Cisco’s best practice recommendations and that I would be able to kick-start the new server and everything would work perfectly.  My plan was to make sure that the new server was as similar to the old one as possible so I made sure the following variables were the same:

1)      Server Hard-drive size

2)      Server memory size

3)      Server NTP Server

4)      Server IP –address

5)      Server Default-Gateway IP

6)      Cluster  security password

7)      Same server version as the old server

8)      Server host-name

Whilst it’s not necessary to use an OVA Template, I decided to use the template shown in the screenshot below as Unity ova template formats the drive in 64kB Blocks which results in ‘ improved storage input/output operation.

one

It is very crucial for the hostname to be the same as the failed server otherwise; the subscriber will complain that the new publisher’s host-file does not match the host-file that it has in its database. So what was needed was to basically trick the subscriber into believing that its connected to the original Publisher server when we connect them up.

So I loaded the Unity Connection installer and started building. As can been seen in the screenshot below, after a while, I got to the part where I had to enter the cluster certificate information.

two

As I did not have this information readily at hand and I was paranoid about build a near exact replica, I logged unto the subscriber unity connection server and then navigated to

i)                    OS admin page

ii)                   Security

iii)                 Certificate management

iv)                 And I selected the tomcat trust certificate

From the screen-shot below, did you noticed that all the information that was required of me in the screenshot above was actually  listed screenshot  below?

three

After filling in the required fields, I selected ‘ next’ on install process.

I was then asked by the install script if this was the first server in the cluster: I selected yes.

After while, the installation process seemed to have frozen or hung.  After a quick rummaging of the internet, I discovered that the server was actually installing in the background: The server appeared to have frozen because it was experiencing Bug  CSCtj33840 so I ignored it and after a while the installation completed.

After the server had rebooted, I followed Cisco’s upgrade guide and did the following.

Step 1 Sign in to Cisco Unity Connection Administration on the publisher server.

Step 2 In Cisco Unity Connection Administration, expand System Settings, then select Cluster.

Step 3 On the Find and List Servers page, select Add New.

Step 4 On the New Server Configuration page, in the Hostname/IP Address field, enter the hostname or IP address of the subscriber server.

Step 5 In the Description field, enter a description for the subscriber server.

Step 6 Select Save.

Step 7 Sign out of Cisco Unity Connection Administration.

To Connect the Subscriber Server to the New Connection Cluster, and Replicate Data and Messages to the Publisher Server

Step 1 Sign in to the command-line interface (CLI) for the subscriber server.

Step 2 Run the CLI command utils cuc cluster renegotiate.

After issuing this command, I got the output below:

 

This command may take require several hours to complete. The duration 

depends on the network bandwidth between servers and on the

total size of the data on this subscriber server.

 

Are you sure you want to continue? (y/n) : y

 

13/03/03 08:31:47  Disabling data replication…

13/03/03 08:31:57  Renegotiating ssh trusts…

13/03/03 08:31:59  Syncronizing platform and LDAP database…

13/03/03 08:35:07  Creating any missing messaging databases on the publisher…

13/03/03 08:35:10  Adding subscriber node to publisher…

13/03/03 08:35:12  Synchronizing Unity Connection databases…

13/03/03 08:41:05  Synchronizing file systems…

13/03/03 08:41:06  Synchronizing message files for mail store UnityMbxDb1…

13/03/03 08:41:07  Copying cluster DSCP configuration to publisher node…

13/03/03 08:41:08  Rebooting publisher node v8-unity-conn-192-168-0-56…

13/03/03 08:41:10  Cluster renegotiation completed.

 

Process completed. 

 

After the completion, I checked the cluster status by issuing the command ‘ show cuc cluster status’ and everything was fine.

The next phase was to re-host the license files from the old server to the new server but in order to do this; I needed the value of the license-mac of the new server. The information below proved crucial.

Host Name    : publisher-cuc

Date         : Sat Mar 2, 2013 09:34:39

Time Zone    : Greenwich Mean Time (Europe/London)

Locale       : en_US.UTF-8

Product Ver  : 7.1.5.30000-1

Platform Ver : 2.0.0.1-1

License MAC  : 0ce90be10d98

As can be seen above, I logged into the command line interface of the new publisher and issued the command ‘show status’. I then copied and sent the new ‘License-Mac’ to Cisco who then sent me the new license file which I then installed.

The server has been happy ever since. . .