High Availability (For Builds Prior to 6301)
(Feature available only in Premium and Enterprise Editions. Procedure applicable only for builds prior to 6301.
For later versions, refer to click here)

In mission-critical environments, one of the crucial requirements is to provide uninterrupted access to passwords. Password Manager Pro provides the 'High Availability' feature just to ensure this.

1. How does High Availability work?

  • There will be redundant Password Manager Pro server and database instances
  • One instance will be the Primary providing read/write access to the users. All users will be connected with primary only
  • The other instance will act as Secondary
  • At any point of time data in both Primary and Secondary will be in sync with each other. Password Manager Pro leverages MySQL's database replication technique for data synchronization. The data replication happens through a secure, encrypted channel
  • When Primary server goes down, the Secondary will offer 'Read Only' access to the users, until the fully-functional primary server is brought back to service. The changes made in the database in the intervening period will be automatically synchronized upon connection restoration

2. Example Scenarios

Scenario 1 - Primary & Secondary in different geographical locations and WAN Link failure happens between the locations

Assume that the Primary Server is in one geographical location 'A' and Secondary is deployed in another location 'B'. The users in both the locations will be connected to the Primary and will be carrying out password management activities. At any point of time data in both Primary and Secondary will be sync with each other. Assume there happens loss of network connectivity between the two locations. In such a scenario, users in location 'A' will continue to remain connected with the primary and will be doing all operations. Users in location 'B' will be able to get emergency read-only access to the passwords from Secondary. Once the network between the two locations is up again, data in both the locations will be synchronized.

Scenario 2 - Primary & Secondary within the same network & Primary goes down

In case, the Primary crashes or goes down, the users in location 'A' & 'B' can rely upon the emergency read-only access to the passwords from the Secondary.

What happens to Audit Trails?

In the high availability scenarios mentioned above, audit trails will be recorded as usual. In scenario 1, as long as there is network connectivity between the two locations, the audit trails will be printed by the primary. When users connect to the Secondary, it will print operations such as 'password retrieval', 'login' and 'logout'. When the two locations get back network connectivity, the audit data will be synchronized. In scenario 2, when the primary crashes, the 'password retrieval', 'login' and 'logout' done by the users in secondary will be audited. Other audit records will already be in sync at the Standby.

3. How to set up High Availability?

Setting up high availability in Password Manager Pro consists of the following four steps:

    1. Installing Primary & Secondary instances (you can use your existing installation as Primary and install another Password Manager Pro instance as standby in a separate work station)
    2. Configurations to be done in Primary Installation
    3. Configurations to be done in Secondary Installation
    4. Enabling database replication

Carry out the steps on-by-one as detailed below.

Step 1: Primary & Secondary Setup

  • Before trying High Availability, you should have both Primary and Standby installations of Password Manager Pro in place. You can either install one instance as 'Primary' or you can use your current Password Manager Pro installation as primary server. You can install another instance of Password Manager Pro as secondary server in a separate workstation. To install Password Manager Pro as secondary, during installation, you need to choose the option "Configure this server as High availability secondary server (Read Only)".

Important Note:

  • After installation, the Password Manager Pro Primary & Secondary servers should have been started and stopped at least once
  • It is recommended to delete all files except passtrix folder, mysql folder, test, PasstrixTemp, and ibdata1, ca-cert.pem, server-cert.pem, server-key.pem files present under <PMP_Home>/mysql/data folder in both Primary and Secondary before commencing the High Availability step.

Take care not to delete

  • 'passtrix' folder,
  • 'mysql' folder,
  • 'test' folder,
  • 'PasstrixTemp'
  • ''ibdata1',
  • 'ca-cert.pem',
  • 'server-cert.pem'
  • server-key.pem

If you delete them, you will lose all your data.

Step 2: Configurations to be done in Primary Installation

Prerequisite

    1. Stop Password Manager Pro Primary server, if already running
    2. Check if Password Manager Pro Secondary Server is reachable from Primary Server and vice-versa (do a ping)
    3. Go to the Password Manager Pro Primary folders and carry out the following:
      • Navigate to <PMP_Installation_Folder>/mysql/data folder. The following file & folders are important:
        • 'passtrix' folder,
        • 'mysql' folder,
        • 'test' folder,
        • 'PasstrixTemp'
        • ''ibdata1',
        • 'ca-cert.pem',
        • 'server-cert.pem'
        • server-key.pem

    KEEP THESE FOLDERS and delete all other files.

Steps

    1. Open a command prompt and navigate to <PMP_Installation_Folder>/bin directory and run the script replicationPack.bat (Windows)/ replicationPack.sh (Linux)
    2. This will create a replication package named 'Replication.zip' under <PMP_Home>/replication folder. This zip contains the database package for standby
    3. Go to <PMP_Installation_Folder>/mysql/bin directory. You will find a file named HAPrimary.conf, rename that file as HASecondary.conf. Important Note: Before proceeding further, ensure that HAPrimary.conf is NOT present under <PMP_Installation_Folder>/mysql/bin folder. It should only contain HASecondary.conf
    4. Edit the HASecondary.conf and enter the following details:

      5. Enter the name of the host where the secondary server is running:

      master_host=<hostname of Secondary>

      For example, 'test_workstation' is the machine where the secondary Password Manager Pro server is running, you need to enter the information as below:

      master_host=test_workstation

      Enter the host name, DNS Name and IP address of the secondary server:

      slave_hosts=<hostname of Secondary>,<DNS Name of Secondary>,<IP of Secondary>

      For example, 'test_workstation' is the machine where the secondary Password Manager Pro server is running, test_workstation.test.com is the DNS name and 192.168.10.1 is its IP, you need to enter the information as below (in comma separated form without any space):

      slave_hosts=test_workstation,test_workstation.test.com,192.168.10.1

    5. Open a command prompt and navigate to <PMP_Installation_Folder>/bin and run the script startDB.bat <MySQL Port> (Windows) / startDB.sh <MySQL Port> (Linux). You need to provide the MySQL port of Password Manager Pro while executing the above script as shown below. By default, the MySQL port in Password Manager Pro is 2345.

      6. startDB.bat <MySQL Port> (For Windows)

      startDB.sh <MySQL Port> (For Linux)

      For example, with the default the MySQL port 2345, you need to execute this as:

      startDB.bat 2345 (For Windows)

      startDB.sh 2345 (For Linux)

      This will start the Primary Database (Default MySQL port is 2345)

    6. Copy the Replication.zip file present under <PMP_Installation_Folder>/replication directory. This has to be put in the Password Manager Pro Secondary installation machine as detailed in Step 3 below.

Step 3: Changes in Secondary Installation

Prerequisite

    1. Stop Password Manager Pro Secondary server, if already running
    2. Check if Password Manager Pro Primary Server is reachable from Secondary Server and vice-versa (do a ping)
    3. Go to the Password Manager Pro Secondary folder and carry out the following:
    • Navigate to <PMP_Installation_Folder>/mysql/data folder. The following file & folders are important:
      • 'passtrix' folder,
      • 'mysql' folder,
      • 'test' folder,
      • 'PasstrixTemp'
      • ''ibdata1',
      • 'ca-cert.pem',
      • 'server-cert.pem'
      • server-key.pem
    • KEEP THESE FOLDERS and delete all other file
    1. Put the Replication.zip file copied from the PRIMARY Installation (as detailed in the previous step) in to the <PMP_Installation_Folder> of Secondary and unzip it. Take care to extract the files under <PMP_Installation_Folder> only. It will overwrite the existing data files.
    2. Copy the <PMP_Installation_Folder>/mysql/bin/database_params.conf file of secondary installation and put it over <PMP_Installation_Folder>/conf directory of secondary installation
    3. Go to <PMP_Installation_Folder>/mysql/bin directory. You will find a file named HAPrimary.conf. Important Note: Before proceeding further, ensure that HASecondary.conf is NOT present under PMP_Installation_Folder>/mysql/bin folder. It should only contain HAPrimary.conf
    4. Edit the HAPrimary.conf and enter the following details:

      5. Enter the name of the host where the primary server is running:

      master_host=<hostname of Primary>

      For example, 'test_workstation' is the machine where the primary Password Manager Pro server is running, you need to enter the information as below:

      master_host=test_workstation

      Enter the host name, DNS Name and IP address of the primary server:

      slave_hosts=<hostname of Primary>,<DNS Name of Primary>,<IP of Primary>

      For example, 'test_workstation' is the machine where the primary Password Manager Pro server is running, test_workstation.test.com is the DNS name and 192.168.10.1 is its IP, you need to enter the information as below:

      slave_hosts=test_workstation,test_workstation.test.com,192.168.10.1

    5. Copy the startDB.bat (in Windows) / startDB.sh (in Linux) file present under <PMP_Installation_Folder>/mysql/bin of secondary installation and put it over <PMP_Installation_Folder>/bin directory of secondary installation
    6. Go to <PMP_Installation_Folder>/bin of secondary installation and execute startDB.bat <MySQL Port> (in Windows) / startDB.sh <MySQL Port> (in Linux) to start Secondary database (Default MySQL port is 2345)

      7. For example, with the default the MySQL port 2345, you need to execute this as:

      startDB.bat 2345 (For Windows)

      startDB.sh 2345 (For Linux)

    7. This will start the Secondary Database (Default MySQL port is 2345)

Step 4: Enabling Database Replication - This is to be done in both Primary and Secondary Installations

  • Run enableReplication.bat (in Windows) / enableReplication.sh (in Linux) present in <PMP_Installation_Folder>/mysql/bin of both Primary and Secondary installations

Step 5: Start Primary and Secondary

  • Navigate to Password Manager Pro Primary server installation and terminate the Password Manager Pro's mysqld process and start Password Manager Pro service
  • Similarly, navigate to the Password Manager Pro Secondary installation and terminate the corresponding mysqld process and start Password Manager Pro service on Secondary
  • High Availability setup is now ready

4. Verify High Availability setup

After carrying out the above steps, you can verify if the High Availability setup is working properly by looking at the message in "Admin >> General >> High Availability" page of Primary server. If the setup is proper, you will see the following:

High Availability Status: Alive

Replication Status: Alive

If both the above messages show "Alive", it indicates that high availability is working fine. In case, if the status turns 'Failed', it indicates failure of the setup.

What does 'High Availability Status' Indicate (Alive/Failed)?

Constant replication of data between Primary and Standby server is the technology underlying high availability. The status 'Alive' indicates perfect data replication and data synchronization. If there happens any disruption like network problems between Primary and Standby (in turn between the databases), the status will get changed to 'Failed'.

This may happen when there is no communication/connection between the database of primary server and that of the standby server. When the connection gets reestablished, data synchronization will happen and both databases will be in sync with each other. During the intervening period, those who have connected to the primary and standby will not face any disruption in service.

In short, this status is only an indication of the connection/communication between databases and does not warrant any troubleshooting.

What does 'Replication Status' Indicate (Alive/Failed)?

As mentioned above, high availability leverages the MySQL replication feature. The database of Primary server acts as the Master and the one with Standby acts as the 'Slave'. When the data gets replicated properly, the status will be 'Alive'. In case, there happens any error in updating the data or query failure, the replication status become 'Failed'.

Once the status becomes failed, Password Manager Pro High Availability setup also breaks down. That means, you will have to configure high availability setup all over again.

If you find the status marked as 'Failed' even after re-configuring High Availability, you may have to contact Password Manager Pro support with the following log files:

Password Manager Pro In Windows: <PMP Installation Folder>/mysql/data/.err file

Password Manager Pro In Linux: <PMP Installation Folder>/mysql/data/tmp/ .out file

Alerting Mechanism for Status Failure

Since the above two conditions assume importance in high availability setup, it is important to receive real-time alerts when the status turns Alive to Failed and vice-versa. To configure alerts, go to Audit >> Resource Audit >> Configure User Audit >> General Operations and select the mode of alert (email/SNMP trap/Syslog message) for the events 'High Availability Alive' and 'High Availability Failed'.

Note 1 : In case, the Primary Server crashes, when carrying out disaster recovery, please ensure the following:

  • Go to the <PMP_Home>/mysql/data of secondary server and copy the files ibdata1, passtrix
  • Install another instance of Password Manager Pro afresh (same version as that of the Password Manager Pro secondary from which you copied the above files)
  • In the new installation, go to <PMP_Home>/mysql/data and overwrite the ibdata1, passtrix files
  • Start the new Password Manager Pro installation

Note 2 :

  • After configuring high availability, if you change the port of the Primary Password Manager Pro server, the high availability setup will not work. It has to be re-configured with suitable changes.

Note 3 :

If you have configured TFA

  • Whenever you enable TFA or when you change the TFA type (PhoneFactor or RSA SecurID or One-time password) AND if you have configured high availability, you need to restart the Password Manager Pro secondary server once.

Troubleshooting Tips

Refer to the 'Troubleshooting Tips' section in the High Availability Tutorial.
Top