Applying Patches & Upgrades With HA
With a HA environment, patches and upgrades can be applied while experiencing minimal downtime in data monitoring.
If the system being upgraded is version 11.4 with HA enabled, then you must use the upgrade procedure described here, HA Upgrade from Version 11.4.
For all upgrades from version 11.5 and higher, use the following procedure.
Upgrading from 11.5 or later
To ensure the system is patched or upgraded correctly whilst minimizing downtime, follow the steps below when patching or upgrading an installation. For the purposes of the procedure below, 'upgrade' means either to apply a patch to an existing version or to upgrade the to a newer version.
In these steps, the following abbreviations are used:
- M for Management Server
- A refers to Primary Server
- B refers to Secondary Server
Check the State of the HA Pair
At this point, server B is going to be upgraded, ensure that server B is Passive (hence server A is Active).
If not, use the Manual Switchover of the HA Pair function so that server A becomes Active and data is collected while upgrading server B.
Verify that the servers are synchronized by checking
- PostgreSQL replication status is OK. This can be done one of the following ways:
- (Easy Way) Check the status of the HA Pair on the HA Administration page in Web Application - Administration tool.
(Hard Way) Run the following command on server B and enter the 'postgres' user password if prompted:
> psql -U postgres -W -c "select pg_is_in_recovery()"
the output should be as follows:
pg_is_in_recovery ------------------- t (1 row)
- Configurations and other items have been synchronized and applied by the HA service.
- This can be confirmed by Monitoring High Availability
Update PostgreSQL DB schema to target version
Use the Upgrade PostgreSQL Schema procedure that will upgrade schema on server A before starting the Prognosis upgrade installation.
Patch/Upgrade server B
Follow the standard operating procedures to either patch or upgrade the Prognosis Server.
Investigate success of the patch/upgrade on server B
Server B has now been upgraded, it is configured as a Passive server, which means the success of the patch/upgrade cannot be determined by checking if data is being collected. In order to check if the upgrade or patch was successful, it is suggested that:
- Documentation regarding the particular systems being monitored is consulted to do an initial check if upgrades were successfully applied.
- Verify that PostgreSQL replication from server A to server B is working, this can be confirmed through the HA Pair status on the HA Administration page.
- Verify that server A is running as an Active node and server B is running as a Passive node.
While Paused, Switchover the HA Pair
Use the Manual Switchover of the HA Pair procedure for this.
This operation cannot be undone. It is recommended that any current Analysts, Thresholds and Database Collections/Summaries are backed up prior to running this step to ensure that those configurations can be restarted if needed.
Note that switching over while Paused, the behavior will be slightly different from a regular switchover in the following manner: As expected, the Configurations will not be synchronized so no modifications will occur on either server, however the PROGNOSIS Configuration is an exception to this, where it will switch from being a Passive mode Configuration to being an Active mode Configuration and will allow any changes applied during the upgrade to be merged.
Verify Server B is in Active mode and collecting data
Using the HA Administration page
- Verify that server B is running as an Active node.
Now that the Active and Passive servers have been switched and B is now active, confirm that data is being collected on server B.
If data is not being collected on server B, then verify that the collectors are running and start them by updating the PROGNOSIS Configuration. Also, ensure that all Databases, Thresholds and Analysts are running on server B and have been stopped on server A.
Patch/Upgrade server A
Follow the standard operating procedures to either patch or upgrade the Prognosis Server.
Investigate success of the patch/upgrade on server A
Using the verification items referenced in Step 6 above (except reverse servers A ↔ B)
Resume HA synchronization
Click the same button used to Pause the synchronization (See Step 4).
Verify that synchronization is working
Verify synchronization by Monitoring High Availability. The majority of items should be showing SYNCED in the 'Sync Status' column.
Switchback, returning the server A to the Active mode.
Use the Manual Switchover of the HA Pair procedure for this.
Run the irpgdataload tool
This tool is used to load the Device and Component based Alert Suppression data into PostgreSQL. This step is essential if using the Alert Suppression feature.
This should be executed from the Prognosis configuration folder on server A after the switchback:
> cd <Prognosis_Home>\Server\Configuration > ..\x64\irpgdataload.exe ..\..\postgreSQL ..\..