Implement a Standby Site for Disaster Recovery

You can create an entire standby data center site that is ready to spring into action if your primary site becomes unavailable, also known as hot recovery. Once both primary and standby sites are installed, enable replication between them. Replication is the process by which the database is copied to produce two identical databases. This occurs on the primary site.

Hot recovery by implementing a standby site

Monitored devices should report either to the primary site's load balancer for Aggregation Servers, or to a parallel load balancer on the standby site for disaster recovery. When the primary site fails, route the Agents installed on the monitored devices to the standby site.

Tip

If your sizing requires that you deploy more than one dedicated Aternity Aggregation Server, you must deploy a third party load balancer (LB). Then configure the Agent for End User Devices of all devices to point to the LB's virtual IP address (learn more). Configure the LB with persistent (sticky) sessions to ensure the Agent maintains its connection with the same server. Aternity also supports sticky sessions when the LB implements stickiness via cookies.

Note

Every software patch installed in the primary site must be also installed in a hot standby site.

Before you begin

  • You must start with a fully functioning deployment of Aternity on-premise 12.0. If you have an older version, upgrade to on-premise 11.0.3. Learn more. The hot standby Vertica cluster must be IDENTICAL to the primary cluster (NUMBER OF NODES, STORAGE CAPACITY, VERTICA VERSION, DBA username, and DBA password). Learn more.

  • The default Vertica DBA username used in the sample commands in this article is dbadmin. Make sure to use the DBA username as defined in your company.

  • The default DBA folder used in the sample commands in this article is /home/dbadmin. Make sure to use the DBA home folder path as defined in your company.

  • Use domain names (DNS) instead of IP addresses when deploying Aternity.

  • For upgraded systems: the following steps must be carried out AFTER you upgraded to Aternity on-premise 11.0.3 and BEFORE proceeding with implementing a standby site for Disaster Recovery (DR):
    1. setup_dr.tar is part of the downloaded Aternity on-premise's 11.0.3 main setup package. On the primary Vertica Database node 1, copy setup_dr.tar to the DBA user home folder (the default location is /home/dbadmin ).
    2. Note that DBA user must be the owner of the file. Change the owner of the file to the DBA user. Run the command as a user with root or sudo root privileges on the computer.: Replace dbadmin with the DBA username as defined in your deployment.
      $ sudo chown dbadmin /home/dbadmin/setup_dr.tar
    3. Unpack the .tar file by entering tar -xvf setup_dr.tar. Run this command as DBADMIN user. The unpacked files are: change_vertica_schema_owner.py, pip.whl, setup_vertica_dr.py, vertica_replicate.sh.
    4. Install Vertica client. Run the command as a user with root or sudo root privileges on the computer. Replace dbadmin with the DBA username as defined in your deployment.
      $ sudo  /opt/vertica/oss/python/bin/python /home/dbadmin/pip.whl/pip install --no-index -f /opt/vertica/Python/vertica_db_client*.whl vertica-db-client
  • You must validate from time to time that the Aternity hot standby site properly functions by adding end user devices and starting all components.

  • Make yourself familiar with the script parameters.

  • Your DBA should be available to make changes to the Oracle Database Server.

Procedure

  1. Step 1 Deploy the fully functioning Aternity on-premise 11.0.3 site. Learn more.

    During the setup of the Aternity Management Server, configure the connection of the Management Server to the Vertica Database Server and Oracle Database Server, and automatically initialize the database by creating its schema. The two schemas are:

    Field Description
    ATERNITY schema

    This is the name of the business data schema, which stores the performance measurements and device data over the past year or two, along with contextual data, like device details and user details.

    GR schema

    This is the name of the system settings schema, which stores Aternity's system settings.

  2. Step 2 Deploy the fully IDENTICAL hot standby site of Aternity on-premise 11.0.3. Learn more.

    The hot standby Vertica cluster must be IDENTICAL to the primary cluster (NUMBER OF NODES, STORAGE CAPACITY, VERTICA VERSION, DBA username, and DBA password). Learn more.

    Important
    During setup of the Aternity Management Server on a hot standby site, to connect with databases do the following:
    • For Oracle, in the Database Connection screen, enter the address of the PRIMARY Oracle instance, NOT a standby.

    • For Oracle, in the Database Connection screen, typically you enter details in the Connection tab, but if you have a custom database connection URL, use the Advanced Properties tab. Enter the URL of the PRIMARY Oracle instance, NOT a standby.

    • For Vertica, in the Connection screen, enter the IP address of the PRIMARY Vertica instance, NOT a standby.

    • Define the names of the two schemas in the Aternity database, and their passwords (Aternity and GR schemas). In the Database Schemas screen, make sure these names are different from the schema names used for the PRIMARY site, for example enter the names Aternity_DR and GR_DR. If you do not change the names, you will overwrite the primary database!

    • In the Database Schemas screen, when prompted to define usernames and passwords, ensure the Configuration Database (GR) password is the same as the one used for the PRIMARY site!
      Enter the names and passwords of the two schemas
      Field Description
      Platform Database Credentials

      Enter a name for the business data schema (for example, Aternity_DR), and its password.

      Configuration Database Credentials

      Enter a name for the system settings schema (for example, GR_DR), and the same password as in the PRIMARY instance.

    Enter the IP address and URL of the PRIMARY instance, NOT a hot standby and set the schemas with alternative names: for example, the ATERNITY schema call ATERNITY_DR and the GR schema call ATERNITY_DR_GR.
    Important

    For the GR schema of the hot standby site, use the same password you have used for the primary GR schema.

    Learn more.

  3. Step 3 Ensure that the hot standby site is properly installed, up and running:
    Add end-points and start all components. After validation tests, you can stop all servers except for databases. To lower costs, it is possible in the meantime to use these servers for other needs. Only the database servers must always be up an running the replication process.
  4. Step 4 Use a tool to backup the Oracle Database Server constantly (live) to a clone server which remains in a state of read-only.

    Aternity supports creating a redundant clone of the Oracle Database Server using Oracle Data Guard, which streams all database updates to the clone server to ensure its content remains the same as the main server.

    Clone the Aternity Oracle Database Server

    To implement a cloned Oracle Database Server, you must enable Oracle's archive log for hot backups.

    If you have Oracle RAC (Real Application Clusters) deployed, configure the connection string of Aternity's database service to use only a one single instance in the cluster. You can configure the failover option to use another server if required.

    Tip

    If you have your own custom queries and reports which you want to run directly on the Aternity database, use the clone.

    Therefore Aternity does NOT support RAC's load balancing and high availability features. We recommend you contact Customer Services to discuss your overall DR / high availability architecture.

  5. Step 5 Set up the Aternity Vertica Database Server replication process.
    1. a Ensure that the primary Vertica cluster nodes have network access to the hot standby Vertica cluster (rsync command uses port 50000, make sure it is open).
    2. b Ensure that the port 5433 TCP is open on the primary and on the standby Vertica clusters allowing proper communication between those servers. It is required for replication.
    3. c Set the passwordless SSH between primary and hot standby nodes:
      • From the primary Vertica Database Server, copy the content from the file <primary node 1 cluster>/home/dbadmin/.ssh/id_rsa.pub to the /home/dbadmin/.ssh/authorized_keys file. Make sure to replace DBA username as defined in your deployment. Repeat this on every node in the hot standby cluster.

      • If Linux DBA user has a password, do the following: copy the public key using the ssh-copy-id command from the primary Node 1 to every node in the hot standby cluster. Make sure to replace DBA username as defined in your deployment.
        $ ssh-copy-id -i /home/dbadmin/.ssh/id_rsa.pub dbadmin@dr_node1_hostname
  6. Step 6 Enable a disaster recovery functionality by starting a replication process.
    1. a On the primary Vertica Database Server cluster, on Node 1, execute the Python setup_vertica_dr.py script. Run this command as DBADMIN user.

      First of all, enter the relevant parameters instead of default values. See the table below.

      The script resides in the DBADMIN home folder.

      To run these commands, you need DBADMIN user privileges.

      Execute the following command. Make sure to replace DBA username and password as defined in your deployment. Make sure to use the DBA home folder path as defined in your company.

      $ /opt/vertica/oss/python/bin/python /home/dbadmin/setup_vertica_dr.py --task enable --dbadmin_user dbadmin --dbadmin_password admin --database_name aternity --dr_cluster_nodes_ips “10.0.0.1,10.0.0.2,10.0.0.3”
      Field Description

      dbadmin user

      The default name in the sample is dbadmin. Enter the name as defined in your deployment.

      This user name is for both Linux and Vertica databases.
      dbadmin_password

      The default password in the sample isadmin. Enter the password as defined in your deployment.

      This is Vertica Database password, not Linux.

      database_name

      The default name in the sample is aternity. Enter the name as defined in your deployment.

      Note that the name is case sensitive and should be a lower case! It must begin with an alphabetic character, and can contain letters (lower case), numbers and underscores, and must be up to 30 characters.

    2. b On the primary Vertica Database Server cluster, configure the settings to show the replication status in the System Health dashboard.
      Go to the Gear Icon > Settings > tableau > vertica > monitorReplicationStatus, and apply Yes for Replication Status.
      Configure the settings to view the status in the System Health

      Now, open the System Health dashboard and see the details of replication. Hover your mouse over the Vertica Replicator component whose status you check and see the details in the pop-up window.

    In case of disaster, switch to a standby site. Learn more.