Table of contents Migrate Your Stored Data from Version 10 to Version 11 Aternity allows a direct upgrade path from on-premise version 10 to version 11. For better quality and performance, in version 11 Aternity moved most of business analytical data to the Vertica database. To view version 11 dashboards with the historical data from version 10, perform the process of data migration as detailed in this article. To allow a full migration of data, run the process several days before Aternity upgrade. Once migration is complete, the upgrade can begin. In the current release, most of business data is stored in Vertica database Data migration process serves customers interested in keeping historical data after upgrade. To ensure a successful data migration to the latest version, you should deploy the data migration tool (ETL) and run the migration process. This process transfers existing data from the Oracle database to Vertica. Without data migration, some historical data will not be available after upgrade, and several dashboards and APIs will open empty. Dashboards from version 10.x that use a new database storage in version 11 Retention period in version 11 After data migration Without data migration SLA 400 days SLA transforms to Vertica in chunks. It takes a few days (3-4) to transform and load the last 400 days of SLA data.Following data migration, when you open Aternity version 11 for the first time, your historical data will show on the dashboard. Without data migration, the dashboard opens empty after upgrade. Only new data collected by the upgraded deployment will appear on this dashboard. It will be fully populated with data in a month after upgrade as new data flows in from devices. Boots Boot Analysis One month Following data migration, when you open Aternity version 11 for the first time, your historical data will show on dashboards. Without data migration, these dashboards open empty after upgrade. They will be fully populated with data in a month after upgrade as new data flows in from devices. New in version 11 dashboards, those that did not exist in version 10 at all, will open empty. (For example, Installed Applications, Wi-Fi, Host Resources, Skype Commonalities and more). One month N/A N/A New dashboards that did not exist in previous versions initially appear empty and will be fully populated with data in a month as new data flows in from devices. As shown on the below diagram, the workflow is as follows: Update the Vertica server to version 9.1.1-11. Deploy the migration tool (ETL). Start the migration process by activating migration cron jobs. Wait for 3-4 days until all historical data is copied. When the data migration finishes, stop the process and delete all ETL-related cron jobs. Now start upgrading Aternity. Important Stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Proceed with data migration in this order Tip New attributes that did not exist in older versions appear in the new database tables as not available (N/A) (for example, MS Office License Type = N/A, Machine Power Plan = N/A). Before you begin Before migrating stored data to Aternity on-premise Make yourself familiar with a new architecture of version 11, as well as with new hardware and software requirements. Always check the sizing before updating. This version uses different database sizes, dashboard requirements and new servers. Learn more. You will need a new server dedicated for the Docker Components. Purchase a new server that meets hardware requirements for Aternity Docker Components Server. Prepare a dedicated server for the data migration tool. As the data migration tool runs prior to the Aternity version update, we recommend using the same server that will be used for the Aternity Docker Components Server once Aternity 11 is installed. Therefore, the migration process does not require additional hardware purchases beyond the required for any Aternity 11 deployments. Note Although in version 10 architecture there is already one ETL Server that sends data from the Oracle Database Server to the Vertica Database Server, you need another temporary ETL that is dedicated to the data migration process. Version 10 ETL and the migration ETL tool must run in parallel on two different machines. Do not install the tool for data migration on exiting Version 10 ETL Server. Once version 11 is up and running, you will not need anymore the old ETL Server. You must start with a fully functioning deployment of Aternity on-premise 10.x. To run a Python script that migrates data, pip 9.0.3 is necessary. Older or newer versions of the pip package will automatically be replaced by pip 9.0.3 as part of the migration ETL component installation. Ensure you have administrator permissions on Windows machines and sudo permissions on Linux. Check that the dedicated server for the data migration tool conforms to the minimum system requirements: Attribute Requirement Operating system for ETL Linux CentOS 7.4 - 7.6. To verify your version of CentOS, enter cat /etc/centos-release. For the kernel version, enter uname -r and see the first two numbers. Red Hat Enterprise Linux (RHEL) 7.4 - 7.6. To verify the RHEL version, enter cat /etc/redhat-release Virtual memory settings for ETL Verify that your system has a swap file of at least 6GB which is active, by entering free -m. To create a 6GB swap file and enable it, enter the following commands, one line at a time: dd if=/dev/zero of=/tmp/swapfile bs=10240 count=630194 mkswap /swapfile swapon /swapfile To set the swap file as permanent, edit the system partition settings in /etc/fstab in a plain text editor such as vim, and add the following line at the end of the file: /swapfile swap swap defaults 0 0 Then save and exit the text editor. The system enables the new swap file when you restart the server. Resource limits for ETL The user who runs the ETL must have an open file limit of 65536 or more. Set the number by editing the system's resource limits in etc/security/limits.conf in a plain text editor like vim, and add the following line at the end of the file: etl_username - nofile 65536 If the username is root, enter root - nofile 65536 Now enter ulimit -n to verify the limit. Then save and exit the editor. Permissions The user who runs the server setup need to have sudo privileges. The user with permissions to run the ETL must also be allowed to run scheduled (crontab) tasks. The user which runs the ETL process does not need to have root or sudo root privileges. ProcedureStep 1 Perform an update of Vertica Database Server to version 9.1.1-11. Learn more. This process requires shutting down Aternity servers, so expect some downtime. The Aternity Vertica Database Server stores the performance data in the Vertica format, which is most efficient for displaying in Aternity dashboards.The number of Vertica Database Servers you require depends on the size of your deployment, and the level of high availability that you need.Even if you are not interested in migrating your historical data, the Vertica Database Server must still be updated as part of the general system update because it is part of the Aternity 11 architecture. Step 2 Start all Aternity 10 components. Step 3 Install a temporary ETL component for data migration process. a Extract the data_migration.zip file which you downloaded as part of the Aternity on-premise server setup package. b In the extracted folder, locate the file data_migration_v10_v11.tar and copy it to a temporary folder on the data migration host. c Unpack this tar file by entering tar -xvf data_migration_v10_v11.tar. Unpack it on the Linux machine dedicated to the Aternity Docker Components Server that temporarily serves as the ETL component for data migration. Unpack the ETL setup files Field Description -x Use -x to unzip the contents of the package. -v Use -v to output all messages (verbose). -f Use -f to specify the filename. d Configure the user who will set up the server to have the required sudo privileges. Edit the file sudoer_cfg.sh (located under the unpacked tar file). At the beginning of the file, replace the values of the specified in the below table parameters with the values relevant for your deployment: Field Description INSTALLING_USER='etlinstall' Replace with a relevant installing user name. This user must already exist in the system. ETL_INSTALLATION_DIRECTORY='/home/etluser/migration' Replace with location of the installation files (not the ETL target location). ETL_USER='etluser' Replace with name of the user with permissions to run the migration ETL. This user must have permissions to run scheduled (crontab) tasks. This user must already exist in the system. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. ETL_USER_GROUP='etluser' Replace with group of the etl_user. ETL_HOME_FOLDER='/etl_data' Replace with target location for the ETL (where the ETL will run from). VERTICA_DB_NAME='aternity' Replace with Vertica Database name. VERTICA_SCHEMA_NAME='aternity' Replace with Aternity schema name (same as the Aternity schema name in Oracle) – In migrations from 10.0.x, this already exists, in migrations from 9.0.x, this was created in step 2e. Do not edit additional text below these parameters. Save changes and exit. Run the following command: # /bin/bash sudoer_cfg.sh Copy the output and provide it to your IT representative. (IT people should grant access permissions in the sudoers file allowing privileged user(s) to set up ETL. Learn more.) Follow the below steps, running commands as the user defined under INSTALLING_USER. e Navigate to a temporary folder with the unpacked setup files and run this command, replacing the parameters with those from your configuration: python ETL_Install.py -etl_home_folder <target_folder> -oracle_user <aternity_user_name> -oracle_db_password <aternity_user_password> -oracle_ip <oracle_server_address> -oracle_port <oracle_port> -oracle_service <oracle_aternity_service_name> -vertica_db_name <vertica_aternity_db_name> -vertica_node <vertica_node_address> -vertica_db_admin_user <dbadmin_user_name> -vertica_db_admin_password <dbadmin_password> -vertica_cluster_ips <list_of_all_vertica_node_addresses> -etl_user <dedicated_linux_account> -etl_user_group_name <group_containing_dedicated_etl_user_account>You run the setup as a Python script with a long list of parameters needed. This is a single command line containing all of the parameters needed to run the migration ETL component setup. We recommend that you carefully edit this command in a text editor to ensure that you have correctly entered the parameters, and then paste the text into the command prompt. Run the setup script using Python Field Description etl_home_folder Provide a path to a destination folder where setup writes the program files. This folder also stores ETL data files. Note While you choose the destination folder of the setup, note that it also inserts files in /opt/vertica and /opt/sqlcl. oracle_user oracle_db_password Enter the name of the ATERNITY schema (username) and password of the Aternity Oracle Database Server. oracle_ip Enter the hostname or IP v4 address of the Aternity Oracle Database Server. oracle_port Enter the port required to access the Oracle Database Server (default is 1521). oracle_service Enter the Oracle database service name (usually Aternity). This is the alias to the instance of the Aternity database. vertica_db_name Enter the name of the Vertica database (not the schema). vertica_node Enter the hostname or IP v4 address of any one of the Vertica Database Servers in the Vertica cluster. The ETL attempts to connect to this server first, and if it is unable, it tries to connect with the next server in the cluster. If you only have one Vertica Database Server, enter its hostname or IP v4 address. vertica_db_admin_user vertica_db_admin_password Enter the username and password of the Vertica database administrator, with privileges to create schemas and prepare the database. vertica_cluster_ips Enter a comma-separated list of all of the hostnames or FQDNs (recommended) or IP v4 addresses of all of the Vertica Database Server nodes in the cluster, for example, vertica_db1.mydomain.com,vertica_db2.mydomain.com,vertica_db3.mydomain.com. If you only have one Vertica Database Server, enter its name or address. etl_user Enter the name of the user with permissions to run the ETL. This user must have permissions to run scheduled (crontab) tasks. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. etl_user_group_name Enter the group of the etl_user. The setup notifies you when it completes successfully. Your setup of the ETL Server completed successfully Step 4 Validate that data migration is successful by checking that cron jobs have been created for the user who is running data migration. Execute the following command and enter the name of the Linux account dedicated to running data migration, as set when the migration tool was deployed: crontab -l -u <dedicated_linux_user> The deployment of the data migration creates several scheduled cron jobs. You can verify that the data migration was deployed successfully by checking that these cron jobs were created. Run the command shown in this slide, replacing the placeholder with the name of the Linux account dedicated to running the migration, as set when the migration was deployed. example of the expected output Step 5 Keep track of the data migration process's state by checking the ETL log files periodically. Two log files can be found at: <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/log/etl.log For example, /data_migration/aternity/customerSchema/log/etl.log . Note Folder names that are being created during data migration depend on the Oracle service name and the Aternity schema name of the current Aternity installation. Keep track of the data migration process's state by checking the ETL log files Verify that the log file contains entries and check it for ERROR. If the log contains entries without errors, the data migration is running properly. <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/migration_status/current_migration_status.log This log file should contain three entries, each showing the migration progress for a different database table. Once all three reach 100%, the migration is complete. Check logs for the progress status Step 6 Let the data migration tool run until Aternity upgrade begins. The tool can complete its job in a few days and then continue running in parallel to the existing version until upgrade process begins. We recommend to stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Note Not stopping the migration tool may prevent the upgrade from completing successfully. Before upgrade, stop the migration tool (ETL). On the data migration Linux host, execute the following command to delete the migration cron jobs: crontab -r -u <dedicated_linux_user>. Run the command as a user with root or sudo root privileges. To verify, enter sudo id. Note This command will delete all cron jobs for any user who runs this command, so it is important to specify the dedicated_linux_user in the command. The dedicated_linux_useris the etl_user as defined during the Docker Components installation.Alternatively, run the command as a dedicated_linux_user. In this case, the command is as follows: crontab -r. Step 7 Continue with Aternity upgrade. Learn more. If you see in the upgrade instructions some steps that you have already performed during data migration, skip and continue. Parent topic Upgrade to Aternity on-premise 11.0.3Related tasksMigrate Your Stored Data from Version 9 to a New Database Format of Version 11Update the Aternity Oracle Database ServerUpdate the Aternity Vertica Database ServerUpdate the Aternity Dashboard ServerUpdate the Aternity Dashboard Gateway ServerQuickly Update the Aternity Management ServerUpdate (Publish) Dashboard Layouts for AternityUpdate the Aternity Docker Components ServerQuickly Update Aternity Data Warehouse Server on a Dedicated ComputerQuickly Update an Aternity Aggregation Server on a Dedicated Computer SavePDF Selected topic Selected topic and subtopics All content Related Links
Migrate Your Stored Data from Version 10 to Version 11 Aternity allows a direct upgrade path from on-premise version 10 to version 11. For better quality and performance, in version 11 Aternity moved most of business analytical data to the Vertica database. To view version 11 dashboards with the historical data from version 10, perform the process of data migration as detailed in this article. To allow a full migration of data, run the process several days before Aternity upgrade. Once migration is complete, the upgrade can begin. In the current release, most of business data is stored in Vertica database Data migration process serves customers interested in keeping historical data after upgrade. To ensure a successful data migration to the latest version, you should deploy the data migration tool (ETL) and run the migration process. This process transfers existing data from the Oracle database to Vertica. Without data migration, some historical data will not be available after upgrade, and several dashboards and APIs will open empty. Dashboards from version 10.x that use a new database storage in version 11 Retention period in version 11 After data migration Without data migration SLA 400 days SLA transforms to Vertica in chunks. It takes a few days (3-4) to transform and load the last 400 days of SLA data.Following data migration, when you open Aternity version 11 for the first time, your historical data will show on the dashboard. Without data migration, the dashboard opens empty after upgrade. Only new data collected by the upgraded deployment will appear on this dashboard. It will be fully populated with data in a month after upgrade as new data flows in from devices. Boots Boot Analysis One month Following data migration, when you open Aternity version 11 for the first time, your historical data will show on dashboards. Without data migration, these dashboards open empty after upgrade. They will be fully populated with data in a month after upgrade as new data flows in from devices. New in version 11 dashboards, those that did not exist in version 10 at all, will open empty. (For example, Installed Applications, Wi-Fi, Host Resources, Skype Commonalities and more). One month N/A N/A New dashboards that did not exist in previous versions initially appear empty and will be fully populated with data in a month as new data flows in from devices. As shown on the below diagram, the workflow is as follows: Update the Vertica server to version 9.1.1-11. Deploy the migration tool (ETL). Start the migration process by activating migration cron jobs. Wait for 3-4 days until all historical data is copied. When the data migration finishes, stop the process and delete all ETL-related cron jobs. Now start upgrading Aternity. Important Stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Proceed with data migration in this order Tip New attributes that did not exist in older versions appear in the new database tables as not available (N/A) (for example, MS Office License Type = N/A, Machine Power Plan = N/A). Before you begin Before migrating stored data to Aternity on-premise Make yourself familiar with a new architecture of version 11, as well as with new hardware and software requirements. Always check the sizing before updating. This version uses different database sizes, dashboard requirements and new servers. Learn more. You will need a new server dedicated for the Docker Components. Purchase a new server that meets hardware requirements for Aternity Docker Components Server. Prepare a dedicated server for the data migration tool. As the data migration tool runs prior to the Aternity version update, we recommend using the same server that will be used for the Aternity Docker Components Server once Aternity 11 is installed. Therefore, the migration process does not require additional hardware purchases beyond the required for any Aternity 11 deployments. Note Although in version 10 architecture there is already one ETL Server that sends data from the Oracle Database Server to the Vertica Database Server, you need another temporary ETL that is dedicated to the data migration process. Version 10 ETL and the migration ETL tool must run in parallel on two different machines. Do not install the tool for data migration on exiting Version 10 ETL Server. Once version 11 is up and running, you will not need anymore the old ETL Server. You must start with a fully functioning deployment of Aternity on-premise 10.x. To run a Python script that migrates data, pip 9.0.3 is necessary. Older or newer versions of the pip package will automatically be replaced by pip 9.0.3 as part of the migration ETL component installation. Ensure you have administrator permissions on Windows machines and sudo permissions on Linux. Check that the dedicated server for the data migration tool conforms to the minimum system requirements: Attribute Requirement Operating system for ETL Linux CentOS 7.4 - 7.6. To verify your version of CentOS, enter cat /etc/centos-release. For the kernel version, enter uname -r and see the first two numbers. Red Hat Enterprise Linux (RHEL) 7.4 - 7.6. To verify the RHEL version, enter cat /etc/redhat-release Virtual memory settings for ETL Verify that your system has a swap file of at least 6GB which is active, by entering free -m. To create a 6GB swap file and enable it, enter the following commands, one line at a time: dd if=/dev/zero of=/tmp/swapfile bs=10240 count=630194 mkswap /swapfile swapon /swapfile To set the swap file as permanent, edit the system partition settings in /etc/fstab in a plain text editor such as vim, and add the following line at the end of the file: /swapfile swap swap defaults 0 0 Then save and exit the text editor. The system enables the new swap file when you restart the server. Resource limits for ETL The user who runs the ETL must have an open file limit of 65536 or more. Set the number by editing the system's resource limits in etc/security/limits.conf in a plain text editor like vim, and add the following line at the end of the file: etl_username - nofile 65536 If the username is root, enter root - nofile 65536 Now enter ulimit -n to verify the limit. Then save and exit the editor. Permissions The user who runs the server setup need to have sudo privileges. The user with permissions to run the ETL must also be allowed to run scheduled (crontab) tasks. The user which runs the ETL process does not need to have root or sudo root privileges. ProcedureStep 1 Perform an update of Vertica Database Server to version 9.1.1-11. Learn more. This process requires shutting down Aternity servers, so expect some downtime. The Aternity Vertica Database Server stores the performance data in the Vertica format, which is most efficient for displaying in Aternity dashboards.The number of Vertica Database Servers you require depends on the size of your deployment, and the level of high availability that you need.Even if you are not interested in migrating your historical data, the Vertica Database Server must still be updated as part of the general system update because it is part of the Aternity 11 architecture. Step 2 Start all Aternity 10 components. Step 3 Install a temporary ETL component for data migration process. a Extract the data_migration.zip file which you downloaded as part of the Aternity on-premise server setup package. b In the extracted folder, locate the file data_migration_v10_v11.tar and copy it to a temporary folder on the data migration host. c Unpack this tar file by entering tar -xvf data_migration_v10_v11.tar. Unpack it on the Linux machine dedicated to the Aternity Docker Components Server that temporarily serves as the ETL component for data migration. Unpack the ETL setup files Field Description -x Use -x to unzip the contents of the package. -v Use -v to output all messages (verbose). -f Use -f to specify the filename. d Configure the user who will set up the server to have the required sudo privileges. Edit the file sudoer_cfg.sh (located under the unpacked tar file). At the beginning of the file, replace the values of the specified in the below table parameters with the values relevant for your deployment: Field Description INSTALLING_USER='etlinstall' Replace with a relevant installing user name. This user must already exist in the system. ETL_INSTALLATION_DIRECTORY='/home/etluser/migration' Replace with location of the installation files (not the ETL target location). ETL_USER='etluser' Replace with name of the user with permissions to run the migration ETL. This user must have permissions to run scheduled (crontab) tasks. This user must already exist in the system. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. ETL_USER_GROUP='etluser' Replace with group of the etl_user. ETL_HOME_FOLDER='/etl_data' Replace with target location for the ETL (where the ETL will run from). VERTICA_DB_NAME='aternity' Replace with Vertica Database name. VERTICA_SCHEMA_NAME='aternity' Replace with Aternity schema name (same as the Aternity schema name in Oracle) – In migrations from 10.0.x, this already exists, in migrations from 9.0.x, this was created in step 2e. Do not edit additional text below these parameters. Save changes and exit. Run the following command: # /bin/bash sudoer_cfg.sh Copy the output and provide it to your IT representative. (IT people should grant access permissions in the sudoers file allowing privileged user(s) to set up ETL. Learn more.) Follow the below steps, running commands as the user defined under INSTALLING_USER. e Navigate to a temporary folder with the unpacked setup files and run this command, replacing the parameters with those from your configuration: python ETL_Install.py -etl_home_folder <target_folder> -oracle_user <aternity_user_name> -oracle_db_password <aternity_user_password> -oracle_ip <oracle_server_address> -oracle_port <oracle_port> -oracle_service <oracle_aternity_service_name> -vertica_db_name <vertica_aternity_db_name> -vertica_node <vertica_node_address> -vertica_db_admin_user <dbadmin_user_name> -vertica_db_admin_password <dbadmin_password> -vertica_cluster_ips <list_of_all_vertica_node_addresses> -etl_user <dedicated_linux_account> -etl_user_group_name <group_containing_dedicated_etl_user_account>You run the setup as a Python script with a long list of parameters needed. This is a single command line containing all of the parameters needed to run the migration ETL component setup. We recommend that you carefully edit this command in a text editor to ensure that you have correctly entered the parameters, and then paste the text into the command prompt. Run the setup script using Python Field Description etl_home_folder Provide a path to a destination folder where setup writes the program files. This folder also stores ETL data files. Note While you choose the destination folder of the setup, note that it also inserts files in /opt/vertica and /opt/sqlcl. oracle_user oracle_db_password Enter the name of the ATERNITY schema (username) and password of the Aternity Oracle Database Server. oracle_ip Enter the hostname or IP v4 address of the Aternity Oracle Database Server. oracle_port Enter the port required to access the Oracle Database Server (default is 1521). oracle_service Enter the Oracle database service name (usually Aternity). This is the alias to the instance of the Aternity database. vertica_db_name Enter the name of the Vertica database (not the schema). vertica_node Enter the hostname or IP v4 address of any one of the Vertica Database Servers in the Vertica cluster. The ETL attempts to connect to this server first, and if it is unable, it tries to connect with the next server in the cluster. If you only have one Vertica Database Server, enter its hostname or IP v4 address. vertica_db_admin_user vertica_db_admin_password Enter the username and password of the Vertica database administrator, with privileges to create schemas and prepare the database. vertica_cluster_ips Enter a comma-separated list of all of the hostnames or FQDNs (recommended) or IP v4 addresses of all of the Vertica Database Server nodes in the cluster, for example, vertica_db1.mydomain.com,vertica_db2.mydomain.com,vertica_db3.mydomain.com. If you only have one Vertica Database Server, enter its name or address. etl_user Enter the name of the user with permissions to run the ETL. This user must have permissions to run scheduled (crontab) tasks. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. etl_user_group_name Enter the group of the etl_user. The setup notifies you when it completes successfully. Your setup of the ETL Server completed successfully Step 4 Validate that data migration is successful by checking that cron jobs have been created for the user who is running data migration. Execute the following command and enter the name of the Linux account dedicated to running data migration, as set when the migration tool was deployed: crontab -l -u <dedicated_linux_user> The deployment of the data migration creates several scheduled cron jobs. You can verify that the data migration was deployed successfully by checking that these cron jobs were created. Run the command shown in this slide, replacing the placeholder with the name of the Linux account dedicated to running the migration, as set when the migration was deployed. example of the expected output Step 5 Keep track of the data migration process's state by checking the ETL log files periodically. Two log files can be found at: <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/log/etl.log For example, /data_migration/aternity/customerSchema/log/etl.log . Note Folder names that are being created during data migration depend on the Oracle service name and the Aternity schema name of the current Aternity installation. Keep track of the data migration process's state by checking the ETL log files Verify that the log file contains entries and check it for ERROR. If the log contains entries without errors, the data migration is running properly. <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/migration_status/current_migration_status.log This log file should contain three entries, each showing the migration progress for a different database table. Once all three reach 100%, the migration is complete. Check logs for the progress status Step 6 Let the data migration tool run until Aternity upgrade begins. The tool can complete its job in a few days and then continue running in parallel to the existing version until upgrade process begins. We recommend to stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Note Not stopping the migration tool may prevent the upgrade from completing successfully. Before upgrade, stop the migration tool (ETL). On the data migration Linux host, execute the following command to delete the migration cron jobs: crontab -r -u <dedicated_linux_user>. Run the command as a user with root or sudo root privileges. To verify, enter sudo id. Note This command will delete all cron jobs for any user who runs this command, so it is important to specify the dedicated_linux_user in the command. The dedicated_linux_useris the etl_user as defined during the Docker Components installation.Alternatively, run the command as a dedicated_linux_user. In this case, the command is as follows: crontab -r. Step 7 Continue with Aternity upgrade. Learn more. If you see in the upgrade instructions some steps that you have already performed during data migration, skip and continue. Parent topic Upgrade to Aternity on-premise 11.0.3Related tasksMigrate Your Stored Data from Version 9 to a New Database Format of Version 11Update the Aternity Oracle Database ServerUpdate the Aternity Vertica Database ServerUpdate the Aternity Dashboard ServerUpdate the Aternity Dashboard Gateway ServerQuickly Update the Aternity Management ServerUpdate (Publish) Dashboard Layouts for AternityUpdate the Aternity Docker Components ServerQuickly Update Aternity Data Warehouse Server on a Dedicated ComputerQuickly Update an Aternity Aggregation Server on a Dedicated Computer
Migrate Your Stored Data from Version 10 to Version 11 Aternity allows a direct upgrade path from on-premise version 10 to version 11. For better quality and performance, in version 11 Aternity moved most of business analytical data to the Vertica database. To view version 11 dashboards with the historical data from version 10, perform the process of data migration as detailed in this article. To allow a full migration of data, run the process several days before Aternity upgrade. Once migration is complete, the upgrade can begin. In the current release, most of business data is stored in Vertica database Data migration process serves customers interested in keeping historical data after upgrade. To ensure a successful data migration to the latest version, you should deploy the data migration tool (ETL) and run the migration process. This process transfers existing data from the Oracle database to Vertica. Without data migration, some historical data will not be available after upgrade, and several dashboards and APIs will open empty. Dashboards from version 10.x that use a new database storage in version 11 Retention period in version 11 After data migration Without data migration SLA 400 days SLA transforms to Vertica in chunks. It takes a few days (3-4) to transform and load the last 400 days of SLA data.Following data migration, when you open Aternity version 11 for the first time, your historical data will show on the dashboard. Without data migration, the dashboard opens empty after upgrade. Only new data collected by the upgraded deployment will appear on this dashboard. It will be fully populated with data in a month after upgrade as new data flows in from devices. Boots Boot Analysis One month Following data migration, when you open Aternity version 11 for the first time, your historical data will show on dashboards. Without data migration, these dashboards open empty after upgrade. They will be fully populated with data in a month after upgrade as new data flows in from devices. New in version 11 dashboards, those that did not exist in version 10 at all, will open empty. (For example, Installed Applications, Wi-Fi, Host Resources, Skype Commonalities and more). One month N/A N/A New dashboards that did not exist in previous versions initially appear empty and will be fully populated with data in a month as new data flows in from devices. As shown on the below diagram, the workflow is as follows: Update the Vertica server to version 9.1.1-11. Deploy the migration tool (ETL). Start the migration process by activating migration cron jobs. Wait for 3-4 days until all historical data is copied. When the data migration finishes, stop the process and delete all ETL-related cron jobs. Now start upgrading Aternity. Important Stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Proceed with data migration in this order Tip New attributes that did not exist in older versions appear in the new database tables as not available (N/A) (for example, MS Office License Type = N/A, Machine Power Plan = N/A). Before you begin Before migrating stored data to Aternity on-premise Make yourself familiar with a new architecture of version 11, as well as with new hardware and software requirements. Always check the sizing before updating. This version uses different database sizes, dashboard requirements and new servers. Learn more. You will need a new server dedicated for the Docker Components. Purchase a new server that meets hardware requirements for Aternity Docker Components Server. Prepare a dedicated server for the data migration tool. As the data migration tool runs prior to the Aternity version update, we recommend using the same server that will be used for the Aternity Docker Components Server once Aternity 11 is installed. Therefore, the migration process does not require additional hardware purchases beyond the required for any Aternity 11 deployments. Note Although in version 10 architecture there is already one ETL Server that sends data from the Oracle Database Server to the Vertica Database Server, you need another temporary ETL that is dedicated to the data migration process. Version 10 ETL and the migration ETL tool must run in parallel on two different machines. Do not install the tool for data migration on exiting Version 10 ETL Server. Once version 11 is up and running, you will not need anymore the old ETL Server. You must start with a fully functioning deployment of Aternity on-premise 10.x. To run a Python script that migrates data, pip 9.0.3 is necessary. Older or newer versions of the pip package will automatically be replaced by pip 9.0.3 as part of the migration ETL component installation. Ensure you have administrator permissions on Windows machines and sudo permissions on Linux. Check that the dedicated server for the data migration tool conforms to the minimum system requirements: Attribute Requirement Operating system for ETL Linux CentOS 7.4 - 7.6. To verify your version of CentOS, enter cat /etc/centos-release. For the kernel version, enter uname -r and see the first two numbers. Red Hat Enterprise Linux (RHEL) 7.4 - 7.6. To verify the RHEL version, enter cat /etc/redhat-release Virtual memory settings for ETL Verify that your system has a swap file of at least 6GB which is active, by entering free -m. To create a 6GB swap file and enable it, enter the following commands, one line at a time: dd if=/dev/zero of=/tmp/swapfile bs=10240 count=630194 mkswap /swapfile swapon /swapfile To set the swap file as permanent, edit the system partition settings in /etc/fstab in a plain text editor such as vim, and add the following line at the end of the file: /swapfile swap swap defaults 0 0 Then save and exit the text editor. The system enables the new swap file when you restart the server. Resource limits for ETL The user who runs the ETL must have an open file limit of 65536 or more. Set the number by editing the system's resource limits in etc/security/limits.conf in a plain text editor like vim, and add the following line at the end of the file: etl_username - nofile 65536 If the username is root, enter root - nofile 65536 Now enter ulimit -n to verify the limit. Then save and exit the editor. Permissions The user who runs the server setup need to have sudo privileges. The user with permissions to run the ETL must also be allowed to run scheduled (crontab) tasks. The user which runs the ETL process does not need to have root or sudo root privileges. ProcedureStep 1 Perform an update of Vertica Database Server to version 9.1.1-11. Learn more. This process requires shutting down Aternity servers, so expect some downtime. The Aternity Vertica Database Server stores the performance data in the Vertica format, which is most efficient for displaying in Aternity dashboards.The number of Vertica Database Servers you require depends on the size of your deployment, and the level of high availability that you need.Even if you are not interested in migrating your historical data, the Vertica Database Server must still be updated as part of the general system update because it is part of the Aternity 11 architecture. Step 2 Start all Aternity 10 components. Step 3 Install a temporary ETL component for data migration process. a Extract the data_migration.zip file which you downloaded as part of the Aternity on-premise server setup package. b In the extracted folder, locate the file data_migration_v10_v11.tar and copy it to a temporary folder on the data migration host. c Unpack this tar file by entering tar -xvf data_migration_v10_v11.tar. Unpack it on the Linux machine dedicated to the Aternity Docker Components Server that temporarily serves as the ETL component for data migration. Unpack the ETL setup files Field Description -x Use -x to unzip the contents of the package. -v Use -v to output all messages (verbose). -f Use -f to specify the filename. d Configure the user who will set up the server to have the required sudo privileges. Edit the file sudoer_cfg.sh (located under the unpacked tar file). At the beginning of the file, replace the values of the specified in the below table parameters with the values relevant for your deployment: Field Description INSTALLING_USER='etlinstall' Replace with a relevant installing user name. This user must already exist in the system. ETL_INSTALLATION_DIRECTORY='/home/etluser/migration' Replace with location of the installation files (not the ETL target location). ETL_USER='etluser' Replace with name of the user with permissions to run the migration ETL. This user must have permissions to run scheduled (crontab) tasks. This user must already exist in the system. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. ETL_USER_GROUP='etluser' Replace with group of the etl_user. ETL_HOME_FOLDER='/etl_data' Replace with target location for the ETL (where the ETL will run from). VERTICA_DB_NAME='aternity' Replace with Vertica Database name. VERTICA_SCHEMA_NAME='aternity' Replace with Aternity schema name (same as the Aternity schema name in Oracle) – In migrations from 10.0.x, this already exists, in migrations from 9.0.x, this was created in step 2e. Do not edit additional text below these parameters. Save changes and exit. Run the following command: # /bin/bash sudoer_cfg.sh Copy the output and provide it to your IT representative. (IT people should grant access permissions in the sudoers file allowing privileged user(s) to set up ETL. Learn more.) Follow the below steps, running commands as the user defined under INSTALLING_USER. e Navigate to a temporary folder with the unpacked setup files and run this command, replacing the parameters with those from your configuration: python ETL_Install.py -etl_home_folder <target_folder> -oracle_user <aternity_user_name> -oracle_db_password <aternity_user_password> -oracle_ip <oracle_server_address> -oracle_port <oracle_port> -oracle_service <oracle_aternity_service_name> -vertica_db_name <vertica_aternity_db_name> -vertica_node <vertica_node_address> -vertica_db_admin_user <dbadmin_user_name> -vertica_db_admin_password <dbadmin_password> -vertica_cluster_ips <list_of_all_vertica_node_addresses> -etl_user <dedicated_linux_account> -etl_user_group_name <group_containing_dedicated_etl_user_account>You run the setup as a Python script with a long list of parameters needed. This is a single command line containing all of the parameters needed to run the migration ETL component setup. We recommend that you carefully edit this command in a text editor to ensure that you have correctly entered the parameters, and then paste the text into the command prompt. Run the setup script using Python Field Description etl_home_folder Provide a path to a destination folder where setup writes the program files. This folder also stores ETL data files. Note While you choose the destination folder of the setup, note that it also inserts files in /opt/vertica and /opt/sqlcl. oracle_user oracle_db_password Enter the name of the ATERNITY schema (username) and password of the Aternity Oracle Database Server. oracle_ip Enter the hostname or IP v4 address of the Aternity Oracle Database Server. oracle_port Enter the port required to access the Oracle Database Server (default is 1521). oracle_service Enter the Oracle database service name (usually Aternity). This is the alias to the instance of the Aternity database. vertica_db_name Enter the name of the Vertica database (not the schema). vertica_node Enter the hostname or IP v4 address of any one of the Vertica Database Servers in the Vertica cluster. The ETL attempts to connect to this server first, and if it is unable, it tries to connect with the next server in the cluster. If you only have one Vertica Database Server, enter its hostname or IP v4 address. vertica_db_admin_user vertica_db_admin_password Enter the username and password of the Vertica database administrator, with privileges to create schemas and prepare the database. vertica_cluster_ips Enter a comma-separated list of all of the hostnames or FQDNs (recommended) or IP v4 addresses of all of the Vertica Database Server nodes in the cluster, for example, vertica_db1.mydomain.com,vertica_db2.mydomain.com,vertica_db3.mydomain.com. If you only have one Vertica Database Server, enter its name or address. etl_user Enter the name of the user with permissions to run the ETL. This user must have permissions to run scheduled (crontab) tasks. Typically, you would not run the ETL as a root user, which is the default if you do not specify another user. etl_user_group_name Enter the group of the etl_user. The setup notifies you when it completes successfully. Your setup of the ETL Server completed successfully Step 4 Validate that data migration is successful by checking that cron jobs have been created for the user who is running data migration. Execute the following command and enter the name of the Linux account dedicated to running data migration, as set when the migration tool was deployed: crontab -l -u <dedicated_linux_user> The deployment of the data migration creates several scheduled cron jobs. You can verify that the data migration was deployed successfully by checking that these cron jobs were created. Run the command shown in this slide, replacing the placeholder with the name of the Linux account dedicated to running the migration, as set when the migration was deployed. example of the expected output Step 5 Keep track of the data migration process's state by checking the ETL log files periodically. Two log files can be found at: <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/log/etl.log For example, /data_migration/aternity/customerSchema/log/etl.log . Note Folder names that are being created during data migration depend on the Oracle service name and the Aternity schema name of the current Aternity installation. Keep track of the data migration process's state by checking the ETL log files Verify that the log file contains entries and check it for ERROR. If the log contains entries without errors, the data migration is running properly. <data_migration_home_folder>/<oracle_service_name>/<aternity_schema_name>/migration_status/current_migration_status.log This log file should contain three entries, each showing the migration progress for a different database table. Once all three reach 100%, the migration is complete. Check logs for the progress status Step 6 Let the data migration tool run until Aternity upgrade begins. The tool can complete its job in a few days and then continue running in parallel to the existing version until upgrade process begins. We recommend to stop the data migration tool (ETL) as close as possible to the actual upgrade, to avoid gaps in the retained data. Longer a gap between stopping data migration and starting the upgrade, bigger data gaps will be in the retained data. Note Not stopping the migration tool may prevent the upgrade from completing successfully. Before upgrade, stop the migration tool (ETL). On the data migration Linux host, execute the following command to delete the migration cron jobs: crontab -r -u <dedicated_linux_user>. Run the command as a user with root or sudo root privileges. To verify, enter sudo id. Note This command will delete all cron jobs for any user who runs this command, so it is important to specify the dedicated_linux_user in the command. The dedicated_linux_useris the etl_user as defined during the Docker Components installation.Alternatively, run the command as a dedicated_linux_user. In this case, the command is as follows: crontab -r. Step 7 Continue with Aternity upgrade. Learn more. If you see in the upgrade instructions some steps that you have already performed during data migration, skip and continue. Parent topic Upgrade to Aternity on-premise 11.0.3Related tasksMigrate Your Stored Data from Version 9 to a New Database Format of Version 11Update the Aternity Oracle Database ServerUpdate the Aternity Vertica Database ServerUpdate the Aternity Dashboard ServerUpdate the Aternity Dashboard Gateway ServerQuickly Update the Aternity Management ServerUpdate (Publish) Dashboard Layouts for AternityUpdate the Aternity Docker Components ServerQuickly Update Aternity Data Warehouse Server on a Dedicated ComputerQuickly Update an Aternity Aggregation Server on a Dedicated Computer