Synchronize Aternity REST API (OData) With External Systems (Beta)

REST API Synchronization Service

Use the Aternity REST API Synchronization Service to continually pull all necessary information from Aternity.

Use a client-side script to seamlessly synchronize the Aternity data collected using REST API with external systems, like applications, platforms, and databases. Create a script as explained in this article to enable the automatic sync of data. The number of entries you get can be much more than 1 million allowed in Excel and browsers. So, you can reduce the time of building data pipelines. Using a full sync stream is a fast and simple way for application integration when it comes to high data volumes and retaining the performance capability.

The REST API Synchronization Service enables a continuous data streaming to the client. When there is no new data, an empty page is returned with @odata.nextLink. The client should always keep following the next link. In case of HTTP 5xx errors, the client "sleeps" for a while and then tries again. The "sleep" time is defined in the script.

Fully synchronize data with external systems

The synchronization service is supported only with Aternity REST API v2.0 or above (learn more).

The Service provides a continuous full synchronization by establishing a stream that sleeps for 30 seconds when there is no new data and returns an empty page with @odata.nextLink. When there is a recoverable error (for example, a temporary network failure), the stream returns an empty page with @odata.nextLink and it keeps trying and sending the same request until the server recovers and responds with a valid response containing @odata.nextLink.

The Service provides continuity and completeness of the data flow by maintaining a HighWaterMark for each stream.

There is no way to delete a stream. It will be purged automatically after seven days of inactivity since the last access.

Note

In the future Aternity might change the API syntax and limit the number of stream requests that can be made to the Aternity REST API.

Create Script for FULL_SYNC_QUERY

To create a script, follow the below instructions:

  1. Define STREAM_ID which is the name of the stream. Assign an arbitrary unique STREAM_ID for each query.
  2. Define what API you want to query and use its API_NAME.
  3. Define the name for your stream and add to the script parameter $stream_id=.
  4. Define your script to use the value from the @odata.nextLink property as is without any alterations and to send it to the REST API.
  5. Check that you know the base URL of the REST API you want to sync (for example, https://my-odata.aternity.com).
  6. Prepare a username with permissions to use REST API.
  7. Prepare and add to the script REST_API_USERNAME and REST_API_PASSWORD of a user with REST API permissions in order to access the base URL.
  8. (Optional) You can define details of a search and use the query string with filter and/or select functions to limit the results (for example, $filter=application_name eq 'Outlook').
  9. Create your own script that sends a query to the API providing the $filter=, $select= and $stream_id= fields (for example, https://my-odata.aternity.com/aternity.odata/v2/APPLICATIONS_RAW?$format=json&$filter=application_name%20eq%20'Outlook'&$stream_id=my-applications-raw) or edit the sample script with a plain text editor to configure the predefined parameters.
  10. To initiate a full sync cycle, run the script.
    Tip

    Any change to a FULL_SYNC_QUERY means that you create a new query with a new unique ID. If you try changing anything without changing ID, you get an error message that this STREAM_ID is already in use for a different query and a continuous data streaming will be interrupted. You should not change anything in the query once you initiated the stream. New query requirements require creating a new FULL_SYNC_QUERY.

    The response from the REST API brings data from Aternity according to the query string definitions.

    In case of error... Do This...

    In case of error on the client side when it loses the @odata.nextLink data

    Define the script to send the original stream request again. The stream will continue from where it stopped (crashed), and the last data that was already returned to the client will be returned again. The client must be tolerant to receiving the same data again, in case it crashed after getting and using the response.

    For example, the last client request was for page #100, the client read the response and saved it locally, and then crashed. The client should start over and send the original request. It will get a response for page #100 again (with @odata.nextLink indicating that the next page is #101). The client should process this response while being aware that it got page #100 twice.

    The error is recoverable.

    For example,
    • Unable to open a TCP connection to the server.

    • HTTP 503 indicates that the service is unavailable.

    Define the script to keep trying and sending the same request until the server recovers and responds with a valid response containing @odata.nextLink.

    The error is not recoverable.

    For example,
    • HTTP 4XX error indicates a problem on the client side.

    • HTTP 500 (Internal Server Error) indicates a problem on the server side.

    Fix errors in the script.

    Once errors are fixed, it is necessary to start a new stream. Define the script to provide a new stream_id in order to send a new request to the server.

    A special case is error 400, “Stream requests do not support out-of-order page fetching.” This indicates that the client tried to either get a page that was already received or skip some pages. Define the script to send the original stream request again to sync with the server’s page order.

REST APIs Supported by the Synchronization Service

REST API name Sample links the script should create
APPLICATIONS_RAW (Learn more)

Use the APPLICATIONS_RAW API to retrieve the raw list of performance reports to Aternity, showing the performance and usage of each discovered application at five minute intervals running on each monitored device.

Example: https://my-odata.aternity.com/aternity.odata/v2/APPLICATIONS_RAW?$format=json&$stream_id=my-applications-raw

APPLICATIONS_HOURLY (Learn more)

Use APPLICATIONS_HOURLY or APPLICATIONS_DAILY to view the performance of each discovered application running on a device, where the measurements are aggregated per hour or per day, along with the user, location and device details.

Example: https://my-odata.aternity.com/aternity.odata/v2/APPLICATIONS_HOURLY?$format=json&$stream_id=my-applications-hourly

APPLICATIONS_DAILY (Learn more)

Use APPLICATIONS_HOURLY or APPLICATIONS_DAILY to view the performance of each discovered application running on a device, where the measurements are aggregated per hour or per day, along with the user, location and device details.

Example: https://my-odata.aternity.com/aternity.odata/v2/APPLICATIONS_DAILY?$format=json&$stream_id=my-applications-daily

APPLICATIONS_DAILY_ANONYMIZED (Learn more)

Use APPLICATIONS_DAILY_ANONYMIZED to view the daily average performance of each discovered application WITHOUT any PII (personally identifiable information), hence queries can have a much longer retention. It returns daily summaries of each application's performance, along with location and device attributes.

Example: https://my-odata.aternity.com/aternity.odata/v2/APPLICATIONS_DAILY_ANONYMIZED?$format=json&$stream_id=my-applications-daily-anonymized

APPLICATION_RESOURCES_HOURLY (Learn more)

Use APPLICATION_RESOURCES_HOURLY to view the resource usage (PRC) of the process of each managed application aggregated hourly, along with application, user, location and device details.

Example: https://my-odata.aternity.com/aternity.odata/v2/APPLICATION_RESOURCES_HOURLY?$format=json&$stream_id=my-applications-resources-hourly

BUSINESS_ACTIVITIES_RAW (Learn more)

BUSINESS_ACTIVITIES_RAW returns the list of activities as reported to Aternity one by one, showing the detailed user experience of all activities performed. For example, you can check one, several, or all devices, or any whose hostname contains the text mktg, for one user or any combination of users.

Example: https://my-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_RAW?$format=json&$stream_id=my-activities-raw

BUSINESS_ACTIVITIES_HOURLY (Learn more)

BUSINESS_ACTIVITIES_HOURLY returns the list of activities which each user performed in a one-hour time slot (from o'clock to o'clock), along with the activity's average performance (response time) the application name, user details and device details.

(Replaces API_USER_ACTIVITY_HOURLY)

Example: https://my-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_HOURLY?$format=json&$stream_id=my-activities-hourly

BUSINESS_ACTIVITIES_DAILY (Learn more)

BUSINESS_ACTIVITIES_DAILY returns the list of activities which each user performed each day, along with the activity's average performance (response time) the application name, user details and device details.

Example: https://my-odata-stream.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_DAILY?$format=json&$stream_id=my-activities-daily

BUSINESS_ACTIVITIES_DAILY_ANONYMIZED (Learn more)

Use BUSINESS_ACTIVITIES_DAILY_ANONYMIZED to return the list of activities performed each day WITHOUT any PII (personally identifiable information), hence your queries can go back further in history. It returns daily aggregations of each activity's response time the application name and device attributes.

Example: https://my-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_DAILY_ANONYMIZED?$format=json&$stream_id=my-activities-daily-anonymized

DEVICE_HEALTH_DAILY_ANONYMIZED (Learn more)

Use DEVICE_HEALTH_DAILY_ANONYMIZED to return the list of health events that occurred each day, without PII (personally identifiable information).

Example: https://my-odata.aternity.com/aternity.odata/v2/DEVICE_HEALTH_DAILY_ANONYMIZED?$format=json&$stream_id=my-devices-daily-anonymized

DEVICE_HEALTH_RAW (Learn more)

Use DEVICE_HEALTH_RAW (beta) to view the list of all device health events reported to Aternity up to the last seven days, updated every hour.

Example: https://my-odata.aternity.com/aternity.odata/v2/DEVICE_HEALTH_RAW?$format=json&$stream_id=my-devices-raw

HOST_RESOURCES_HOURLY (Learn more)

Use HOST_RESOURCES_HOURLY to view data regarding CPU and memory usage by devices (HRC, not the usage by one application).

Example: https://my-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_HOURLY?$format=json&$stream_id=my-resources-hourly

HOST_RESOURCES_DAILY (Learn more)

Use HOST_RESOURCES_DAILY to view daily average CPU and memory usage (HRC of all applications running on your monitored devices (NOT the resources of a single app).

Example: https://my-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_DAILY?$format=json&$stream_id=my-resources-daily

HOST_RESOURCES_DAILY_ANONYMIZED (Learn more)

Use HOST_RESOURCES_DAILY_ANONYMIZED to view data regarding CPU and memory usage by devices (HRC, not the usage by one application), without PII (personally identifiable information).

Example: https://my-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_DAILY_ANONYMIZED?$format=json&$stream_id=my-resources-daily-anonymized

A Sample Script

Aternity provides a sample script that uses the REST API synchronization feature. The sample script absorbs communication errors, such as server disconnections, and picks up where you left off last time with no error messages using ‘@odata.nextLink’. To obtain data updates through API, the script “calls” it on a regular basis. You do not need to define the paging size, the script continually brings new data over time using ‘@odata.nextLink’.

To run the sample script:

  • Ensure you have Python 3.0 with the following Python libraries: Requests and Validators.

Prepare the following parameters to use in the script:

  • STREAM_ID – the unique name of the stream

  • API_NAME – the name of the Aternity REST API

  • BASE_URl – the URI of the OData service (for example, https://my-odata.aternity.com

  • REST_API_USERNAME – a username with permissions to use REST API

  • REST_API_PASSWORD – the password for the above user

Tip

You can use the provided below sample script. Copy it to the plain text editor, like Notepad ++ and save as full_sync_sample_script.py. The sample script writes data to a csv file.

All data transferred between the API and client applications is formatted in JavaScript Object Notation (JSON). You can use the XML format as well, but it is less efficient than JSON and more complicated for processing.

A sample script for a full synchronization of Aternity REST API OData
import csv
import time
import traceback
import urllib.parse
from typing import Callable, List, Dict

import requests  # requires pip install requests
import validators  # requires pip install validators

# Parameters
STREAM_ID = "test_Stream_id"  # define your stream id
API_NAME = ''  # define the odata api name
BASE_URl = ''  # the Aternity rest api base url for example https://my-odata.aternity.com
REST_API_USERNAME = '' # the aternity rest api username
REST_API_PASSWORD = ''  # the aternity rest api password

# Constants
NEXT_PAGE_KEY = '@odata.nextLink'
DATA_VALUES_KEY = 'value'
BASE_API_URI = '/aternity.odata/v2/'+ API_NAME
FULL_SYNC_QUERY = BASE_API_URI + '?$format=json&$stream_id=' + STREAM_ID
FAILURE_SLEEP_TIME_IN_SECONDS = 60

# Globals

csv_file = open(file='full_sync_result.csv', mode='w', encoding='utf-8-sig')
csv_writer = csv.writer(csv_file, lineterminator='\n')
write_headers = True
session = requests.session()


# helper function to get the initial query that returns the distinct list of installed applications
def get_full_sync_start_query_url() -> str:
    base_url = BASE_URl[:-1] if BASE_URl.endswith('/') else BASE_URl
    if not validators.url(base_url):
        raise ValueError('base url {} is not valid'.format(BASE_URl))

    return base_url + FULL_SYNC_QUERY


def full_sync_result_handler(data: List[Dict]):
    global write_headers, csv_writer
    for row in data:
        if write_headers:
            header = row.keys()
            csv_writer.writerow(header)
            write_headers = False
        csv_writer.writerow(row.values())
    print('wrote {0} lines'.format(len(data)))


def full_sync_view(result_handler: Callable[[List[Dict]], None]):
    next_link = get_full_sync_start_query_url()
    print('starting query rest api for url {0}'.format(next_link))

    while next_link:
        start = time.time()
        response = session.get(url=next_link, auth=(REST_API_USERNAME, REST_API_PASSWORD))
        end = time.time()
        print('request for url {0} took {1} sec'.format(next_link, end - start))
        start = time.time()
        response.encoding = 'utf-8'
        try:
            response_json_parsed = response.json()
            if DATA_VALUES_KEY not in response_json_parsed:
                print('values key is not in the json the response we got is:')
                print(str(response.text))
                raise Exception
            data = response_json_parsed[DATA_VALUES_KEY]
            next_link = response_json_parsed[NEXT_PAGE_KEY]
            result_handler(data)
        except Exception as e:
            print('Exception Raised response text : {0}'.format(str(response.text)))
            if response.text and "Stream requests do not support out-of-order" in str(response.text):
                next_link = get_full_sync_start_query_url()
            print('Sleeping for {0} seconds due to previous error'.format(FAILURE_SLEEP_TIME_IN_SECONDS))
            time.sleep(FAILURE_SLEEP_TIME_IN_SECONDS)



try:
    full_sync_view(full_sync_result_handler)
except Exception as e:
    traceback.print_exc()
    traceback.print_stack()