Synchronize Aternity REST API (OData) With External Systems (Beta)

REST API Synchronization Service

Use the Aternity REST API Synchronization Service to continually pull all necessary information from Aternity.

Use a client-side script to seamlessly synchronize the Aternity data collected using REST API with external systems, like applications, platforms, and databases. Create a script as explained in this article to enable the automatic sync of data. The number of entries you get can be much more than 1 million allowed in Excel and browsers. So, you can reduce the time of building data pipelines. Using a full sync stream is a fast and simple way for application integration when it comes to high data volumes and retaining the performance capability.

The REST API Synchronization Service enables a continuous data streaming to the Client. When there is no new data, an empty page is returned with @odata.nextLink. Expect empty pages when starting a stream, until the stream catches up with the data. It can take a few hours for the first data to arrive, usually it is longer than the selected aggregation level. The Client always keeps following the next link. In case of HTTP 5xx errors, the Client "sleeps" for a while and then tries again. The "sleep" time is defined in the script.

For example, If you query for daily data and you initialize your stream at 8am, the stream will start looking for data in small increments from 8 am and will only catch the data at midnight, because daily aggregations happen at midnight. While the stream increments the data search, empty pages are returned in order to keep the Client active. After midnight, the relevant data will be returned.

Fully synchronize data with external systems

The synchronization service is supported only with Aternity REST API v2.0 or above (learn more).

The Service provides a continuous full synchronization by establishing a stream that sleeps for 30 seconds when there is no new data and returns an empty page with @odata.nextLink. Expect empty pages when starting a stream, until the stream catches up with the data. It can take a few hours for the first data to arrive. When there is a recoverable error (for example, a temporary network failure), the stream returns an empty page with @odata.nextLink and it keeps trying and sending the same request until the server recovers and responds with a valid response containing @odata.nextLink.

The Service provides continuity and completeness of the data flow by maintaining a HighWaterMark for each stream.

There is no way to delete a stream. It will be purged automatically after seven days of inactivity since the last access.

Note

In the future Aternity might change the API syntax and limit the number of stream requests that can be made to the Aternity REST API.

Create Script for FULL_SYNC_QUERY

To create a script, follow the below instructions:

  1. Define STREAM_ID which is the name of the stream. Assign an arbitrary unique STREAM_ID for each query.
  2. Define what API you want to query and use its API_NAME.
  3. Define the name for your stream and add to the script parameter $stream_id=.
  4. Define your script to use the value from the @odata.nextLink property as is without any alterations and to send it to the REST API.
  5. Check that you know the base URL of the REST API you want to sync (for example, https://my-demo-odata.aternity.com).
  6. Prepare a username with permissions to use REST API.
  7. Prepare and add to the script REST_API_USERNAME and REST_API_PASSWORD of a user with REST API permissions in order to access the base URL.
  8. (Optional) You can define details of a search and use the query string with filter and/or select functions to limit the results (for example, $filter=application_name eq 'Outlook').
  9. Create your own script that sends a query to the API providing the $filter=, $select= and $stream_id= fields (for example, https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATIONS_RAW?$format=json&$filter=application_name%20eq%20'Outlook'&$stream_id=my-applications-raw) or edit the sample script with a plain text editor to configure the predefined parameters.
  10. To initiate a full sync cycle, run the script.
    Tip

    Any change to a FULL_SYNC_QUERY means that you create a new query with a new unique ID. If you try changing anything without changing ID, you get an error message that this STREAM_ID is already in use for a different query and a continuous data streaming will be interrupted. You should not change anything in the query once you initiated the stream. New query requirements require creating a new FULL_SYNC_QUERY.

    The response from the REST API brings data from Aternity according to the query string definitions.

    In case of error... Do This...

    In case of error on the client side when it loses the @odata.nextLink data

    Define the script to send the original stream request again. The stream will continue from where it stopped (crashed), and the last data that was already returned to the client will be returned again. The client must be tolerant to receiving the same data again, in case it crashed after getting and using the response.

    For example, the last client request was for page #100, the client read the response and saved it locally, and then crashed. The client should start over and send the original request. It will get a response for page #100 again (with @odata.nextLink indicating that the next page is #101). The client should process this response while being aware that it got page #100 twice.

    The error is recoverable.

    For example,
    • Unable to open a TCP connection to the server.

    • HTTP 503 indicates that the service is unavailable.

    Define the script to keep trying and sending the same request until the server recovers and responds with a valid response containing @odata.nextLink.

    The error is not recoverable.

    For example,
    • HTTP 4XX error indicates a problem on the client side.

    • HTTP 500 (Internal Server Error) indicates a problem on the server side.

    Fix errors in the script.

    Once errors are fixed, it is necessary to start a new stream. Define the script to provide a new stream_id in order to send a new request to the server.

    A special case is error 400, “Stream requests do not support out-of-order page fetching.” This indicates that the client tried to either get a page that was already received or skip some pages. Define the script to send the original stream request again to sync with the server’s page order.

REST APIs Supported by the Synchronization Service

REST API name Sample links the script should create
APPLICATIONS_RAW (Learn more)

Use the APPLICATIONS_RAW API to retrieve the raw list of performance reports to Aternity, showing the performance and usage of each discovered application at five minute intervals running on each monitored device.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATIONS_RAW?$format=json&$stream_id=my_applications_raw

APPLICATIONS_HOURLY (Learn more)

Use APPLICATIONS_HOURLY or APPLICATIONS_DAILY to view the performance of each discovered application running on a device, where the measurements are aggregated per hour or per day, along with the user, location and device details.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATIONS_HOURLY?$format=json&$stream_id=my_applications_hourly

APPLICATIONS_DAILY (Learn more)

Use APPLICATIONS_HOURLY or APPLICATIONS_DAILY to view the performance of each discovered application running on a device, where the measurements are aggregated per hour or per day, along with the user, location and device details.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATIONS_DAILY?$format=json&$stream_id=my_applications_daily

APPLICATIONS_DAILY_ANONYMIZED (Learn more)

Use APPLICATIONS_DAILY_ANONYMIZED to view the daily average performance of each discovered application WITHOUT any PII (personally identifiable information), hence queries can have a much longer retention. It returns daily summaries of each application's performance, along with location and device attributes.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATIONS_DAILY_ANONYMIZED?$format=json&$stream_id=my_applications_daily_anonymized

APPLICATION_RESOURCES_HOURLY (Learn more)

Use APPLICATION_RESOURCES_HOURLY to view the resource usage (PRC) of the process of each managed application aggregated hourly, along with application, user, location and device details.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/APPLICATION_RESOURCES_HOURLY?$format=json&$stream_id=my_applications_resources_hourly

APPLICATION_EVENTS (Learn more)

Use APPLICATION_EVENTS to return application events which report errors or check how often someone used a feature in your application, like accessing the online help. You create an application event in the Aternity Activity Designer and upload it to Aternity as a custom activity.

ALL_ACTIVITIES (Learn more)

Use ALL_ACTIVITIES to view all activities reported to Aternity over the past seven days. Use this API to get an overview of which activities are taking place, their frequency, and how many users are performing the activity.

BUSINESS_ACTIVITIES_RAW (Learn more)

BUSINESS_ACTIVITIES_RAW returns the list of activities as reported to Aternity one by one, showing the detailed user experience of all activities performed. For example, you can check one, several, or all devices, or any whose hostname contains the text mktg, for one user or any combination of users.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_RAW?$format=json&$stream_id=my_activities_raw

BUSINESS_ACTIVITIES_HOURLY (Learn more)

BUSINESS_ACTIVITIES_HOURLY returns the list of activities which each user performed in a one-hour time slot (from o'clock to o'clock), along with the activity's average performance (response time) the application name, user details and device details.

(Replaces API_USER_ACTIVITY_HOURLY)

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_HOURLY?$format=json&$stream_id=my_activities_hourly

BUSINESS_ACTIVITIES_DAILY (Learn more)

BUSINESS_ACTIVITIES_DAILY returns the list of activities which each user performed each day, along with the activity's average performance (response time) the application name, user details and device details.

Example: https://my-odata-.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_DAILY?$format=json&$stream_id=my_activities_daily

BUSINESS_ACTIVITIES_DAILY_ANONYMIZED (Learn more)

Use BUSINESS_ACTIVITIES_DAILY_ANONYMIZED to return the list of activities performed each day WITHOUT any PII (personally identifiable information), hence your queries can go back further in history. It returns daily aggregations of each activity's response time the application name and device attributes.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/BUSINESS_ACTIVITIES_DAILY_ANONYMIZED?$format=json&$stream_id=my_activities_daily_anonymized

CUSTOM_DATA_RAW (Learn more)

CUSTOM_DATA_RAW returns the raw list of performance metrics you defined and added to Aternity.

CUSTOM_DATA_HOURLY (Learn more)

Use CUSTOM_DATA_HOURLY or CUSTOM_DATA_DAILY to view the list of performance metrics you defined and added to Aternity, along with their contextuals where the measurements are aggregated per hour or per day.

CUSTOM_DATA_DAILY (Learn more)

Use CUSTOM_DATA_HOURLY or CUSTOM_DATA_DAILY to view the list of performance metrics you defined and added to Aternity, along with their contextuals where the measurements are aggregated per hour or per day.

CUSTOM_DATA_DAILY_ANONYMIZED (Learn more)

CUSTOM_DATA_DAILY_ANONYMIZED returns the list of performance metrics you defined and added to Aternity aggregated daily WITHOUT any PII (personally identifiable information), hence your queries can go back further in history.

DASHBOARD_VIEWS_AUDIT (Learn more)

DASHBOARD_VIEWS_AUDIT returns a list of Aternity dashboards that users accessed over the past seven days, and the username and time of each access, so that you can audit or monitor the users who view different parts of Aternity.

DEVICE_HEALTH_DAILY_ANONYMIZED (Learn more)

Use DEVICE_HEALTH_DAILY_ANONYMIZED to return the list of health events that occurred each day, without PII (personally identifiable information).

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/DEVICE_HEALTH_DAILY_ANONYMIZED?$format=json&$stream_id=my_devices_daily_anonymized

DEVICE_HEALTH_RAW (Learn more)

Use DEVICE_HEALTH_RAW to view the list of all device health events reported to Aternity up to the last seven days, updated every hour.

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/DEVICE_HEALTH_RAW?$format=json&$stream_id=my_devices_raw

HOST_RESOURCES_HOURLY (Learn more)

Use HOST_RESOURCES_HOURLY to view data regarding CPU and memory usage by devices (HRC, not the usage by one application).

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_HOURLY?$format=json&$stream_id=my_resources_hourly

HOST_RESOURCES_DAILY (Learn more)

Use HOST_RESOURCES_DAILY to view daily average CPU and memory usage (HRC of all applications running on your monitored devices (NOT the resources of a single app).

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_DAILY?$format=json&$stream_id=my_resources_daily

HOST_RESOURCES_DAILY_ANONYMIZED (Learn more)

Use HOST_RESOURCES_DAILY_ANONYMIZED to view data regarding CPU and memory usage by devices (HRC, not the usage by one application), without PII (personally identifiable information).

Example: https://my-demo-odata.aternity.com/aternity.odata/v2/HOST_RESOURCES_DAILY_ANONYMIZED?$format=json&$stream_id=my_resources_daily_anonymized

INSTALLED_SOFTWARE_CHANGE_LOG(Learn more)

Use INSTALLED_SOFTWARE_CHANGE_LOG to check and analyze the installed software related changes on each Windows physical monitored device, along with its version and deployment date. It includes all software deployed on the device but does NOT include Universal Windows Platform (UWP) applications (learn more).

Learn more.

LICENSE_EVENTS (Learn more)

Use LICENSE_EVENTS to view a device's most recent request to Aternity for a monitoring license, and if rejected, it displays the reason.

MACHINE_BOOT_COMPONENTS

REMOTE_DISPLAY_LATENCY_DAILY (Learn more)

REMOTE_DISPLAY_LATENCY_HOURLY or REMOTE_DISPLAY_LATENCY_DAILY return aggregated values of remote display latency for each device type in each location. For example, you can check the daily latency in one or several locations, for one or all device types.

REMOTE_DISPLAY_LATENCY_HOURLY Learn more

REMOTE_DISPLAY_LATENCY_HOURLY or REMOTE_DISPLAY_LATENCY_DAILY return aggregated values of remote display latency for each device type in each location. For example, you can check the daily latency in one or several locations, for one or all device types.

REMOTE_DISPLAY_LATENCY_RAW (Learn more)

REMOTE_DISPLAY_LATENCY_RAW returns average values of remote display latency for each device type in each location. For example, you can check the latency in one or several locations, for one or all device types.

SERVICE_DESK_ALERTS_RAW (learn more)

SERVICE_DESK_ALERTS_RAW returns all service desk alerts which occurred in your organization. A service desk alert (SDA) indicates that the same health event occurred several times on the same device within a certain time. Aternity sends SDAs to draw attention to devices which suffer repeated application errors, system crashes or hardware issues. Use this API to analyze for patterns on similar types of alerts, and to check for correlations with device attributes, subnet or location.

SKYPE_CALLS_RAW (Learn more)

SKYPE_CALLS_RAW returns the raw list of Skype for Business calls. Use this API for analysis of Skype for Business performance, including in-depth understanding of factors affecting call quality and the frequency of dropped calls.

WIFI_STATISTICS_HOURLY (Learn more)

WIFI_STATISTICS_HOURLY tracks the customer experience of WiFi in your enterprise by returning an hourly summary of the WiFi details like its speed, signal strength, and access points for each monitored device.

Learn more.

WINDOWS_MACHINE_BOOTS (Learn more)

WINDOWS_MACHINE_BOOTS returns the boot times of all monitored Windows devices: the total boot time, the machine boot, and the user logon time.

USER_AUDIT (Learn more)

USER_AUDIT returns a list of users and the times that they successfully signed in, so that you can audit or monitor the people accessing Aternity.

A Sample Script

Aternity provides a sample script that uses the REST API synchronization feature. The sample script absorbs communication errors, such as server disconnections, and picks up where you left off last time with no error messages using ‘@odata.nextLink’. To obtain data updates through API, the script “calls” it on a regular basis. You do not need to define the paging size, the script continually brings new data over time using ‘@odata.nextLink’.

To run the sample script:

  • Ensure you have Python 3.0 with the following Python libraries: Requests and Validators.

Prepare the following parameters to use in the script:

  • STREAM_ID – the unique name of the stream

  • API_NAME – the name of the Aternity REST API

  • BASE_URl – the URI of the OData service (for example, https://my-demo-odata.aternity.com

  • REST_API_USERNAME – a username with permissions to use REST API

  • REST_API_PASSWORD – the password for the above user

Tip

You can use the provided below sample script. Copy it to the plain text editor, like Notepad ++ and save as full_sync_sample_script.py. The sample script writes data to a csv file.

All data transferred between the API and client applications is formatted in JavaScript Object Notation (JSON). You can use the XML format as well, but it is less efficient than JSON and more complicated for processing.

A sample script for a full synchronization of Aternity REST API OData
import csv
import time
import traceback
import urllib.parse
from typing import Callable, List, Dict

import requests  # requires pip install requests
import validators  # requires pip install validators

# Parameters
STREAM_ID = "test_Stream_id"  # define your stream id
API_NAME = ''  # define the odata api name
BASE_URl = ''  # the Aternity rest api base url for example https://my-demo-odata.aternity.com
REST_API_USERNAME = '' # the aternity rest api username
REST_API_PASSWORD = ''  # the aternity rest api password

# Constants
NEXT_PAGE_KEY = '@odata.nextLink'
DATA_VALUES_KEY = 'value'
BASE_API_URI = '/aternity.odata/v2/'+ API_NAME
FULL_SYNC_QUERY = BASE_API_URI + '?$format=json&$stream_id=' + STREAM_ID
FAILURE_SLEEP_TIME_IN_SECONDS = 60

# Globals

csv_file = open(file='full_sync_result.csv', mode='w', encoding='utf-8-sig')
csv_writer = csv.writer(csv_file, lineterminator='\n')
write_headers = True
session = requests.session()


# helper function to get the initial query that returns the distinct list of installed applications
def get_full_sync_start_query_url() -> str:
    base_url = BASE_URl[:-1] if BASE_URl.endswith('/') else BASE_URl
    if not validators.url(base_url):
        raise ValueError('base url {} is not valid'.format(BASE_URl))

    return base_url + FULL_SYNC_QUERY


def full_sync_result_handler(data: List[Dict]):
    global write_headers, csv_writer
    for row in data:
        if write_headers:
            header = row.keys()
            csv_writer.writerow(header)
            write_headers = False
        csv_writer.writerow(row.values())
    print('wrote {0} lines'.format(len(data)))


def full_sync_view(result_handler: Callable[[List[Dict]], None]):
    next_link = get_full_sync_start_query_url()
    print('starting query rest api for url {0}'.format(next_link))

    while next_link:
        start = time.time()
        response = session.get(url=next_link, auth=(REST_API_USERNAME, REST_API_PASSWORD))
        end = time.time()
        print('request for url {0} took {1} sec'.format(next_link, end - start))
        start = time.time()
        response.encoding = 'utf-8'
        try:
            response_json_parsed = response.json()
            if DATA_VALUES_KEY not in response_json_parsed:
                print('values key is not in the json the response we got is:')
                print(str(response.text))
                raise Exception
            data = response_json_parsed[DATA_VALUES_KEY]
            next_link = response_json_parsed[NEXT_PAGE_KEY]
            result_handler(data)
        except Exception as e:
            print('Exception Raised response text : {0}'.format(str(response.text)))
            if response.text and "Stream requests do not support out-of-order" in str(response.text):
                next_link = get_full_sync_start_query_url()
            print('Sleeping for {0} seconds due to previous error'.format(FAILURE_SLEEP_TIME_IN_SECONDS))
            time.sleep(FAILURE_SLEEP_TIME_IN_SECONDS)



try:
    full_sync_view(full_sync_result_handler)
except Exception as e:
    traceback.print_exc()
    traceback.print_stack()