Data Stream from AerSight – Aeris

Once AerSight is installed and set up in your GCP environment, data is streamed automatically on a periodic basis (every 5 minutes).

Data is provided according to a structure as follows.

{
    "data": "<payload in CSV format, see below>",
    "id": "df88e808-f7ba-4486-9c41-f3b8cd78a23f",
    "source": "aeris/acp/ipstream",
    "specversion": "1.0",
    "type": "ipstream.data",
    "datacontenttype": "text/csv",
    "time": "2021-03-30T09:18:09.695729Z"
}

Payload format:

device_id, imsi, msisdn, primary_min, meid, technology, session_id, session_start_time, source_ip, 
network_protocol, destination_ip, uplink_data, downlink_data, received_time, eventdate, matched_ip, 
application_protocol, source_port, destination_port, product_name, destination_host

In GCP, you can view the BigQuery DataSet, aersight_acp_ds, click the device table, and click Preview to see the data.

Data Monitoring

It is recommended that you enable monitoring on the following GCP services to proactively identify any issues with the data stream.

PubSub: You can apply monitoring on the Google Cloud Metrics named subscription/num_undelivered_messages. This allows you to monitor the number of unacknowledged messages in a subscription.
DataFlow: You can apply monitoring on the data flow jobs:
- System Lag (job/system_lag)
- VM Resource utilization (CPU, Memory) (job/current_num_vcpus,job/gpu_memory_utilization)
- Job Failed / Status: Alert if it is in the failed state (job/is_failed)
- PubSub – read latency (job/pubsub/read_latencies)