1. Aeris
  2. APIs
  3. AerSight Streaming

 

Install AerSight

Avatar
Leslie Johnson

AerSight is a product that you can add to your connectivity subscription. It allows you to capture raw device data in your company's Google Cloud Platform account. Using this data, you can view predefined Data Studio reports, create your own Data Studio reports, or directly read the raw data.

AerSight Prerequisites

To use AerSight, you must satisfy the following prerequisites:

  • You must have active provisioned and billed IoT devices in use.
  • You must have an active Google Cloud Platform account.
  • You must have at least 1 subnet in the region which supports the DataFlow and BigQuery dataset. For a list of supported regions, refer to the following links:
  • Your GCP project must have cloud NAT enabled, or the subnet should have private Google access enabled.
  • You must have an existing or create a new Service Account or User with the following roles:
    1. Project IAM Admin
    2. Cloud Functions Admin
    3. Role Administrator
    4. Editor
  • Technical knowledge to maintain a GCP project.
  • Familiarity with the Google Cloud Shell and some experience with common Unix commands, such as wget, ls, chmod, and unzip.

AerSight Onboarding

  1. Subscribe to AerSight.
  2. Contact Aeris Sales Engineering to exchange information about your devices and GCP account. Aeris will request the following information:
    1. The GCP Project ID.
    2. The region in which to create the DataFlow job and BigQuery dataset. DataFlow and BigQuery are required to be in the same region.
    3. The subnet used by the DataFlow job to create virtual machines as needed.
    4. The BigQuery data retention period (optional). The default is 90 days.
    5. The DataFlow maximum number of workers. The default is 2.
  3. Wait for Aeris Sales Engineering to execute the initial provisioning command. Aeris will then send you an email with a link to a Google Cloud Services location where you will find a ZIP file. The email will also provide you with a URL which you will use in step 8 below to run scripts in your GCP environment.
    Note: The remaining steps below require that the user logged in has the following roles: Project IAM Admin, Cloud Functions Admin, Role Administrator, and Editor.
  4. Download the ZIP file and give it any name of your choice.
  5. Start GCP.
    Note: The next set of steps will take 30 minutes.
  6. Go to the project you have designated to be used with AerSight. The project must adhere to the pre-requisites noted above.
  7. Select the icon to Activate Cloud Shell.

  8. At the Cloud Shell prompt, enter the following command, where filename.zip is the file you downloaded and named in step 4, and where URL is the address provided by Aeris in the email.
    wget -O filename.zip "URL"

For example:
wget -O provision.zip "https://storage.googleapis.com/aersight_acp_int_pre_provisioning_bucket/provisioning/...."
  9. Enter the following command to view and then unzip the ZIP file that you downloaded and named in step 4. Ignore any warning messages that you see.
    ls
unzip filename.zip

For example:
ls
unzip provision.zip
  10. Repeat the ls command to view the files that were unzipped into this location. You will see a script file named provisioning-agent-deployer.sh.
    Note: Do not change any of the contents of the script or zip file.
    ls
  11. Enter the following command to give executable rights to the script. No response is given by GCP after entering this command.
    chmod u+x provisioning-agent-deployer.sh
  12. Now execute the script by entering the following command. 
    ./provisioning-agent-deployer.sh
  13. GCP will ask you to authorize Google Cloud Shell to provide your credentials while running APIs in the script. Click Authorize.
  14. Wait while the script runs. It could take up to 30 minutes.

Changes to Your GCP Environment

The script makes the following changes in your GCP environment.

  1. Custom roles are created. The table in the section "Custom Role Definitions" provides a detailed listing of permissions provided to these roles.
    • AERSIGHT_ACP_AERSTREAM_USER
    • AERSIGHT_ACP_CUSTOM_USER_ROLE
    • AERSIGHT_ACP_PROVSIONING_USER
  2. Aeris service accounts are added to IAM in your project:
    • 716751814012@cloudbuild.gserviceaccount.com. This account is granted the role AERSIGHT_ACP_PROVSIONING_USER.
    • sa-acpcmn-prod-aerstream@aeriscom-acpcmn-prod-202006.iam.gserviceaccount.com. This account is granted the role AERSIGHT_ACP_AERSTREAM_USER.
    • sa-composer@aeriscom-acpcmn-prod-202006.iam.gserviceaccount.com. This account is granted the role AERSIGHT_ACP_PROVSIONING_USER.
  3. A new service account is added to your project:
    • aersight-acp-<aerport-portal-account-Id>-sa@<GCP-Project-Id>.iam.gserviceaccount.com. This account is granted the roles AERSIGHT_ACP_CUSTOM_USER_ROLE, Dataflow Admin, Dataflow Worker, Service Account User, and Monitoring Editor.
  4. An AerSight resource provisioning agent is deployed in your project.
  5. The script will run an API to confirm success or failure of all the provisioning actions in the script. It then automatically emails the logs to Aeris Support so they can assist if any errors occurred.

After the script completes these changes, an automated script is triggered to Aeris to continue the provisioning process by creating the following resources in your GCP project. This step will take up to another 30 minutes.

  • Google Cloud Storage buckets
    • aersight_acp_<GCP-Project_ID>_staging_bucket
    • aersight_acp_<GCP-Project_ID>_warehouse_bucket
  • Pub Sub Topic. This is where the data will be published.
    • aersight_acp_ipstream-flows_topic

  • Pub Sub Subscription
    • aersight_acp_ipstream-flows_subscription

  • Data Flow
    • aersight-acp-ipstream-flows-dataflow

  • Big Query
    • DataSet - aersight_acp_ds
    • Tables
      • ipstream_flows
      • device
      • hostnames
Note: The schema for these tables is listed in the section "Big Query Table Schemas."

At the end of the provisioning processing, a device snapshot of device data for all non-cancelled devices is published into your project.

Custom Role Definitions

Three roles were created by the provisioning script. The following table lists the permissions defined for each role.

Role Name Permissions
AERSIGHT_ACP_AERSTREAM_USER
  • pubsub.subscriptions.consume
  • pubsub.topics.publish
AERSIGHT_ACP_CUSTOM_USER_ROLE
  • bigquery.datasets.create
  • bigquery.datasets.get
  • bigquery.tables.create
  • bigquery.tables.get
  • bigquery.tables.getData
  • bigquery.tables.updateData
  • compute.machineTypes.get
  • dataflow.jobs.cancel
  • dataflow.jobs.create
  • dataflow.jobs.get
  • dataflow.jobs.list
  • iam.roles.delete
  • pubsub.subscriptions.consume
  • pubsub.subscriptions.create
  • pubsub.subscriptions.delete
  • pubsub.subscriptions.get
  • pubsub.subscriptions.list
  • pubsub.topics.attachSubscription
  • pubsub.topics.create
  • pubsub.topics.delete
  • pubsub.topics.detachSubscription
  • pubsub.topics.get
  • pubsub.topics.list
  • pubsub.topics.publish
  • storage.buckets.create
  • storage.buckets.delete
  • storage.buckets.get
  • storage.objects.create
  • storage.objects.delete
  • storage.objects.get
  • storage.objects.getIamPolicy
  • storage.objects.list
AERSIGHT_ACP_PROVSIONING_USER
  • bigquery.tables.update
  • bigquery.tables.updateData
  • storage.buckets.create
  • storage.buckets.get
  • storage.objects.create
  • storage.objects.delete
  • storage.objects.get
  • storage.objects.list

Big Query Table Schemas

The Big Query tables created during the provisioning process have the following schemas.

Device Table

Field Name Type Mode
device_id INTEGER NULLABLE
subscription_id STRING NULLABLE
msisdn INTEGER NULLABLE
imsi INTEGER NULLABLE
primary_min INTEGER NULLABLE
imei STRING NULLABLE
device_status STRING NULLABLE
carrier_provider STRING NULLABLE
technology STRING NULLABLE
product_name STRING NULLABLE
activation_date DATE NULLABLE
rate_plan_name STRING NULLABLE
pool_name STRING NULLABLE
report_group INTEGER NULLABLE
access_fee FLOAT NULLABLE
custom_field_1 STRING NULLABLE
custom_field_2 STRING NULLABLE
custom_field_3 STRING NULLABLE
custom_field_4 STRING NULLABLE
custom_field_5 STRING NULLABLE
iccid STRING NULLABLE

Ipstream_flows Table

Field Name Type Mode Description
device_id INTEGER NULLABLE Aeris device identifier
imsi STRING NULLABLE International mobile subscriber identity (for GSM and LTE devices)
msisdn STRING NULLABLE Mobile Station International Subscriber Directory Number
primary_min STRING NULLABLE Primary Mobile Identification Number (for CDMA device)
meid STRING NULLABLE Mobile Equipment Identifier (for CDMA device)
technology STRING REQUIRED CDMA, GSM, LTE
session_id STRING NULLABLE Session identifier of corresponding AAA record
source_port FLOAT NULLABLE Port used by connection initiator
session_start_time TIMESTAMP NULLABLE Session start time of corresponding AAA record
source_ip STRING NULLABLE IP address of the connection initiator
network_protocol STRING NULLABLE Network protocol used for data exchange
application_protocol STRING NULLABLE Interpreted application protocol based on network protocol and destination port. "Proprietary" if unable to infer.
matched_ip STRING NULLABLE Fixed set of values (Source, Destination). If Source: Connection initiated by device; If Destination: Connection initiated by server.
destination_ip STRING NULLABLE IP address of the connection termination endpoint
destination_port FLOAT NULLABLE Port of the connection termination endpoint
destination_host STRING NULLABLE Host name inferred from destination IP
uplink_data FLOAT NULLABLE Data uploaded by the source (in bytes)
downlink_data FLOAT NULLABLE Data downloaded on the source (in bytes)
eventdate TIMESTAMP REQUIRED Partitioned column. granularity at the hour level
received_time TIMESTAMP NULLABLE Timestamp when the flow was received
product_name STRING NULLABLE Aeris product name like Global SIM, CDMA and Dual-Mode A-LH.

Hostnames Table

Field Name Type Mode
ip_address STRING REQUIRED
hostname STRING REQUIRED
Have more questions? Submit a request

0 Comments

Article is closed for comments.