IoT telemetry streaming ingest, warehouse & reporting on GCP.
- cloud sdk
brew cask install google-cloud-sdk
gcloud init
gcloud components update
gcloud auth application-default login- terraform
brew install terraform- ssh key paired with your gitlab account
- repo
git clone git@gitlab.com:vgoslo/iot-on-gcp
cd iot-on-gcp- beam sdk
cd beam
virtualenv env
source env/bin/activate
pip install 'apache-beam[gcp]'- create sa key and point env var to it
export GOOGLE_APPLICATION_CREDENTIALS={path to sa.json}
export TF_VAR_gac=$GOOGLE_APPLICATION_CREDENTIALS- initialize terraform
cd ..
terraform init- verify you are all set
terraform validate
terraform plan -out "main.tfplan"- create cloud infra
terraform apply "main.tfplan"create dataflow job via console
template=pubsub topic to bigquery
topic=projects/{project}/topics/telemetry
bigquery table={project}:iot.telemetry
gcs bucket=gs://{project}-dataflow-staging/tmp
NB: repo also contains templated beam job with sliding window as well as terraform resource for it
- deploy mqtt simulator
- ssh into vm, install dependencies, initilize gcp sdk
gcloud beta compute --project "{project id}" ssh --zone "europe-west1-b" "mqtt-client-{random id}"
sudo apt-get update
sudo apt-get install python-pip openssl git -y
sudo pip install pyjwt paho-mqtt cryptography
git clone https://gitlab.com/vgoslo/iot-on-gcp
gcloud init
exit- re-establish ssh tunnel, initialize device registry and auth
gcloud beta compute --project "{project id}" ssh --zone "europe-west1-b" "mqtt-client-{random id}"
cd iot-on-gcp/gce
export PROJECT_ID=$(gcloud config get-value project)
export REGION="us-central1"
chmod +x init.sh
./init.sh- run simulations to deploy IoT fleet with 20 devices posting 10,000 signals each in background
chmod +x run_fleet.sh
./run_fleet.shto deploy single device posting signals in foreground
chmod +x run_one.sh
./run_one.sh {device name} {number of signals}
models supported:
- linear regression: numeric values, i.e. value of smth
- logistic regression: binary or multiclass classification, i.e. is smth
- k-means clustering: unsupervised learning exploration
objective: predict temperature
CREATE OR REPLACE MODEL `iot.predict_temperature_v0`
OPTIONS(model_type='linear_reg') AS
SELECT
humidity,
device,
temperature as label
FROM
`iot.telemetry`
WITH eval_table AS (
SELECT
humidity,
device,
temperature as label
FROM
iot.telemetry
)
SELECT
*
FROM
ML.EVALUATE(MODEL iot.predict_temperature_v0,
TABLE eval_table)
CREATE OR REPLACE MODEL `iot.predict_temperature_v1`
OPTIONS(model_type='linear_reg') AS
SELECT
timestamp,
humidity,
device,
temperature as label
FROM
`iot.telemetry`WITH eval_table AS (
SELECT
timestamp,
humidity,
device,
temperature as label
FROM
iot.telemetry
)
SELECT
*
FROM
ML.EVALUATE(MODEL iot.predict_temperature_v1,
TABLE eval_table)
r2 increases, err decreases
i.e. device 19. ended with t 60 and hum 17. if we bump down hum t will be even higher
WITH pred_table AS (
SELECT
'fleet-device-19-1566737352000' as device,
11.0 as humidity,
1566737789 as timestamp
)
SELECT
*
FROM
ML.PREDICT(MODEL `iot.predict_temperature_v1`,
TABLE pred_table)
- destroy cloud infra
terraform destroy- stop dataflow job
- delete iot core registry