• Usage
• Variables
  • Project-wide
  • Export
  • Load
• Schemas
  • Blocks
  • Transactions
  • Logs
  • Receipts
• Credits

To use with terragrunt-icon-analytics:

• copy contents of ./dags folder into /etc/airflow/dags and ensure airflow is the owner of files

sudo cp ./dags /etc/airflow/dags  # DAGs directory in ~/airflow/dags for docker based deployment
sudo chown -R airflow:airflow /etc/airflow/dags

• install requirements as the airflow user

sudo su - airflow
pip3 install -r requirements.txt
exit

• restart airflow-scheduler and airflow-webserver to load new packages

sudo systemctl restart airflow-scheduler
sudo systemctl restart airflow-webserver

# Or if using docker based deployment 
docker-compose down && docker-compose up -d 

• Set mandatory variables in the Airflow UI.
  • icon_cloud_provider: aws
  • icon_database: postgres
  • icon_output_bucket: whatever-your-bucket-is-named
  • icon_provider_uris: https://ctz.solidwallet.io/api/v3
• configure postgres connection in connections tab

Further information about the data used in the schemas is available in the ICON JSON RPC documentation.

Primary keys for each table are indicated in bold.

Caution should be exercised when using the timestamp field directly from the API. The time resolution of the epoch timestamp changes throughout. For ease of use, it is suggested to perform a transformation on this column in all tables to use only the left 16 digits (corresponding to the epoch timestamp in seconds).

A suggested transformation query is:

select left(timestamp::text, 10)::int8 as timestamp_s

and can be used to create a transformed column in a new table or materialized view.

ICON ETL Airflow was written by Richard Mah (@shinyfoil) for Insight Infrastructure, and was based on Ethereum ETL Airflow. The development was funded by the ICON Foundation.