ACROPOLIS Data Processing & Visualization

The ACROPOLIS (Autonomous and Calibrated Roof-top Observatory for MetroPOLIton Sensing) network spans 20 roof-top CO2 sensor systems located in and around Munich. The network is run by the open source software ACROPOLIS-edge. The project is part of ICOS Cities, funded by the European Union's Horizon 2020 Research and Innovation Programme under grant agreement No. 101037319.

Input data for the processing pipeline is downloaded from the ThingsBoard Open-Source IoT platform using the ThingsBoard-Downloader.

This repository contains a Python based processing pipeline realized with performant Polars Dataframes and a collection of Jupyter notebooks to visualise the ACROPOLIS network data. Python scripts are statically typed and continuously checked using mypy via GitHub Actions.

Key Pipeline Features

• ✅ Customizable data processing via JSON configuration
• ✅ Efficient storage using Parquet files for fast querying
• ✅ Performant processing using Polars DataFrames
• ✅ Dilution and calibration correction of $CO_2$ data
• ✅ Time series outlier detection via Hampel filter
• ✅ Easy CSV reformatting and header generation to ICOS format
• ✅ Automated ICOS Cities data portal upload

Structure

• data/: Contains input and output data directories.
• pipeline/: Contains the processing pipeline scripts and configuration files.
• notebooks/: Contains Jupyter notebooks for data visualization.
• scripts/: Contains utility scripts for running type checks and other tasks.

Installation

Prerequisites

• Python 3.12 or later
• Poetry installed (pip install poetry)
• Local data from ThingsBoard-Downloader: Repository

Set up the virtual environment and install dependencies

python3 -m venv .venv  # Create virtual environment
source .venv/bin/activate  # Activate it
poetry install --with dev  # Install dependencies

Run the pipeline

Instructions on how to set up the pipeline can be found in the pipeline README.

python pipeline/01_acropolis_postprocessing.py
python pipeline/02_timeseries_despiking.py
python pipeline/03_L1_write_csv_icos_cp.py
python pipeline/04_L1_upload_csv_icos_cp.py

Run Type Checks (MyPy)

To ensure type safety and catch potential errors, run:

bash scripts/run_mypy.sh

Related Work

Aigner et. al.: Advancing Urban Greenhouse Gas Monitoring: Development and Evaluation of a High-Density CO2 Sensor Network in Munich. ICOS Science Conference 2024, Versailles, France, 10.-12. Sept, Link