Skip to content
All projects
Data Engineering · Cloud & DevOpsJun 2024

Jogja Open Data Pipeline

A freelance data engineering project: streaming and batch pipelines that ingest Yogyakarta open-government data into a warehouse, with Grafana and Power BI dashboards on top.

No public repository

Streaming

Kafka producers poll the Jogja open-data APIs, and consumers transform the records and load them into PostgreSQL. The Confluent stack (broker, schema registry and control center) runs in Docker Compose.

Batch

An Airflow DAG pulls unstructured CSV datasets, such as health-workforce statistics, from Google Drive, stages them in MinIO object storage, cleans them and loads them into PostgreSQL.

Reporting

Grafana monitors the pipelines, and a Power BI report presents the cleaned data. The project shipped with full written documentation for the client.