All projects
Data Engineering · Cloud & DevOpsJun 2024
Jogja Open Data Pipeline
A freelance data engineering project: streaming and batch pipelines that ingest Yogyakarta open-government data into a warehouse, with Grafana and Power BI dashboards on top.
No public repository
Streaming
Kafka producers poll the Jogja open-data APIs, and consumers transform the records and load them into PostgreSQL. The Confluent stack (broker, schema registry and control center) runs in Docker Compose.
Batch
An Airflow DAG pulls unstructured CSV datasets, such as health-workforce statistics, from Google Drive, stages them in MinIO object storage, cleans them and loads them into PostgreSQL.
Reporting
Grafana monitors the pipelines, and a Power BI report presents the cleaned data. The project shipped with full written documentation for the client.