Weather Databricks Pipeline

Context

Companion to Weather Trend Forecasting on the same Global Weather Repository dataset. Where the Jupyter project focuses on Kyiv forecasting, anomalies, and statistical inference, this repo implements a lakehouse-style pipeline in Databricks: land CSV on a Unity Catalog volume, transform in SQL into Delta tables, quarantine bad rows, publish gold aggregates, and monitor with DQ checks plus a dashboard.

Pipeline

1. Bronze — read_files from the volume into weather_bronze.global_weather_raw with ingested_at / source_file.
2. Silver — parse timestamps, dedupe keys, route failures to weather_silver.global_weather_quarantine, clean rows to weather_silver.global_weather.
3. Gold — country/city/Kyiv aggregates and gold_data_profile row counts.
4. DQ — append check results to weather_dq.pipeline_checks (latest run queryable for the dashboard).

Operations

• Job: weather_medallion_pipeline (manual or scheduled re-run).
• Dashboard: Weather pipeline ops (Databricks workspace; screenshots in the GitHub repo).

What this demonstrates

ELT on a lakehouse: raw land first, transforms in the warehouse, auditable quarantine path, and explicit DQ gates suitable for analyst-facing tables—not a one-off notebook export.