DataTwinLabsAgricultural data platform on Google Cloud
Production ELT for an African-led agri-data company: dozens of sources into a governed BigQuery warehouse that powers analytics and an AI assistant.
01 — The problem
Farm, market and partner data arrived through APIs, spreadsheets and databases with no shared model. Analysts spent their time fixing numbers instead of using them.
02 — What I built
- ✦Managed connectors with Airbyte plus custom Python extractors for REST APIs
- ✦Airflow orchestration with retries, SLAs and alerting
- ✦Layered dbt models (staging → intermediate → marts) with tests, docs and lineage
- ✦Partitioned, clustered BigQuery tables tuned for cost and query speed
03 — What changed
One trusted warehouse behind the company's dashboards and its AI question-answering tool, with failures caught before anyone sees a wrong number.
04 — Tools
GCPBigQuerydbtAirflowAirbytePythonSQLTerraform
Next project
Traffic forecasting with MLOps