Make the journey reproducible.
Airflow orchestrates ingestion and dbt runs. BigQuery stores staged and modeled data; dbt tests check agreed rules before downstream use. Monitor freshness and failures, and rerun safely when inputs change.
A DATA ENGINEERING FIELD NOTE / 001
Good data has a backstory.
A hands-on exploration of lineage, metadata and human review: from a question about a number to context people and AI can inspect.
Created by Shubham Singh · Data Engineer & Applied AI
-- An analyst asks a familiar question
$ trace net_claim_value
01 source.claims.amount
│ normalize_currency()
02 staging.claims
│ apply_adjustments()
03 analytics.claim_summary
✓ lineage found / fictional example
$ inspect metadata_
01 / THE INCIDENT
A question lands.
Where did this number come from?
An analyst spots a value that doesn't look right. The final table alone cannot explain the original field, the transformations applied, or the business rules that shaped it.
Follow the trail below. The example is fictional, but the engineering question is real: how can we give people and downstream AI enough context to understand and challenge data?
02 / TRACE → DESCRIBE → REVIEW
Open the workbench.
Start with a field. Follow its lineage. Then see how a description earns its place in the catalog.
FICTIONAL DATA · SIMULATED WORKFLOW
Metadata workbench / interactive illustration
● LOCAL DEMO02 / EXPLORE LINEAGE
Context for a description
03 / REVIEW THE DRAFT
From context to confidence.
Try the quality check, then review the regenerated draft.
01 / CHECK02 / REVISE03 / REVIEW
Simulation only: descriptions, scoring and regeneration are prewritten examples. No AI model, backend or real publication is connected. Review state lasts only in this browser session.
03 / ENGINEERING NOTES
From demo to data platform.
A possible implementation for the same problem, grounded in familiar data engineering tools. This is a proposal, not an implementation of the interactive demo.
01 / INGESTSource systemsSAP-derived data, APIs & files
02 / MODELBigQuery + dbtTransform, test & document
03 / CONTEXTMetadata storeLineage, evidence & versions
04 / REVIEWSteward approvalFeedback before publication
05 / CONSUMECatalog & AIApproved descriptions + links
Keep evidence alongside the draft.
Capture source and transformation context alongside table and column identifiers, description versions, evaluation outcomes and review status. Use dbt artifacts as a starting point for lineage; supplement and validate column-level links where coverage is incomplete.
Review before publishing.
A proposed AI-assisted step could draft descriptions using approved context, evaluate them and retry weak outputs. Stewards approve or reject drafts; feedback informs a later revision. Only approved text reaches the catalog. Apply role-based access and keep an audit trail.
Design sketch only. No Airflow, BigQuery, dbt, SAP, catalog or AI services are connected to this website. Tool choices and safeguards would be validated against the actual environment and data policies.
DESIGN THINKING / 04
Make trust visible.
A useful data foundation gives people more than a label. It gives them a way to understand and challenge it.
Show the evidence.
Connect a field to its origins and the transformations that shape its meaning.
Catch weak drafts.
Check whether a description uses available context before asking people to review it.
Keep a human in control.
Make feedback part of the next iteration, and publish only after approval.
ABOUT THE BUILDER / 05
Hi, I’m Shubham.
I build data foundations.
I'm Shubham Singh, a data engineer working across cloud data platforms, governance and applied AI. I enjoy building the foundations that help people find, understand and use data with confidence.
This independent demonstration was created while exploring CNTXT AI's Data Engineer role. It uses fictional examples and is not affiliated with CNTXT or any employer.
Connect on LinkedIn ↗