Data engineering and warehousing
Analysis goes wrong most often before it starts, in the data. We build the pipelines and warehouse models that take raw, untidy source data and turn it into tables that are tested, documented and quick to query.
Signs you could use this
- Reports are slow, or break when someone changes a source system.
- Your backend stores data as nested JSON and nobody can query it comfortably.
- Analysts rebuild the same joins in every query.
- Numbers change between runs and nobody knows why.
- Data arrives by CSV and copy and paste.
What we do
- Model the data properly
- We design star schemas with clear facts and dimensions, so queries are simple and consistent. Turning messy JSON from an application backend into a tidy model is something we do often.
- Build reliable pipelines
- Extraction and loading with Airbyte or custom Python, orchestration in Airflow, transformations in SQL, and checks that raise an alert when something looks wrong.
- Keep costs and speed in check
- Partitioning, sensible materialisation and query tuning. Cutting a slow query to a fraction of its time is common work.
- Document it
- A data dictionary and lineage notes, so the next person doesn't have to reverse-engineer what we built.
What you receive
A first working model and its pipelines usually takes four to eight weeks. We can also take on a defined piece, such as one troublesome source.
- A warehouse model, with documentation
- Scheduled pipelines with alerts
- Data quality checks on the tables people rely on
- A runbook for your team
Tools we use: Airflow, Airbyte, Python, SQL, Google BigQuery, AWS Redshift, PostgreSQL, MySQL, Supermetrics.
Common questions
BigQuery or Redshift?
It depends mostly on where the rest of your stack lives and who will run it. We've worked with both, and the better question is usually which one your team can look after.
Can you improve a pipeline we already have?
Yes. We often start with a review: what's fragile, what's slow and what's costing money, then fix those in order.
Do you need access to production systems?
We prefer read-only access, or a replica, and we only touch what the project needs. Access is agreed in writing at the start.