Senior Data Engineer

About Zest

The first AI-driven platform empowering food & drink manufacturers to take action against food waste. We're looking for passionate problem solvers who want to join our mission and eradicate food waste in manufacturing.

• As featured on BBC (from minute 11), The Guardian and The Grocer
• Recipients of InnovateUK grants
• Working in partnership with some of the largest food manufacturers in the UK

Why we exist: There’s 7 x more food waste at a manufacturing level vs retail, adding around 10% to food manufacturers cost of goods (a cost that is then passed down to us, the end consumer!). Most of that loss is never measured properly, it's scattered across paper records, gauges, enterprise systems and people's memory. Zest brings that data together, shows manufacturers where waste is lost on each production run, and helps them act on that data by creating insights that drive decisions faster.

Working at Zest

We’re a small, friendly team (you’ll be employee no. 6), who want to see a measurable impact from our work. We’re doing that by helping food manufacturers waste less food throughout their production process. Our culture is rooted in transparency and honesty. As we grow, we’re hiring people who are excellent at what they do and trusting them to be the expert in their area whilst in true startup style, mucking in elsewhere wherever they can support.

We’re looking for your expertise from day one, and we’ll give you the context and support to put it to use. We’ll help you understand the industry, walk you through what we’ve built and explain how everything works today. You’ll have a friendly team around you who want you to do your best work.

What you’ll work on

Your starting point will be understanding our current platform: how customer data reaches us, how we process it, and how the results reach the product. You’ll assess what’s working well, where the risks are and which changes are worth making.

A particular priority is how we provision and operate Dagster across multiple customers. We want your experience to help us make sound decisions about tenant isolation, resource allocation and running different workloads safely alongside one another.

You’ll:

• Own the foundation: Take full ownership of our data platform - improving pipeline reliability and scalability today while shaping our long-term architecture.
• Evaluate and improve the platform. Recommend what to keep, change or simplify, explain the trade-offs and deliver agreed improvements.
• Help us scale safely across customers. Work with engineering to improve how we separate customer data and access, provision resources, manage concurrent runs and backfills, and contain the impact of failures.
• Build dependable integrations. Bring different customer systems and delivery formats into tested, maintainable pipelines that fit our shared data model.
• Make processing safe and repeatable. Handle changing schemas, late or corrected data, historical backfills and reruns without losing or duplicating valid data.
• Make problems easier to investigate. Improve testing, monitoring and visibility into how data moves through the pipeline, and help the team resolve failures.
• Build trust in the data. Work with the data science team to turn business rules and validation criteria into automated checks, helping us identify questionable results even when processing succeeds.

Working with Data Science

Our data science team leads the understanding of customers’ manufacturing processes and analytical requirements. You’ll work closely with them to understand the source data and agree how it maps into Zest.

You’ll bring an analytical mindset to that work: exploring unfamiliar data, questioning assumptions, tracing how figures are produced and helping investigate inconsistencies. Together, you’ll work out whether unexpected results reflect something happening in the factory, an issue in the source data or a problem in how we’re processing it.

You’ll use that understanding to help shape our data models, transformations and validation rules, and take ownership of making the processing reliable, testable and maintainable.

The experience we're looking for

We’re looking for depth in production data engineering and the judgement to apply it in a small team.

The core experience we need is:

• Strong Python and SQL, used to build and maintain production data systems.
• Production Dagster experience. You’ve built, operated and scaled Dagster pipelines across multiple customers or independently owned datasets, with hands-on experience of tenant isolation, resource allocation and managing concurrent workloads.
• Integrating complex source data, understanding inconsistencies and bringing different sources into a clear, usable data model.
• Reliable data processing, including schema changes, incremental updates and safe reprocessing of historical or corrected data.
• An analytical approach to data quality, using exploration, automated tests, monitoring and reconciliation to investigate problems and prevent them recurring.
• Sound engineering judgement and clear communication, including evaluating existing systems, prioritising practical improvements and explaining decisions to technical and non-technical colleagues.

Our current stack includes Python, SQL, Dagster, BigQuery and GCP. Experience with other cloud warehouses and infrastructure providers is transferable.

Helpful experience

These would help you get started, but aren’t requirements:

• Manufacturing or other operational data, including ERP systems, production records or sensor data.
• BigQuery, dbt or GCP.
• Data platforms that provide analytics directly to customer-facing products.
• Infrastructure and delivery tooling such as Terraform, automated deployment and GitHub Actions.

Our Benefits

• Equity package
• Flexible holidays
• Work from abroad (1 month per year)
• Enhanced maternity / paternity packages
• Flexible working hours
• Family friendly benefits

Interview process and timing

1. Application (CV + video / VN)
2. 1st stage - Initial screening interview (with Founder)
3. 2nd stage - Technical interview (data modelling and pipeline design, plus a walk through systems you've built and run)
4. 3rd stage - Final interview and meet the team (preference to be in person)
5. Job offer