Skip to content
SketchLabs.
AI engineering

We build the evaluationharness first.

Most AI projects fail quietly. A retrieval system produces a confident, fluent, wrong answer and nobody notices until a customer does. Our approach is to make that impossible: measure before you ship, keep every answer traceable, and let the system say it does not know.

Our commitments

Evaluation before features

The evaluation harness ships before the feature does. If you cannot measure whether an answer got better, you have changed the system, not improved it.

A labelled regression suite in your repo, run on every prompt or model change.

Shadow mode first

The system runs against live traffic silently for at least a week before it answers a single real question.

A scored comparison of shadow output against human answers, with the accuracy number in writing.

Ability to decline

Confidence thresholds that escalate to a human instead of guessing. A refusal costs seconds; a confident wrong answer costs a customer.

Every answer carries a confidence score, and low-confidence queries route to a person.

Full traceability

Every answer stores the retrieved documents, their scores, the prompt and the model version.

Any historical answer can be reproduced exactly, which is usually a legal requirement anyway.

Cost treated as a first-class metric

Cost per resolved task is tracked from week one, alongside accuracy, in the same dashboard.

A cost-per-outcome figure you can put next to the accuracy figure and trust.

Boring retrieval over clever prompts

Most retrieval failures are data problems, not prompt problems. We fix chunking and metadata before reaching for prompt engineering.

A chunking strategy chosen against a real query set, with the failing queries documented.

Retrieval architectures

"RAG" covers a lot of ground, and the honest answer to "which one do I need?" is usually "the simplest one that passes your evaluation set". These are the patterns we deploy, in ascending order of cost and complexity.

  • 01

    Baseline RAG

    Embed, retrieve top-k, generate. The honest starting point.

    Use it when

    Your corpus is small, well-structured and reasonably homogeneous — documentation, a policy library, a help centre.

    The trade-off

    Fails quietly on the questions you care most about. It has no way to say "I do not know", so it will confabulate a confident answer from three loosely related passages.

    • Embeddings API
    • pgvector
    • Basic retrieval loop

  • 02

    Hybrid retrieval

    Semantic and lexical search fused with reciprocal rank fusion.

    Use it when

    Your users search with exact identifiers — part numbers, error codes, product SKUs, names.

    The trade-off

    Roughly doubles the retrieval budget and adds a fusion parameter that needs tuning against a real query set.

    • pgvector
    • Postgres full-text
    • RRF
    • Cross-encoder reranking

  • 03

    Agentic retrieval

    A model plans queries, calls tools, and iterates until it can answer or gives up.

    Use it when

    Answering genuinely requires multiple lookups — cross-referencing records, or a question whose scope you cannot know in advance.

    The trade-off

    Cost and latency both multiply per hop, and it is much harder to evaluate. We cap it at a fixed number of iterations for this reason.

    • Tool calling
    • Query planning
    • Iterative retrieval
    • Hard iteration cap

  • 04

    Entity-aware RAG

    Retrieval anchored to a knowledge graph of real entities and their relations.

    Use it when

    Questions hinge on relationships — "which of these suppliers also serves that region", "what changed on this account".

    The trade-off

    Requires reliable entity data to start with, and graph upkeep becomes an ongoing cost rather than a one-off.

    • Knowledge graph
    • Entity resolution
    • Hybrid retrieval

  • 05

    Self-correcting RAG

    The answer is retrieved, checked against its sources, and revised if the check fails.

    Use it when

    Hallucination risk is the dominant concern — regulated advice, legal or clinical context.

    The trade-off

    Doubles inference cost for a measurable accuracy gain, but the checker needs its own evaluation or you are just adding confidence.

    • Citation verification
    • Answer grading
    • Revision loop

Models we build against

We are deliberately not model-agnostic in the unhelpful sense — we have opinions backed by measurement. Where a task is routine, we will argue for the cheapest model that clears the bar, because the expensive one is usually buying accuracy you do not need.

OpenAI

  • GPT-4.1
  • GPT-4o mini
  • o3
  • text-embedding-3-large
Best for
Strongest general tool-calling and instruction following, plus the most reliable structured output we have measured.
Avoid when
You need data residency outside approved regions, or per-token cost is prohibitive at your volume.

Anthropic

  • Claude Sonnet
  • Claude Opus
  • Claude Haiku
Best for
Long-context work where most of the document actually matters, and careful, low-hallucination analysis.
Avoid when
You need a very small model for high-volume classification — Haiku is good but a fine-tuned classifier is cheaper.

Google

  • Gemini 2.5 Pro
  • Gemini 2.5 Flash
Best for
Very large context windows, multimodal input, and competitive cost on high-volume tasks.
Avoid when
Your evaluation shows weaker structured output reliability for your schema than frontier alternatives.

Meta (self-hosted)

  • Llama 3.3 70B
  • Llama 3.1 8B
Best for
Data that cannot leave your infrastructure, or predictable high-volume cost after the hardware is bought.
Avoid when
You do not have GPU capacity and someone who can operate it — self-hosting is an ongoing commitment.

Mistral

  • Mistral Large
  • Mistral Small
  • Codestral
Best for
Strong European-hosted options and efficient small models for classification workloads.
Avoid when
You need the deepest tool-calling reliability and cannot tolerate a higher error rate.

Open models (via a provider)

  • DeepSeek
  • Qwen 2.5
  • Mistral Nemo
Best for
Cost-sensitive workloads where a hosted open-weight model clears your quality bar.
Avoid when
Behaviour is load-bearing and you need the strongest reasoning available at that price point.

What it actually costs to run

Retrieval cost scales with usage in ways that are easy to miss during a demo. These are indicative figures from our engagements — your corpus and query shape move them, which is why we track cost per resolved task from week one.

Indicative monthly cost by retrieval pattern
PatternAt 10k queries/moAt 500k queries/moDominant cost
Baseline RAG£80 – £200£2k – £5kGeneration tokens
Hybrid + rerank£150 – £400£4k – £10kReranker + generation
Agentic (3 hops)£400 – £1k£12k – £30kRepeated generation
Self-correcting£250 – £600£7k – £16kChecker passes

Self-hosting open models flips this: high fixed cost, near-zero marginal cost. It wins past roughly 2M queries a month — and only if someone operates the GPUs.

When RAG is the wrong answer

The answer fits in a database query

If the question is "what is the status of order 4821", you need a lookup, not retrieval. RAG adds latency, cost and hallucination risk to something SQL answers exactly.

The corpus changes faster than you re-index

Sub-minute freshness requirements (live prices, seat availability) break the embed-and-retrieve loop. Retrieve the pointer, not the content — then read live.

Nobody can define a right answer

If domain experts disagree on what correct looks like, no evaluation set exists and no accuracy number is meaningful. Fix the process first; automate it second.

AI case studies

Heliostat · 2026

An evaluation harness that caught three regressions before launch

Retrieval over 14,000 support articles, gated behind an accuracy number before it answered anything.

The challenge

The team wanted to deflect routine enquiries but had no way to tell whether the assistant was helping or quietly making things worse. There was no measurement, so there was no way to improve.

What we did

  1. Built a labelled evaluation set from 2,000 historical tickets, scored by the support team who knew the answers.
  2. Tested four chunking strategies against that set rather than picking one by intuition — the obvious choice lost.
  3. Added confidence thresholds that escalate rather than guess, calibrated against the same set.
  4. Ran in shadow mode against live traffic for eleven days before enabling responses.

Outcome

Now resolves a third of routine enquiries at 94% measured accuracy. The harness caught a documentation restructure that broke chunk boundaries, and a model upgrade that was three points worse on their domain.

Ashgrove Group · 2026

Turning 40,000 scanned invoices into structured data

A vision-based extraction pipeline with a human review queue for anything below confidence.

The challenge

Four decades of scanned invoices in a filing cabinet, with no searchable record. Manual entry would take a person-year and still be inconsistent.

What we did

  1. Started with a stratified sample of 400 documents to establish a realistic accuracy baseline before committing to an approach.
  2. Used vision models for the scan quality problems rather than trying to preprocess 40,000 images into something cleaner.
  3. Routed anything below threshold to a review queue, which turned out to be roughly 6% of documents — cheap to staff.
  4. Structured the output against the existing finance chart of accounts rather than inventing a schema.

Outcome

Processed the archive in five weeks with 97.8% field accuracy. The chart of accounts was finally queryable, which changed how month-end reconciliation worked.

Precision Components · 2025

Flagging contract risk clauses across a live portfolio

Clause-level extraction against a house playbook, with every finding traceable to its source text.

The challenge

Legal review was the bottleneck in signing, and it was being done by reading every agreement in full. The risk was in the clauses nobody had time to read.

What we did

  1. Encoded the company playbook as explicit checks rather than a general "is this risky" prompt, which made results testable.
  2. Extracted clauses to their source paragraph, so a lawyer could check a finding in seconds rather than minutes.
  3. Set a deliberately conservative threshold — a false negative in a contract is far more costly than a false positive.
  4. Tracked precision per clause type separately, because "acceptable" varies by section of the agreement.

Outcome

Legal now reviews only flagged clauses. Average contract turnaround went from twelve days to four, and two liability caps that would have been signed were caught.

AI

Questions we get about AI work.

01Which AI model should we use?
The cheapest one that clears your evaluation bar. We build against OpenAI, Anthropic, Google, Mistral and self-hosted Llama, and we will recommend a specific model only after testing candidates against a set of your real queries. The expensive frontier model is often buying accuracy you do not need.
02Do we need RAG, or would fine-tuning work?
Retrieval, almost certainly. Fine-tuning changes how a model behaves; it does not teach it your documents. If you need grounding in current, specific knowledge, retrieval is the right tool — and we will tell you if you do not need either.
03How do you prevent hallucinations?
We make the system able to decline. Confidence thresholds escalate to a human instead of guessing, every answer stores the passages it was derived from, and the accuracy number comes from an evaluation set built from your real queries rather than a generic benchmark.
04What does an AI engagement cost?
A focused retrieval feature with a proper evaluation harness is a mid-sized engagement, not a sprint. The harness is not optional overhead — without it you have changed the system rather than improved it, and you cannot tell the difference.
05Can you work with our existing data science team?
Yes. We are often the engineering half of that team, building the serving infrastructure, the evaluation harness and the integration into the product while they handle modelling.
Start here

Bring us a retrieval problem.

The first conversation is usually the most useful one we have: tell us what you are trying to answer and we will tell you which pattern — including whether you need one at all.

Start a projectRead how we evaluate