01What I built
An informative assistant, and the server behind it.
WorkPerks is a corporate benefits app. The assistant inside it says what the company offers, what a benefit covers, and what someone is already holding.
Three weeks of it went on finding where the build met the specification only on paper.
Claude Code wrote every line, and the three specification documents behind it. Section 03 is that process.
This page is the optional deep dive
The case study itself is the main page. Come here for the engineering behind it:
- 02 the prototype’s tools, running in this page
- 03 how the spec, the build and the audits went
- 04 the five layers, by filename
- 05 why a proxy had to exist, and its routes
- 06 six supplier behaviours found by running it
- 07 exactly what the model can see
- 08 every capability, and what it renders
- 09 what happens when the model is down
02Try it
The tools are running in this page. Ask them something.
The search, the balances, the orders and the renderer have been lifted out of the build and dropped into this page, unchanged. They are reading the same catalogue the employee app posts with every question: 35 benefits, 27 of them switched on, eight orders and a wallet.
The model is the one thing that could not come with them. Everything you see below the question was written by the app.
Tool chosen none yet
Questions it handles
No model runs on this page. Your typing is routed by the keyword rules the app falls back to when every model has failed. Type a situation, like “my back has been hurting”, and you will see how far keywords get: this layer scored 3 of 20 on situation questions where the model scored 18. The main page replays four routing decisions the model made in a recorded run.
Where a question goes
Counts are the 268 generated scenarios. Anything the router cannot place lands in the last group, which names what it cannot do rather than guessing.
Two kinds of no, and they must never sound alike
Understood, and still no
Name the thing
Ask it to cancel a gym booking. It understood perfectly, so it says which thing it cannot do and offers the screen where you can.
Telling you it did not follow the question would send you off to rephrase something that was already clear.
Genuinely did not parse
Say so, then help anyway
Type something it cannot place and it says it is not sure, then shows live benefits as rows, category chips and a way to browse.
A shrug with nothing under it is the one answer that leaves someone worse off than before they asked.
03How it was built
Three specification revisions in three weeks, each rewritten against what the live systems returned.
Vibe-coded start to finish with Claude Code: the React screen, the server, the tools, the proxy, the renderer, the test suite, and the three specifications that steered it.
The loop was never specify and accept. It was specify, build, run the result against the live database and the supplier’s API, and rewrite the specification against what those returned.
Three revisions in three weeks. The first written before anyone read the schema, the second against the live database, the third once the app proved able to answer from the state it already posts with each question.
Revision one, written before anyone read the database
Specified
18
tools. Ten already existed, with the semantics it proposed to invent.
Actually to build
8
Reading the schema first removed more than half the work.
What I brought
The parts a prompt does not produce
- An architecture
- The model routes, the app answers. The currency rule, the masking rule and the provider-independence of an answer all derive from it.
- The rules it cannot break
- No writes. One currency per answer. No code ever spoken. Each moved from a prompt request to a refusal in code.
- Establishing the facts
- Reading the live schema before specifying against it, and recording what the supplier API returned rather than what its guide stated.
- Reversing on evidence
- The specification had the assistant issuing vouchers. Three of six issued orders were delivered under a different brand than the one requested, so it now hands off to the voucher screen instead.
What Claude Code brought
The code and the documents
- The code
- Frontend, API route, tools, proxy, renderer, fallback ladder, suite. Every file in section 04.
- The documents
- Three specification revisions, the integration notes, the internals reference this page draws on.
- Turnaround
- A respecification in the morning came back running the same day. That is what made three revisions affordable.
Agents were pointed at the build too, not only used to produce it
-
A separate agent audited the build against the spec
It found nine defects. One was a guarantee the specification asserted and the code did not provide: route validation was described as failing closed, but validated action buttons only, leaving the row links an employee actually clicks unchecked. No stake in the implementation it reviewed
-
The held-out questions were written by an agent denied the answer key
Twenty-five questions phrased as situations, with the search vocabulary file withheld, and the list committed before any were run. Twenty have a benefit that covers them; five test answers to things nothing covers. The file was not kept afterwards, so the run cannot be repeated as it stands. A set written by whoever tuned the search measures the tuning rather than the search. Committed first, so it could not be edited into passing
-
Then the tests themselves were audited
One instruction: identify checks that pass while the property they assert is false. Four of eight did. One counted apologies in copy that contains none, so it had passed every run while testing nothing, and was deleted rather than repaired. A check counts only once it has been observed failing
The suite is generated from the catalogue, so it cannot fall out of date silently
Every benefit, variant, category and funding type turns into questions on its own, across nine families. Eleven flat rules then read each answer back: no two currencies together, no route that leads nowhere, no unmasked code, no count that disagrees with the rows under it.
Scenarios
268
generated from the live catalogue, not written by hand.
Real answer, no rule broken
261/268
generated questions that ended in a real answer and passed all eleven automated rules, up from 251. Of the seven left, five are correct refusals.
Situation questions, held out
18/20
reached the right benefit with the model, against 3 of 20 for keyword search on the top result. Measured once, on 1 September.
The failure mode of this method is not defective code, which review rejects readily. It is plausible code beneath a passing suite, performing something marginally different from the specification.
04How it works
Five layers, and only one of them is allowed to contain a model.
The model decides which tool. The app decides what the answer looks like. Those two never swap. Every item below is a file or a service that exists in the build.
-
Client
The employee’s phone
Two surfaces, one server.
- Chat.tsxThe screen. Thread state, every block renderer, the send loop.
- chat.tsBuilds the context posted with each question: benefits, balances, orders, and the list of screens that exist.
- chat-shim.jsThe older HTML client. Intercepts the navigation sentinel, anchors the newest message in view.
-
API route
Everything before a model runs
A greeting or a flood never reaches the provider.
- limit.tsCaps how many turns a conversation gets a minute, before any model runs.
- isGreeting“hi”, “ok”, “?”, an emoji. Answered without a model.
- redactForModelThe data boundary. It runs on the server, over whatever the client sent.
- audit.tsAppend-only trail. Tool, timing, outcome, argument names. Never values, never results.
-
Decision
Where the model lives
Two calls at most, both wrapped against provider failure.
- classifyIntentSends the rules and the tool schemas. Reads back a tool name and its arguments.
- composeAnswerOptional. Writes one sentence, thrown away if it states a figure the data does not have, or slips into system voice.
- withProviderRetries after the delay the provider specifies, then falls back to a second model, then returns null.
- safetyNetRouteKeyword rules, below the model rather than above it.
-
Tools
One switch, one case per tool
Data, or the reason there is none.
- tools/app.tsSeven. Catalogue search, one benefit, definitions, orders, balances, screens.
- tools/vouchers.tsFour, against the supplier. Code masking lives here.
- tools/about.tsOne. Eight whitelisted passages about the app itself.
-
Data and render
Where facts come from
One tool result, two outputs, one renderer.
- appContextPosted by the page. Benefits, balances, orders, screens, search vocabulary.
- proxy :8787The voucher supplier. The only network call any tool makes.
- render.tsSix block kinds for the app, twelve components for the other surface.
One question in, one rendered answer out
Two model steps. Everything else is application code, identical whichever provider is configured.
05The API proxy
Three independent reasons the supplier API cannot be called from a browser.
The employee app reads everything else straight from the browser. The gift-voucher supplier’s API is the one exception, so a small Node proxy sits in front of it. It keeps both logins, does the AES encryption, holds the auth token for thirty minutes at a time, and hands the app four read-only routes on localhost.
The assistant never holds a credential. It calls one local endpoint and gets JSON back.
-
01
The key would ship in the page
The API expects an AES-256-CBC encrypted body on its stock and voucher calls. Performing that in the browser would ship the key, the initialisation vector and the login to every employee who opens developer tools.
-
02
The egress IP is allowlisted
The supplier rejects unrecognised origins with
IP not whitelisted. A browser calls from whatever network the employee happens to be on, which no allowlist can enumerate. -
03
CORS is closed
The API sets no cross-origin headers for browser origins. With the first two conditions resolved, the request would still be blocked before dispatch.
Seven routes exist. The assistant reaches four.
| Proxy route | Encrypted | What it returns | Chat |
|---|---|---|---|
| Brands | No | The catalogue, merged across two supplier accounts | Yes |
| Stock | Yes | Availability per brand and denomination | Yes |
| Stores | No | Where a brand is accepted, by town | Yes |
| Vouchers | Yes | What is held, codes masked on the way out | Yes |
| Accounts | No body | Which credential sets are configured | No |
| Status | Yes | Reconciles one order after the fact | No |
| Pull | Yes | Issues a voucher. The only transacting route, and no tool references it | No |
Issuing is the only write on the supplier, and the only POST. The one thing that calls it is the app’s own buy button, and no tool the assistant has points at that route.
06What the API taught me
Six supplier behaviours the integration guide does not document.
All six were found by running the integration against the supplier’s API, on the account I had access to. One contradicts the vendor’s own documentation outright.
A wrapper written from the documentation alone would have got these wrong, and one of them would have double-spent.
-
Issuing is asynchronous, and a non-zero code does not mean failure
The first authorised issue returned code “2812”, which appears in no error table, and no voucher. The order was queued, not rejected. The outcome is only knowable by polling the status call; reissuing the request purchases a second voucher. Poll, never retry. Nothing was consumed
-
An error documented as one thing meant another
The code the vendor’s own catalogue calls “inactive product” comes back over and over while an order is still settling. Read as a rejection, it would have killed good orders. It means pending. Contradicted the vendor’s own catalogue
-
Settlement is inconsistent, so the client needs three states
Four orders settled in eighty to a hundred seconds. The fifth returned immediately. So the client keeps three states, pending, issued and failed, and holds on to the order id to check back against. Why the schema carries three statuses, not two
-
The stock call does not decrement
Every brand reported unchanged availability after a successful purchase. It reports what the catalogue holds, not what has been set aside, so it cannot confirm a purchase and should not be read as a failure either. Two plausible inferences, both wrong
-
The brand delivered is not always the brand ordered
Three of six issued orders were delivered under a brand other than the one requested. It may be a quirk of the account, and I could not rule that out, so the assistant checks what arrived against what was asked for, and warns with both names. The primary reason the assistant hands off rather than purchasing
-
Published addresses were not usable
The outlet records carry usable brand names and towns, and street addresses that do not resolve to real locations. The assistant names the town and never an address. The data decided the copy
07The data boundary
Two model calls, and only one of them transmits tool results.
One call sends the question. The other sends the tool result along with it, so the model can write a sentence about the data. Which one is running comes down to a single environment variable, and that is what to check first before believing anything on this page about what leaves the machine.
It runs on the server, over whatever the client sent, so a client that oversends cannot widen the boundary.
ANSWER_MODE
Crosses: the question, with email addresses and runs of ten or more digits stripped. Never crosses: a full voucher code, under either setting. The tool masks it before returning.
ANSWER_MODE=app
Routing only
- Crosses
- The redacted question, the rules, the tool schemas.
- Stays
- Balances, orders, voucher codes, benefit data. No code path sends them anywhere.
- Held by
- The architecture. Nothing in the code builds a request with a tool result in it, so there is nothing to get wrong at runtime.
ANSWER_MODE=model
Routing and phrasing
- Crosses
- The question and the full tool result, so the model can write about it.
- Stays
- Full voucher codes only. The tool masks them on the way out, so an unmasked one never exists further up.
- Held by
- Three checks, none of which need the model to behave. One catches a figure the data does not have. One catches a denial the data contradicts. One catches system voice. Any of them throws the sentence away.
08The tools
Every tool hands back data, or the reason it has none.
Six things it will not do
- It cannot actIt cannot book, buy, cancel, or change a declaration. Those tools were never written.
- One currencyRupees and health coins never appear in the same answer.
- No codesA voucher code is masked before the model ever sees it. Revealing one is a deliberate tap in the app.
- No addressesThe supplier publishes none, so it names the town rather than inventing a location.
- Live items onlyComing soon and not in your city are stated as facts, never offered as options.
- No recallEvery answer comes from a lookup. If nothing fits, it says so rather than inventing.
One tool result, two surfaces, six block kinds
- Header
- The figure or the subject on its own line, with its qualifier beneath rather than competing for the same line.App
- Sentence
- The only part a model can replace, and only if every figure in it is in the data.Model, supervised
- Note
- A caveat that must not be smoothed over: a voucher under the wrong brand, an order that never went through.App
- Rows
- The data. Each row carries a validated route, so a result set behaves as a list of links.App
- Follow-ups
- Written as complete questions, because no conversation state is retained between turns.App
- Buttons
- Only to screens the app confirmed it has. No screen list means no buttons.App
09The fallback ladder
Four steps down, and the last one still answers from live data.
Every fallback in the first build guarded against faults in our own code. None guarded against the provider itself failing, which proved to be the more frequent condition. Both were observed in testing: an exhausted free-tier quota, and a model retired mid-project.
Quota is allocated per model, so a second model on the same key carries its own allowance, not a share of the first.
- Retry the same modelA rate limit, a timeout, a server errorA slower answer. It waits however long the provider asks for, not a fixed interval
- Try the second modelThe first is gone or retiredThe same answer, and no waiting, because retrying a retired model cannot work
- Keyword rulesEvery model failedA correct but narrower answer. The lookups still return live data, but fewer phrasings are understood
- Helpful fallbackNothing recognised the questionLive benefits as rows, category chips and a browse button, rather than a flat failure
The question
The rung it falls to
What the employee gets
10What I learned
Three conclusions.
-
01
The bottleneck moved from writing code to verifying it
Implementation stopped being the constraint. Establishing what was actually true, of the schema, of the supplier API, and of my own earlier decisions, did not get faster, and is now where the time goes.
-
02
Two functions call a model; the rest is ordinary application code
Routing and composition are the only model calls. Everything else is a route handler, a proxy, a switch statement and a renderer, and behaves identically whichever provider is configured.
-
03
Integrate against the live API, not the document
The supplier’s documentation and its behaviour disagreed, as section 06 shows. Acting on one finding as documented would have issued a second voucher.