A Fireworks AI user story
An attorney, a weekend, and the docket
What one lawyer who does not program built on Fireworks AI in a weekend: a publication that reads the court filings, then grades the news against them.
We read the news about federal cases.We pull the actual filings.We summarize what they say.Then we debunk the coverage — and grade it.All with a lawyer in the loop.
Walkthrough coming soon: five minutes, narrated by the editor, with the site and the cartoons on screen. The written version is below.
The problem
A judge enters a temporary order that binds two plaintiffs. By lunchtime the headline says the law was "struck down". An indictment becomes a finding of guilt; a dissent becomes what "the court warned"; "up to 20 years" becomes a sentence nobody imposed. None of this needs bad faith. It needs a deadline and a 40-page PDF nobody had time to open.
The filings are public. CourtListener and the Free Law Project have put millions of them online. The distance between what a court did and what the news said it did is measurable, page by page, and almost nobody measures it.
One weekend, one attorney
Andrew Benard is an attorney. He has never programmed. Over one weekend in October 2026 he set out to build the thing that measures that distance, and to learn Fireworks AI by building something real on it, rather than by reading about it.
He made every design decision: which courts, which outlets, what gets published and when, what a grade means, and that nothing goes public until he has read it. The code was written by Claude Code, Anthropic's coding agent, working from those decisions and asking him whenever a choice was his to make. The pipeline runs on Shito, a home server in New York; this site runs on Cloudflare.
What got built
- News feedsRSS + Google News
Coverage from 48 outlets across the spectrum: their own feeds, plus Google News searches for the outlets that publish none. While the pipeline is proven, it replays a collected week of real stories one day at a time; the live reader is next.
- ClusteringFireworks embeddings · qwen3-embedding-8b
Stories about the same case are grouped: docket numbers and captions first, then meaning. Two headlines about the same EPA rule scored 0.73 similar; one of them against a baseball headline, 0.41.
- Ranking and taggingFireworks · gpt-oss-120b, JSON schema
Each group is scored by how many outlets covered it, then tagged: is this a federal court case, and what is it about. Four news picks a morning, chosen for variety; two more come from CourtListener (see the architecture below).
- Case identificationFireworks · DeepSeek V4.1 Flash + CourtListener
The model extracts court, docket number, parties and the filings the stories mention; CourtListener confirms the docket. Below the confidence line, the case goes to the editor rather than being guessed.
- The filingsCourtListener · supremecourt.gov
Docket entries and documents from CourtListener's RECAP archive; Supreme Court papers from supremecourt.gov.
- The memoFireworks · GLM 5.3 and Kimi K3
The model reads the filings whole and writes five sections, every fact pinned to a document and page. A checker in code rejects any draft with an uncited claim or a page that does not exist.
- GradingFireworks · the memo's writer
Each news story is graded A to F against the same filings: what it got right, what it got wrong, the exact sentence, the correction and the page.
- The judgeClaude Opus
An independent check from a different model family: does every correction cite a real passage that supports it? Unsupported issues are dropped.
- The editor's deskThe desk app
Andrew Benard, attorney, approves, edits or rejects every new memo, with a reason. Every decision is recorded.
- PublicationCloudflare Workers + D1
Approved pages are pushed to this site. Later filings and new story grades join the page; a rewrite of the memo goes back to the desk.
The result, for each case: a plain-English memo of what the documents say, a side-by-side table of how each outlet did, and a growing taxonomy of the ways the press gets law wrong. The front page is the latest of them.
How Fireworks AI carries it
Every model call in the pipeline runs on Fireworks AI, through its OpenAI-compatible chat completions API, chosen per job from Fireworks' own model guidance and then tested with live calls. The judge is the one deliberate exception: a model from a different family checks the work, so the checker does not share the writer's blind spots. Each feature says whether it is running now or still being built.
JSON-schema structured output Running
Sorting, case identification and grading all return JSON that must match a strict schema: every key required, no extras, enums where the answer is one of a few words. A starved or malformed answer is an error, never a half-answer passed along.
Embeddings Running
qwen3-embedding-8b turns each headline and snippet into 4,096 numbers. Vectors are normalised before comparison, as Fireworks' docs advise, and stories within 72 hours merge when the docket or caption matches, or when they are close in meaning and share a party, court or judge.
Long context Running
Court filings are long. The memo writers read the filings whole rather than in chunks, so a ruling on page 41 is read in the light of the motion on page 3. CourtListener's own extracted text is used; Fireworks no longer inlines PDFs.
Prompt caching with prompt_cache_key Running
The filings are the fixed prefix of every prompt about a case, and each call carries the case as its prompt_cache_key, so grading the fifth story re-reads the same documents from cache. Following Fireworks' caching rules: static text first, no timestamps in the system prompt.
Reasoning effort, sized deliberately Running
Every call sets reasoning_effort and max_tokens explicitly: low for sorting and identification, high for memos and grades. The reasoning shares the token allowance with the answer, so an undersized call thinks and then says nothing.
Tool calling for Shoki Being built
Shoki (書記, "the clerk") is the research agent: a loop on Fireworks models with tools to search CourtListener, read docket entries, fetch documents and read pages, with a step limit per case. In the editor's desk it is also a chat with read-only tools over the database, and an ask_claude tool for hard legal questions that says when it has been used.
The A/B experiment Running
The first 20 cases are an experiment. GLM 5.3 and Kimi K3 each write two versions of every memo from the same filings and cached prefix, varying one thing per five cases: structure, citation strictness, tone, reasoning depth. Claude scores every memo on a fixed checklist; the editor answers four questions per case, blind to which model wrote what.
Where the attorney sits
The models are fast and tireless. They are not admitted to any bar. The legal judgement in Law-Bunker is a person's, at four points:
- Approval. Nothing is published until Andrew Benard approves the memo. He can edit it or reject it with a reason, and every one of those decisions is recorded as the draft, the final version and why.
- The blind review. During the 20-case experiment he reads the competing memos without knowing which model wrote which, and answers four questions: which reads best, which is most accurate, whether he agrees with Claude's scorecard, and anything it missed.
- The taxonomy. The taxonomy of legal confusion grows from the grader's tags, but he names, merges and rewords the categories.
- Corrections. Reports of errors go to him; he decides, and corrections are dated and listed on the Corrections page.
His decisions also flow back into the next memo. The house style guide, STYLE.md, is revised from his answers after cases 5, 10, 15 and 20 of the experiment; he approves each revision, and the current version goes into every prompt. Its first rule: the documents are the only source.
The fine-tune loop
Built, collecting examples — not yet trained
The models are good at law in general. The point of the loop is to make one of them good at Law-Bunker in particular: its voice, and the editor's standards, which are stricter than most.
- Every decision is an example. Each approval, edit and rejection at the desk is stored from day one as the draft, the editor's final version and his reason, and each blind-review pick records which of two memos he preferred.
- Two kinds of training data. His final versions become supervised examples (SFT): this is what the memo should have said. His picks, and accepted drafts set against rejected ones, become preference pairs (DPO): this one, not that one.
- A LoRA adapter on Fireworks. A small adapter is trained on Fireworks on top of an open base model, the writer that wins the 20-case experiment, and served on Fireworks just like the model it was trained from.
- Measured, not asserted. Before it replaces anything, the adapter is compared with the base model on held-out cases it never saw in training, with Eval Protocol tests: pinpoint accuracy, unsupported claims, whether each cited page supports its sentence, adherence to the house style, and agreement with the editor's own grades.
Until it has earned its place on those numbers, the base models keep writing, and the editor keeps approving every new memo either way.
The cartoons Being built
Fireworks serves no image or video models, so a Fireworks model writes the animation and Shito draws it. GLM 5.3 writes the voice-over, with a Kimi K3 version alongside for comparison; a Fireworks model then fills in a strict scene document in a small scene language, where every field is a fixed choice, a number in a range, or text to be drawn as text. The model never writes code. A renderer turns that document into a 1080 × 1920 cartoon in the style of Microsoft Paint, deterministically: the same document and seed give the same video, frame for frame.
Image and video models reached through OpenRouter supply the pieces Fireworks does not: sprites cut from a generated character sheet, and hero shots. The stick figures themselves are always drawn by code, so poses never drift between frames or between days.
What surprised us
Recorded the afternoon the keys arrived, before any pipeline code existed. The docs were right far more often than not; these are the places they were silent or out of date.
- List the models before planning around them.
- Three model ids taken from blog posts and older docs answered "Model not found". The live model list is the only list.
- Thinking costs tokens you asked for the answer.
- With max_tokens set to 300, GLM 5.3 Flash spent all 300 thinking and returned an empty answer; finish_reason "length" was the only clue.
- Some models only think.
- GLM 5.3 and GLM 5.3 Flash refuse reasoning_effort "none": they are thinking-only. gpt-oss-120b accepts only low, medium and high.
- Grammar mode and thinking models do not mix.
- A tiny grammar to force a one-word answer came back empty every time on GLM and DeepSeek, and on gpt-oss-120b with a stray <|call|> token attached. None of that is in the docs, so the classifier uses a JSON schema with an enum instead.
- JSON schema works better than the docs say.
- The documentation says JSON-schema mode disables reasoning. In practice GLM Flash and DeepSeek Flash reasoned first and then produced valid JSON, which is better, but the token allowance has to cover both.
- A cache is broken by one second.
- Fireworks' docs warn that a timestamp in the system prompt invalidates the whole cached prefix. Law-Bunker keeps the day's date out of the system prompt; it goes in the user message.
- Tool turns need their reasoning back.
- On each tool turn the model's reasoning_content has to be sent back with the history, or it loses its train of thought.
- Choose the API that keeps nothing.
- The Responses API stores conversations for 30 days; chat completions stores nothing. Law-Bunker uses chat completions.
What's next
The first training run Planned
Once there are enough of the editor's decisions, the fine-tune loop trains its first adapter and the evaluation decides whether it writes.
Batch API Planned
Backlogs, and re-grading old stories after a style change, are natural batch jobs: a file of requests, results within hours.
Shoki at full strength Being built
The research agent's tool loop, and its chat in the editor's desk, are the next pieces to finish.
The whole architecture
The whole system, component by component: what each part does, and what it runs on. The numbered story above is the short version; this is the map.
Finding the news
- The corpus and the scoutPython · RSS · Google News
48 outlets from left, center and right, chosen by the editor; no legal or expert press, so as not to step on those toes. Outlets with no feed are reached through Google News searches, used only to count coverage. The test week was collected and curated before any pipeline code ran, and the pipeline replays it a day at a time so a story published tomorrow is invisible today.
- ClusteringFireworks · Qwen3-Embedding-8B
Docket numbers and captions are pulled out by pattern first; then headlines and snippets are turned into 4,096-number vectors and stories close in meaning, sharing a party, court or judge, are merged.
- Ranking and taggingFireworks · gpt-oss-120b
Groups are scored by how many outlets of which kind covered them, then classified with a JSON schema whose answers are fixed lists: federal court or not, and the subject.
Finding the case and its papers
- IdentificationFireworks · DeepSeek V4.1 Flash 1017 · CourtListener · supremecourt.gov
The model names the court, the docket number, the parties and the filings the stories mention; CourtListener's search confirms the docket. Cases are keyed on court and docket number, never the case name, because names change on appeal. Low confidence goes to the editor instead of a guess.
- DiscoveryCourtListener · Big Cases Bot
Two cases a morning come from the other direction: cases the Free Law Project's Big Cases Bot has flagged, whose documents are already public, matched afterwards to any stories about them. If nobody has covered it yet, Law-Bunker writes it first.
- The covered-case watchCourtListener webhooks
Every case already covered is checked at least once a day for new docket entries, through CourtListener's docket-alert webhooks (delivered to this site and collected by the pipeline) and a polite round-robin poll. A new filing re-runs the memo and the grades for that case.
- Fetching filingsRECAP · supremecourt.gov
Documents come from CourtListener's RECAP archive, using its already-extracted text, and Supreme Court papers from supremecourt.gov. A document that is not in RECAP goes on a list for the editor rather than being bought automatically.
Writing and checking
- The memoFireworks · GLM 5.3 · Kimi K3 · STYLE.md
Two models write from the same filings for the first 20 cases, each in two versions that differ in one thing at a time, and every version follows the house style guide. A checker in code rejects any draft with an uncited claim of fact or a page that does not exist, and allows one automatic repair round.
- GradingFireworks · the memo's writer
Each story gets a letter grade, what it got right, and each error with the exact sentence, the correction, the page and a category from the taxonomy. An issue quoting more than 25 words of the story is dropped in code and kept for the record.
- The judgeClaude Opus · Claude Code CLI
A model from a different family reads every memo and its grades against the cited pages and scores them on a fixed checklist; unsupported issues are dropped. One judge runs at a time.
The editor's desk
- The deskPython · PostgreSQL
A private web app, installed on the editor's phone: an inbox of drafts to approve, edit or reject; the blind review for the experiment; the taxonomy, where categories are named and merged; updates to cases already live, which can be confirmed, fixed or pulled; readers' error reports, collected from this site; and the list of documents that are not yet public.
- ShokiClaude Agent SDK · Fireworks (next)
書記, "the clerk". A research agent and a chat: today a Claude Code session driven from a phone app called the Shoki Console, one session at a time; the plan is to point the same agent at Fireworks through its Anthropic-compatible endpoint.
Publishing
- PublishPython
Approved pages, the grades and the taxonomy are pushed to this site over a private, authenticated API. The editor can pull a grade or a memo from the desk and the next run removes it here.
- This siteCloudflare Workers · D1
A small program at the edge with its own database, so the site stays up when the server at home is off. It holds only what was approved, and receives readers' error reports and court webhooks.
The short videos
- Script and scenesFireworks · GLM 5.3 · Kimi K3
GLM 5.3 writes the voice-over, with a Kimi K3 version to compare; Kimi K3 then writes the scene document in a small scene language of fixed choices, which code checks and re-times to the real recording.
- DrawingPillow · ffmpeg · Comic Neue · Gemini image models via OpenRouter
Code draws every frame in the style of Microsoft Paint and ffmpeg encodes it; the same document and seed always give the same video. Hats and props can come from one generated sprite sheet.
- Video reviewFireworks · GLM 5.3 Flash
A fast model watches a small preview of the finished video, two frames a second, and reports whether every subtitle is legible and nothing on screen is wrong or misleading. It advises the editor; it does not decide.
Running it
- The nightsystemd timers · PostgreSQL dumps · ntfy
One run after midnight, New York time; a database backup that is kept only after it has been read back in full, and copied off-site; and a morning briefing to the editor's phone.
- BrakesPython
Every paid call is checked against a hard daily allowance before it is made, using its worst case, and recorded after with what it really used. A call with no known price is refused, never assumed free. A stage that cannot do its job says so loudly rather than passing on an empty result.
- LearningPostgreSQL
Every approval, edit and rejection is recorded as draft, final version and reason, for the fine-tuning described above.
Every tool used
Everything used to build and run Law-Bunker, grouped by role.
| Tool | What it does here |
|---|---|
| Building it | |
| Claude Code | The coding agent that wrote the software from the editor's decisions, running Claude Opus 5.5 with subagents for parallel work |
| GitHub | A private repository |
| Models | |
| Fireworks AI | Every model call in the pipeline: gpt-oss-120b (ranking and tagging), Qwen3-Embedding-8B (clustering), DeepSeek V4.1 Flash 1017 (case identification), GLM 5.3 and Kimi K3 (memos, grades, scripts, scenes), GLM 5.3 Flash (video review) |
| Anthropic | Claude Opus as the independent judge, run through the Claude Code command line; the Claude Agent SDK behind the Shoki Console |
| OpenRouter | Gemini image models for the cartoons' sprite sheet |
| Court records | |
| CourtListener and RECAP | Free Law Project's archive of federal dockets and documents: search, docket entries, extracted text, docket-alert webhooks |
| Big Cases Bot | Free Law Project's list of notable cases, the source of discovered cases |
| supremecourt.gov | Supreme Court dockets and papers |
| Software | |
| Python 3.12 | The pipeline, the desk and the Shoki Console: the OpenAI SDK (Fireworks speaks its API), psycopg, pydantic, httpx, trafilatura for article text, Pillow for drawing |
| PostgreSQL 16 | The system of record |
| Docker and Compose | The database and the desk, each a small container |
| ffmpeg | Video encoding and the review preview |
| Comic Neue | The cartoons' typeface, under the SIL Open Font License |
| Playwright | Scripted browser recordings for the walkthrough, and page checks |
| Running it | |
| Cloudflare | Workers, D1 and custom domains for this site |
| systemd timers | The nightly run, the backup and the morning briefing |
| Tailscale | The private network the desk and the console live on |
| ntfy | Alerts and the briefing on the editor's phone |
| Proton Pass | Where a copy of every secret is kept |
The machines
Three ordinary machines, none of them new.
Shito, the home server
It runs the pipeline, the database, the desk and the console. Memory is the scarcest thing on it, and the design shows it: one judge at a time, one console session at a time, and the nightly run held to a fixed memory ceiling.
Lenny, the laptop
The browser: reviewing the desk and the site, and recording the walkthrough.
Pixie, the phone
The desk and the Shoki Console, each installed as a home-screen app; the briefing arrives here.
What it costs to run
| Tool | What it does | Period | Cost (USD) |
|---|---|---|---|
| Fireworks AI | Every model call in the pipeline | 2026-10 | $0.30 |
| Claude API | The independent judge | — | Not yet measured |
| OpenRouter | Image and video models for the shorts | — | Not yet measured |
| CourtListener | Dockets, filings and alerts | — | Not yet measured |
| Cloudflare | This site and its database | — | Not yet measured |