We Fed 5 Open Models a Broken Checkout Replay—Here's Who Wrote the Best Fix

Why replay exports make good LLM prompts

Most coding benchmarks use synthetic LeetCode-style tasks or hand-written bug reports. Real production failures are messier: timing, third-party scripts, mobile viewports, and user behavior that no ticket writer captures fully. Session replay closes that gap.

A checkout replay gives a model the same evidence a human debugger would want—click sequence, element state, network requests, console errors, and how long the spinner sat on screen. Anonymize PII, strip payment tokens, and you have a reproducible prompt artifact you can attach to any open-weight assistant without sending live customer data to a vendor API.

What we benchmarked

A Next.js + Stripe checkout where users could double-submit “Place order” during a slow 3G response, creating duplicate PaymentIntents. The bug was confirmed in replay: rage clicks on the submit button, two identical POSTs 400ms apart, no client-side guard. We shipped a minimal reproduction repo and the same replay export JSON to each model.

Benchmark setup

We kept the protocol boring on purpose so others can rerun it. Same machine, same context window budget, same system prompt, same temperature (0.2). Each model got:

Anonymized replay export

DOM mutations, click coordinates, network HAR subset, console log, custom rage-click markers, and session metadata (browser, viewport, release tag).

Repo snapshot at the buggy commit

checkout/page.tsx, payment API route, and existing test folder—roughly 40 files, no monorepo noise.

A single instruction block

“Diagnose from the replay, propose a minimal fix, add a regression test, explain trade-offs.” No follow-up turns—one shot only.

Models tested: DeepSeek Coder V2, Qwen2.5-Coder 32B, Llama 3.3 70B Instruct, Mistral Codestral 22B, and StarCoder2 15B—all run locally via Ollama/vLLM on the same hardware class. We are not publishing token counts or latency tables here; hardware varies too much for fair comparison. The rubric below is what mattered for merge decisions.

Scoring rubric

Two engineers blind-scored each patch independently, then reconciled disagreements. Each dimension uses a three-point scale (miss / partial / pass). No aggregate “winner score”—we cared about which fixes we would actually ship.

Root-cause accuracy

Did the patch address the actual failure mode visible in the replay—double submit, stale cart state, race on payment confirmation—or did it paper over symptoms with defensive hacks?

Test coverage

Did the model add a regression test that reproduces the replay sequence? Unit tests for the handler, integration test for the checkout flow, or at minimum a clear test plan?

Regression risk

How invasive was the change? Idempotent submit guards and debouncing are low risk; rewriting the payment pipeline or removing validation is high risk.

Explainability

Noticed ancillary noise in the export

Gap

Misidentified root cause despite clear click sequence

Who wrote the best fix?

DeepSeek Coder V2 for the patch we would merge: correct root cause, client + server defense, low regression risk. Qwen2.5-Coder if your priority is test-first workflow—the regression test was the easiest to adapt into CI. For a one-shot benchmark without follow-ups, pairing replay context with a larger instruct model beat smaller code-specialized weights that latched onto irrelevant network noise.

Building a replay export for LLM prompts

You do not need a custom format. Structure whatever your replay tool exports so the model sees time-ordered evidence:

Replay segment	Include	Redact
User actions	Click targets, timestamps, rage/dead click flags	Free-text form fields, email, address
Network	Method, path, status, duration, duplicate detection	Auth headers, card tokens, PII query params
Console	Errors and warnings during the failure window	User identifiers in log messages
Context	Release version, feature flags, viewport, connection type	Session IDs linkable to accounts

Add a one-paragraph human summary at the top—what the PM thinks happened— but let the structured replay data carry the proof. Models that ignored the click timeline and over-indexed on console warnings performed worse across the board. For background on capturing this kind of context, see our session replay architecture guide.

Replay-driven debugging in your workflow

Triage with analytics

Filter sessions where checkout_started fired twice without checkout_completed. Rank by rage clicks on the submit button. Export the top affected replays.

Prompt your assistant

Attach the export to your IDE agent with the repo checked out at the release tag from the replay. Ask for diagnosis, fix, and test in one pass—then review like any other PR.

This loop mirrors how teams already use replay for human investigation; the benchmark only asks whether the handoff to an open model is safe enough to try. It is not a replacement for code review. Every patch here needed human edits before merge—especially async test timing and Stripe idempotency key wiring.

Common failure modes

Chasing analytics scripts instead of UX

Network panels are noisy. Tell the model to prioritize user actions and duplicate API calls over third-party warnings unless errors correlate in time.

Client-only fixes for server bugs

Debounce buttons help, but payment flows need idempotency on the server. Replay shows both requests landed—only one layer of defense is incomplete.

Skipping regression tests

Without a test tied to the replay sequence, you will re-discover the bug via another session export. Insist on tests in the prompt.

Where LogNroll fits

LogNroll captures the evidence this benchmark depends on: DOM replay, network timing, console output, rage clicks, and custom events you can filter before export. When checkout completion drops, you already know which sessions to bundle—not a generic stack trace from server logs alone.

Replay-driven debugging does not mean auto-merge AI patches. It means your prompts start from the same ground truth your PM and support team watch. Pair that with a simple rubric—root cause, tests, regression risk—and open models become another reviewer in the loop, not a oracle. For turning UX signals into prioritized work after you ship the fix, see heatmaps vs session replay.

Checklist: rerun this benchmark yourself

→Pick a confirmed bug with a session replay and a minimal repro repo.
→Export anonymized replay JSON with clicks, network, and console in time order.
→Use one system prompt and one shot per model—no coaching.
→Score on accuracy, tests, regression risk, and explainability—not vibes.
→Human-review every patch before merge; treat models as draft authors.

Conclusion

Open-weight coding assistants can turn session replay exports into plausible fixes—but only if the replay carries enough structured context and you grade the output like a PR, not a trivia score. DeepSeek Coder V2 edged the field on merge-ready patches; Qwen2.5-Coder led on tests. The smaller model reminded us that without rage-click and duplicate-request markers, models wander. Build the export format once, reuse it across models, and keep humans in the review path. That is replay-driven debugging for the LLM era: evidence first, patches second, dashboards never.