Simulator and backtesting

Langbot records the order books of the markets it follows while events happen, alongside every event it emitted. Langbot Sim replays the events and the books at their original timestamps against a strategy, with the venue's fee formulas.

Run one

langbot login                                   # Sim needs a token
langbot sim events                              # replayable events: hits, book rows, settled markets
langbot sim strategies                          # the built-ins and their parameters
langbot sim run -e <event_id> -s buy_yes_on_first_counting_hit
langbot sim run -e <event_id> -s my_bot.py --timeline ev.ndjson   # your file runs locally; the timeline is cached
langbot sim study -e <event_id> --words tariff,china                # how the YES mid moved after the words

Built-ins can also run on the feed (--server, or sim_run over MCP); a strategy file of yours only ever runs on your machine. Server runs and studies stream the event's timeline through the engine page by page, so any recorded event runs there, the 1 Nov rally (~426k book snapshots) included; only an event over 5 million snapshots is sent back to run locally, because it would take minutes. In code, langbot.sim.ScorecardRun does the same with your own pages and returns the same scorecard as run_scorecard on the whole timeline.

Across the whole record

One event is a sample of one. The research tools run an idea across everything recorded and put a sample size, an interval and the caveats on every figure. They measure; they never signal.

langbot research bench -s quote_both_sides --latency 250,2250,5250        # every recorded event, locally
langbot research bench -s my_bot.py --speaker powell --since 2026-10-01   # your file, a subset
langbot research bench -s buy_yes_on_first_counting_hit --server          # built-ins, on the feed
langbot research calibration --speaker powell --basis mid                 # how openings were priced vs settled
langbot research calibration --series KXFEDMENTION --basis trade          # the first trade (read the caveat)

Bench. One row per event (net after fees, fills, drawdown at each latency), then across events: total and mean net, the median, a bootstrap 90% interval of the mean (the event is the resampling unit), the share of events that ended positive, a sign test and a bootstrap p-value with Benjamini-Hochberg control across the latency rows, the same fills re-priced at no fee and at double fees, and an event-level walk-forward split (the earlier 70% of events against the later 30%). On the feed each event replayed is one Sim run; past the allowance the rest are listed as skipped. Every bench and every calibration query is written to your trials ledger (GET /v1/research/trials), and each result says how many configurations you have tried in 30 days: every one is another comparison.

Calibration. For every settled mention market the filters keep (speaker, series, category, phrase, venue, dates, volume): by opening-price bucket, how many markets, their average price, how often they settled Yes with a 90% interval, the gap between the two in cents, the Brier score and a binomial p-value; the same by series and by speaker; a cluster bootstrap by venue event; and the first 70% of markets by settlement time against the last 30%. The default price is the first two-sided mid; the first trade (--basis trade) reads as YES overpriced on the record because early buyers lift wide asks, a selection artifact the caveats name.

Fill-model settings on every run and bench: --queue-position makes a resting order wait behind the size recorded at its price (it fills on a touch only once that size has gone; a move through still fills it whole), --taker-delay MS adds a venue speed bump to taking orders only, --maker-fee RATE sets the maker fee (0.0175 by default; some series charge makers nothing). The built-in quote_both_sides is the market-making baseline these settings exist for: a bid and an ask 3 c around the Yes mid, re-quoted when the mid moves, inventory capped, flat before the end.

langbot backtest strategy.py --fixture file.jsonl replays a local JSONL file (events and books) through the same strategy file with the paper broker. --events ev_a,ev_b pulls phrase hits from GET /v1/events/recent (up to 1,000 per event) without books, so no order can fill: the run exits with an error saying 0 order books were replayed. Use langbot sim run for book-accurate replays; it fails the same way on a timeline with no books.

The scorecard lists fills, P&L per market and a latency table: the same strategy at each --latency (default 250,2250,5250 ms after t_emitted), so you can see how much of a result depends on speed.

What is simulated honestly

• Your latency. Fills are evaluated at t_emitted plus each --latency value. The book used is the snapshot at that time, not at t_spoken.
• Fills. An IOC bid fills against resting asks at or below your price, consuming size level by level from the snapshot. A GTC bid rests and fills when a later snapshot shows the ask crossing it. With --queue-position a resting order also waits behind the size recorded at its price; without it, a touch fills it.
• Venue delay. Add Kalshi's live-event order delay to --latency to stress the result (Kalshi's rules allow it; see Data and use).
• Fees. The venue's own formula: Kalshi's taker and maker fees by series, Polymarket's per-market rate.
• Settlement. Per the archived market resolution.

What is not simulated

• Market impact beyond consuming the visible book.
• Other traders reacting to the same word between two snapshots.
• Resting orders cancelled between two snapshots.

Treat backtest P&L as an upper bound on what the strategy could have made, not a forecast.

Data availability by plan

Every plan gets the full record and Langbot Sim on any recorded event; plans differ only in how much you run:

A run is one server run or one client-side replay (langbot sim run, which reads the timeline once). See Pricing.