Journals, replay and PnL¶

Journal files¶

With [engine] journal = true (the default), fastmm-live writes <journal_dir>/<engine name>-<session id>.fmj; --journal <path> chooses the file and --no-journal turns it off. The log names the file at startup (journal: <path>).

A journal holds the session header, the instrument table, the effective configuration (without API keys and secrets), each consumed event with the engine clock at which it was processed, and a copy of each order message the engine sent (marked out). Blocks carry CRC32C checksums, and a clean shutdown writes a trailer block. The current format is version 3 (Journal format, ADR 0010); the tools also read versions 1 and 2.

Size: one-symbol Binance Demo sessions wrote 23 MB to 32 MB per hour. [engine] journal_max_bytes rolls the file over into numbered parts and [engine] journal_retention_days deletes old ones at start-up; [engine] journal_sync chooses how far a write is pushed before the writer moves on, and a journal that cannot be written trips the kill switch (Journal format).

Next to the journal, [storage] writes the same session as rows: fills, orders, positions, PnL by day and kill events, queryable without a replay (Query what you traded). Where the two disagree, the journal is right.

The session epoch¶

[engine] epoch_file (default runs/session_epoch) stores a counter that goes up by one for every session. Client order ids are the epoch in the upper 32 bits and a sequence number in the lower 32 bits, so ids stay unique across restarts. Keep the file between sessions that trade on the same account. The startup log shows the value (epoch=22).

Read a journal¶

fastmm report runs/demo-1/session.fmj writes the session as one HTML page (equity, inventory, fees, order and reject counts) beside the journal (Run report). For the events themselves, tools/journal_dump.py needs only Python 3:

python3 tools/journal_dump.py runs/demo-1/session.fmj --type OrderFill --first 3

It prints the header, the instrument table, the first matching events and a count per event type. A fill:

#86      OrderFill         in  inst 0 venue 0 flags - exch_ts 1789356325085000000 recv_ts 1789356324721949775 venue_seq 0
          cl_ord_id=94489280513(epoch 22, seq 1) exec_id=300867841 px=77762.68 qty=0.0003 cum=0.0003 leaves=0 fee=0.0000003 fee_asset=base side=Buy liq=Maker

fee is in units of fee_asset: quote (for example USDT), base (for example BTC) or other (for example BNB, which the engine cannot value and leaves out of fees and positions).
liq is Maker, Taker or Unknown.
trailer MISSING at the end means the process did not shut down cleanly. The events before the damaged block are readable; fastmm-replay refuses such a journal unless given --allow-incomplete, and then its outbound comparison proves nothing.
--no-crc skips checksum verification, which is slow in pure Python on large journals.

Replay¶

A journal recorded by fastmm-live or by fastmm-backtest --journal-out replays on its own:

./build/release/bin/fastmm-replay --journal runs/sim-local-1789370000000000000.fmj --verify
./build/release/bin/fastmm-backtest --config configs/backtest-example.toml --data synthetic \
    --out - --journal-out runs/bt/session.fmj
./build/release/bin/fastmm-replay --journal runs/bt/session.fmj --verify

fastmm-replay runs the recorded inbound events through the same engine and strategy with a simulated clock set to the engine clock of the recording, and compares every outbound order message, and their SHA-256, with the copies in the journal. It takes the configuration embedded in the journal, and the session epoch, dry run, RNG seed and each venue's cancel-replace from the header, so it needs neither the config file nor API keys. A 20 s fastmm-live session against fastmm-sim-exchange:

journal  /tmp/e2e/session.fmj: format v3, 2644 messages (850 market data, 320 outbound), rng_seed 42, strategy 'basic_mm'
session  epoch 23, quoting enabled, cancel-replace venues 0x1, engine clock recorded
config   embedded in the journal (hash fb8ab9d4318a634e)
replay   strategy=basic_mm events=2300
recorded outbound 320 msgs sha256 a9463030a4344f26d386ee4bf5ec1f53060895e9b925f6bda62b69b76ef9cd6a
replayed outbound 320 msgs sha256 a9463030a4344f26d386ee4bf5ec1f53060895e9b925f6bda62b69b76ef9cd6a
replay MATCH

--config <file> replays with that file instead. Its effective configuration (secrets and formatting do not count) is hashed and compared with the journal's; if they differ, the run is a what-if replay and prints the warning below. --strategy <name> is a what-if run too.

A mismatch prints the first differing message as recorded and as replayed. The same session with half_spread_bps = 6.0 instead of 5.0:

fastmm-replay: warning: sim-local-whatif.toml is not the configuration this session was recorded with (effective config hash cc9c8e5451e9b57f, journal fb8ab9d4318a634e); this is a what-if replay and is not expected to match. Omit --config to replay with the recorded configuration.
...
first mismatching outbound message: #0
  expected: OutNewOrder venue=0 inst=0 cl_ord_id=98784247809 (epoch 23, seq 1) Buy PostOnly GTC px=59970 qty=0.001 recv_ts=1789376211743451228
  actual:   OutNewOrder venue=0 inst=0 cl_ord_id=98784247809 (epoch 23, seq 1) Buy PostOnly GTC px=59964 qty=0.001 recv_ts=1789376211743451228
replay MISMATCH

Only the first mismatch is meaningful: replay feeds the recorded acknowledgements whatever it sent, so everything after it diverges.

Journals written before format version 2 carry no configuration, session settings or engine clock. Replay them with --config <the config the session ran with>; a live session of that age does not replay to a match. A journal without outbound copies (market data only, such as tests/fixtures/journals/sample_1000.fmj) is backtested first, with configs/backtest-example.toml unless --config is given, and with --verify the run's outbound hash must match the <journal>.sha256 sidecar or --expect <sha256>:

./build/release/bin/fastmm-replay --journal tests/fixtures/journals/sample_1000.fmj --verify

Check before replaying¶

--check says whether a replay can run, and with what, without replaying: the journal (readable, closed by its writer), the configuration (embedded, given and matching the recording, or the market-data default), the strategy registered in this build, its parameters known to it, and for a market-data journal an expected hash.

./build/release/bin/fastmm-replay --journal tests/fixtures/journals/sample_1000.fmj --check

journal        ok   market-data journal, format v1, 1000 messages
complete       ok   closed by its writer
config         ok   configs/backtest-example.toml (default for a market-data journal)
strategy       ok   basic_mm
params         ok   8 set
expected_hash  ok   tests/fixtures/journals/sample_1000.sha256
ready to replay

A warn line runs but proves less (a what-if configuration, no engine clock, an unclosed journal with --allow-incomplete); a fail line stops the replay, and the exit code is 3. --json prints one object, {"ready": ..., "checks": [{"check", "status", "detail"}, ...]}, for a script.

Exit codes: Command lines.

A mismatch with the embedded configuration and the same binary is a determinism bug, and the journal reproduces it (Determinism). A different binary need not match.

What the fills were worth¶

./build/release/bin/fastmm-data analyze runs/mm1-1789370000000000000.fmj
./build/release/bin/fastmm-data analyze runs/mm1-1789370000000000000.fmj --json --horizons-ms 1000,10000,60000

Reads a live session's journal alone (no store, no configuration, nothing replayed) and marks each fill against the venue mid the session recorded (its BookTicker stream): what it captured against the mid at the fill, what it was still worth after each horizon (the markout) and the difference (adverse selection), in basis points of the traded notional, overall, by maker and taker, and by instrument; with the order traffic and the maker share beside them:

session  strategy 'basic_mm', 1 s of venue time
orders   2 sent, 1 cancels, 0 replaces, 0 refused by the venue
fills    1 (1 maker, 100.0%), 1 repeated report(s) counted once

instrument       fills   maker         bought           sold         notional
BTCUSDT              1       1              1              0           100.00

horizon  fills of         marked  capture_bps  markout_bps  adverse_bps  excluded
1s       all                  1         1.00        -1.00         2.00         0
10s      all                  0         0.00         0.00         0.00         1

An execution reported twice (the replay after a reconnect) is counted once. A fill whose horizon falls after the last mid, or that had no mid, is left out and counted under excluded, never marked at a stale price. A session without a BookTicker stream has the counts and no markouts. The definitions are those of a backtest's markouts (Backtesting), so the two compare directly.

Check the fill model against live fills¶

fastmm-data fill-check measures how well [backtest] fill_model = "l2_queue" predicts the passive fills of a live session. It does not re-run the strategy. It takes the orders the session had resting, each from its ack until its cancel ack, last fill or expiry in venue time, puts each one behind the quantity displayed at its price at the ack's venue time, and feeds the queue model the journal's book deltas, trades and book tickers in venue time order, the way a backtest does. A book ticker newer than the depth (by the venue's update id when both carry one, else by venue time) sets the queue at its touch: an order priced better than the touch has nothing ahead, an order at the touch joins behind the touch's quantity less ours and never has more ahead than it shows later. Trades printed after the depth update or the ticker take their quantity from its levels until the next update shows it. A replace follows the new order id. Orders that crossed the book at the ack, market, IOC and FOK orders are left out.

./build/release/bin/fastmm-data fill-check runs/demo-1/session.fmj \
    --csv runs/demo-1/fill-check.csv

The one-hour Binance Demo session quoting at the touch (Example):

orders   2602 sent, 2347 resting after the ack; left out: 255 rejected, 0 not acked, 0 ended before the ack, 0 market/IOC/FOK, 0 crossing the book at the ack
market   32598 book and trade messages, ...
times    venue (order times in ms); from receive time: 0 order events, 1 market messages; ties in the ack / end millisecond: 2 / 0; own orders in depth: yes
live     1509 filled, qty 0.4074

conservatism  filled   both live only model only neither model/live    model qty  |dt| p50
0.00             546    542       967          4     834      0.393    0.1602879     0.0ms
0.50             526    523       986          3     835      0.379      0.15439     0.0ms
1.00             522    519       990          3     835      0.376      0.15319     0.0ms

Each row is one queue_conservatism value (--conservatism 0,0.25,0.5 to choose others). both, live only, model only and neither count orders by whether they filled live and whether the model filled them. model/live is the ratio of filled quantities. |dt| p50 is the median gap between the first model fill and the first live fill, for orders that filled both ways. Pick the conservatism whose model/live is closest to 1 and whose live only and model only are smallest.

The model predicted 38-39% of the quantity this session filled, whatever the conservatism. Binance Demo fills passive orders ahead of the quantity it displays: the session's first buy, 0.0003 BTC at 77762.68, acked behind 1.65 BTC, filled after 0.009 BTC had traded at that price, with 1.43 BTC still displayed there. A Demo session cannot calibrate the model; use a session on the real market.

Times are venue times (exch_ts): a venue sends its execution report before the public trade that filled the order, so by receive time the trade would fall after the order's end. An event without a venue time uses its receive time; the times line counts them. Binance order times are whole milliseconds, so a trade in the ack's millisecond counts for the order (a tie, counted); one in the end's millisecond only when the order had a fill in that millisecond, and only up to that fill's trade id. A fill reported after the order's end (the API's cancel response can arrive before the execution report) is still a live fill. In a live session the depth feed shows our own orders; each level is stripped of what we had resting at its price at the update's venue time before the model sees it (the same stripping as journal:<file>,strip_own=1, Market-data sources). --csv writes one row per order: price, size, the quantity ahead at the ack, resting time, how it ended, and the live and model fills with their times. fastmm-data calibrate runs this over several sessions, fits the conservatism and the latencies, and compares a backtest of each session with it (Backtesting).

Which live fills the model misses¶

After the table, fill-check prints a diagnosis at the last conservatism listed. Each order's first live fill and first model fill get the mid move over --pre-ms (default 100) before it and markouts at --horizons-ms (default 100,1000,10000), all against the journal's book ticker mids, in bps of the order's price, positive when the mid moved the order's way:

  fills                     n    100ms       1s      10s  pre p10  pre p50  pre p90  through  missing  crossed   cancel
  both, live fill         ...
  both, model fill        ...
  live only               ...
  model only              ...

live only against both, live fill says whether the fills the model misses are worse ones. The columns after the markouts say how each fill came about:

Column	Meaning
`through`	live: a trade printed through the order's price in the fill's millisecond (a sweep took it); model: the fill came from such a print
`missing`	no trade at or through the order's price in the live fill's millisecond: the recorded trades do not show it
`crossed`	the opposite touch had reached the order's price by the fill
`cancel`	a cancel or replace of the order had gone out before the live fill: it lost the race

The line under the table counts the first live fills whose exec id is a trade id of the recorded tape (Binance spot's raw trade stream: all of them; an aggregated stream: none, and the fills are matched by price and millisecond instead). Then each feature splits the orders into buckets, with the orders filled live, by the model, their ratio and the live-only count: the queue ahead at the ack in order quantities, ticks behind the touch at the ack, whether the opposite touch reached the price while the order rested, whether a trade printed through it, how much printed at the price against the queue at the ack, resting time, and whether the order was cancelled or replaced. A bucket whose live/model stands far from the others locates what the model gets wrong. --csv adds these per order: the cancel's send time, ticks behind, when the touch reached the price and when a trade printed through it, the quantity printed at it, the live fill's print class, and the pre-fill moves and markouts of the live fill and of each conservatism's model fill.

--market other.fmj replays the market data of another journal recorded over the same period (a recorder on another host, a feed with more depth) instead of the session's own. Its instruments are matched to the session's by venue id and symbol; messages on others are counted. The session's own orders are still taken out of its depth.

Check PnL¶

A session's PnL has four views:

The engine's, from the summary line fastmm-live: realized_pnl=<r> unrealized_pnl=<u> fees=<f> ... (also the final fastmm-top frame). Net PnL is r + u - f, marked at the engine's last mid. On perpetuals, r includes funding.
The journal's, computed from the fills by tools/pnl_report.py.
The store's, from fastmm-pnl or fastmm.open_store(): the same fills, indexed by day and instrument across sessions (Query what you traded).
The account's, from balance snapshots taken before and after the session.

tools/pnl_report.py prints fills, maker share, volume, fees and inventory per hour, post-fill markouts, the journal's trading PnL and, when given, the engine's summary and the account reconciliation:

python3 tools/pnl_report.py runs/demo-1/session.fmj --engine-log runs/demo-1/engine.log \
  --start runs/demo-1/equity_start.json --end runs/demo-1/equity_end.json

The snapshots are JSON objects you produce from the venue's account and ticker endpoints right before and after the session; FastMM ships no tool for this. Balances are free plus locked:

{"utc": "2026-09-14T03:25:21Z", "btc": 0.004995, "usdt": 4614.81995, "mid": 77762.685, "open_orders": 0}

The balance keys are base and quote, or the lower-case asset names given by --base-asset and --quote-asset (default BTC and USDT). equity (or equity_usdt) is optional and computed as quote + base * mid when missing. Each execution is booked once, keyed like the engine's OMS (venue execution id, instrument, side): a reconnect replays executions the stream already delivered, and the journal records both. python3 tools/pnl_report.py --self-test checks the tool on a synthetic journal.

The markout table marks each fill against the mid --markout-horizons seconds later (default 1,10,60), taken from the journal's BookTicker events; a journal without top-of-book updates has no mid to mark against and the report says so. A fill whose horizon falls after the last quote in the journal is excluded, not marked at the last known mid. The spread capture beside it covers the same fills, so the difference is the adverse selection (Backtesting).

equity change = starting base balance * (end mid - start mid) + trading

The session reconciles when:

the unexplained balance change (account balance change minus the journal's inventory or cash change) is zero. A non-zero value means fills that are not in the journal (a second engine or manual trades on the account), deposits or withdrawals, or commission paid in another asset;
the journal's trading PnL equals the account's trading PnL to within rounding;
the engine's net PnL differs from the account's trading PnL only by the engine's inventory marked at its last mid instead of the end snapshot's mid.

Commission charged in the base asset is already inside the inventory: a buy receives qty - fee and a sell delivers qty + fee. Do not subtract it again.

Example: Binance Demo¶

Two one-hour basic_mm sessions on BTCUSDT in Binance Demo Mode, with the same risk limits and quote size (0.0003 BTC):

Session	`half_spread_bps`	Fills in the hour	Account equity change	Of which trading
Quoting 15 bps from the mid	15	0	+2.11 USDT	0.00 USDT
Quoting at the touch	0	1640 (all maker)	-33.36 USDT	-32.94 USDT

The second session's report:

Notional traded: 31,622.62 USDT; fees: 31.62 USDT, 10.00 bps (the 0.1% maker commission). Commission was charged in BTC on buys (0.00020358 BTC in total) and in USDT on sells (15.82 USDT).
Inventory change -0.00044358 BTC and cash change +1.5160 USDT, both after fees, equal to the account's balance changes.
Account: equity change -33.36 USDT = starting inventory revaluation -0.42 USDT + trading -32.94 USDT.
Engine: realised -1.32 USDT, unrealised +0.00 USDT, fees 31.62 USDT, net -32.94 USDT, 0.003 USDT from the account's trading PnL.
Post-only rejects (PostOnlyWouldCross): 260.