Storage

The store holds what a session traded as rows a query can answer: fills, orders and their terminal state, position snapshots, per-day and per-session PnL, kill events and session metadata. It survives a restart and is read without replaying a journal. Code: include/fastmm/store/, src/store/.

Two records, two jobs:

	Journal (.fmj)	Store
Holds	every event the engine consumed and every message it sent, byte for byte	the interpreted result: fills, orders, positions, PnL, kill events
Answers	"does this session reproduce exactly?" (fastmm-replay)	"what did I trade yesterday?" (fastmm-pnl, fastmm.open_store)
Written by	JournalFileWriter on the fm-journal thread	a storage backend on the fm-store thread
Loss policy	a full ring trips the kill switch: replay needs every event	a full ring drops the record and counts it
Authority	yes	no: every row is derivable from the journal, though no tool rebuilds one

Prices, quantities and PnL are raw fixed-point int64 (1e-8) in the columns whose name ends in _raw; timestamps are int64 nanoseconds since the Unix epoch in the columns whose name ends in _ns. day is the UTC day of the row's timestamp, denormalised so a day query needs no date arithmetic.

Configuration

[storage] is free-form, like [sim] and [backtest]: the central schema (include/fastmm/config/schema.hpp) knows no key in it, and each backend reads its own.

[storage]
backend = "sqlite"        # "none" disables the store
path = "runs/mm1.db"      # sqlite: default "<[engine] journal_dir>/<[engine] name>.db"
ring_bytes = 4194304      # engine-to-store ring, a power of two (default 4194304)

Key	Backend	Meaning
backend	all	backend name, or none (default sqlite)
ring_bytes	all	bytes of the ring the engine writes records into, a power of two (default 4194304)
path	sqlite	store file (default <journal_dir>/<engine name>.db)

backend = "none" allocates no ring and starts no thread, and the engine's RecordWriter is disabled. A backend that cannot be opened stops the session before it trades, with exit code 3.

How a record reaches the store

engine thread --RecordWriter--> MsgRing --fm-store thread--> Backend --> rows

The engine copies a trivially copyable record into an SPSC ring, as it writes the journal: no allocation, no syscall, no wait. StoreThread drains the ring, batches up to 512 records into one backend transaction, and sleeps 2 ms when the ring is empty. Records are include/fastmm/core/record_stream.hpp:

Record	Written when	Carries
FillRecord (256 B)	every execution, and every quantity the engine books from a cum_qty jump	price, quantity, booked quantity, fee and fee asset, liquidity, venue order id, exec id, the venue's time of the trade (RecordHeader::exch_ts), and the position it left behind
OrderRecord (192 B)	every OMS state change	the whole Order: state, previous state, price, quantity, filled quantity, reject reason
PositionRecord (192 B)	after every fill, after a venue position snapshot, and once per instrument at finish()	the instrument's position and the portfolio totals
FundingRecord (192 B)	every funding payment the engine books	amount, asset, venue funding id, the venue's time, and the instrument's position, realized PnL and funding after it
ReplayedRecord (64 B)	a reconciliation of a venue begins with the venue's executions complete (its execution replay ended whole)	the venue; every execution it made until then is in the records before this one
KillRecord (128 B)	every kill switch trip, global or per venue	the reason, the flag word and the PnL at the time
RejectRecord (256 B)	every order the pre-trade checks or its venue refused, repeats folded (below)	the account, instrument, side, whether it only reduced the position, the reason, the limit or side that refused it (RejectSource) and the account's budget then: the [risk] bucket's tokens and each order window's count against what the connector admits
ParamRecord (448 B)	every strategy parameter update the engine applies	the update's target instrument, origin (control socket or the strategy's own publisher), source, publisher sequence number and its (field, raw value) pairs, as the ParamUpdate carried them (Journal format)

Session metadata (session_open, session_close, the instrument table) is written by the control thread, not through the ring.

What a backend guarantees

• Ordering: Records arrive on one thread in the order the engine made them; seq is strictly increasing within a session.
• At most once: A record the ring dropped, or a batch the process died inside of, is gone. A backend must not invent one. Records carry keys (session id plus seq, and a venue exec id on a fill) so re-ingesting the same record is a no-op.
• No back pressure: The store never signals the engine. A ring that fills drops records on the engine side and counts them in EngineStats::records_dropped; the count is logged at shutdown and stored in sessions.records_dropped. A backend that fails counts the error, logs the first one and keeps going. Neither stops trading: the journal is the authority for anything lost.
• No exceptions across the interface and no abort().

fastmm-live logs the totals when a session ends:

store: 1284 record(s) in 37 batch(es), 1284 row(s), 0 error(s), 0 dropped

Durability

The SQLite backend opens the file with journal_mode=WAL and synchronous=NORMAL, and checkpoints every 256 pages.

• A committed batch survives the process dying (a crash, SIGKILL, an abort()): the write-ahead log is in the page cache and the next open replays it.
• A batch the process died inside of is rolled back whole: no partial batch is ever visible.
• Power loss can cost the batches written since the last checkpoint. synchronous=FULL would fsync the log on every commit; the journal already holds every record, so the store does not pay for it.
• A reader never blocks the writer and never sees a half-written batch: fastmm-pnl and a notebook can query a store while the session writes it.

Schema

Version 7 (kSqliteSchemaVersion). The SQL is src/store/sqlite_schema.cpp, one migration step per version; an existing store is migrated in place at open, and a store written by a newer FastMM is refused with the version it holds.

sessions

One row per session, written at start and completed at shutdown.

Column	Meaning
session_id	the engine's session id (primary key)
engine, strategy	[engine] name and the strategy that ran
session_epoch	client order id epoch of this session
started_ns, started_day, stopped_ns	start, its UTC day, and stop; stopped_ns is null when the process did not shut down
version, build	FASTMM_VERSION_STRING and the build info
config_hash, config_toml	the effective configuration (secrets omitted) and its hash, as in the journal header
host, pid, dry_run, pnl_carry_raw	where it ran, and the net PnL carried in from earlier sessions
clean_shutdown	1 when session_close ran; 0 means the process was killed
exit_code, kill_reason, kill_latched	how it ended; a clean shutdown asks for the kill switch itself, so kill_reason is then Requested
journal_complete, journal_bytes	whether the journal was closed cleanly, and its size over every part
events, orders_sent, cancels_sent, replaces_sent, fills, risk_rejects, venue_rejects	the runner's counters
records_dropped	store records the ring could not take: the rows below are incomplete by that many
realized_raw, unrealized_raw, fees_raw	the session's final PnL
funding_raw	the part of realized_raw that is funding; null for sessions from before version 4

session_journals

(session_id, part, path): the journal parts the session wrote, in order.

session_venues

(session_id, venue_id, name, replayed_seq): the [venues.<name>] behind each venue_id of the session, so a restart finds its venues by name. Version 3; sessions recorded before it have none. replayed_seq (version 6) is the seq of the last ReplayedRecord of the venue: 0 while the session has not got past an execution replay there, null for a session recorded before version 6.

instruments

(session_id, instrument_id) and the instrument as the session loaded it: venue_id, symbol, base, quote, settlement_ccy, asset_class, inverse, tick_raw, lot_raw, multiplier_raw. Per session, because a restart may load different reference data.

fills

One row per execution, keyed (session_id, seq), with a unique index on (session_id, instrument_id, side, exec_id) for the rows that have one: an execution is the venue's id of it on one instrument and side, as the OMS keys it. Before version 5 the index was (session_id, exec_id), and a second venue or symbol using the same number, or the other half of a self-trade, lost its row; the migration replaces the index and cannot bring those rows back (the journal has them).

Column	Meaning
ts_ns, day, symbol, venue_id, instrument_id	when (the engine's clock), what and where
exch_ns	the venue's time of the trade, in the venue's clock; 0 for a synthetic fill, a connector that reports none, and rows from before version 3
cl_ord_id, venue_order_id, exec_id	the ids the venue and FastMM know it by
side, liquidity	Buy/Sell, Maker/Taker/Unknown
price_raw, qty_raw, booked_qty_raw	the execution, and the quantity the position moved by (a base-asset fee is deducted)
cum_qty_raw, leaves_qty_raw	the order after it
fee_raw, fee_amount_raw, fee_asset	the fee booked in the settlement currency, the amount the venue reported, and its asset (quote, base, other; an other fee is reported but not booked)
position_qty_raw, position_avg_px_raw, position_realized_raw, position_fees_raw	the instrument's position after this fill
synthetic	1 for a quantity the engine booked from a cum_qty jump, at the order's own price and with no fee
late	1 for a fill that arrived after the order was already terminal

funding

One row per perpetual funding payment the engine booked, keyed (session_id, seq), with a unique index on (session_id, instrument_id, funding_id). Version 4.

Column	Meaning
ts_ns, day, symbol, venue_id, instrument_id	when (the engine's clock), on what, where
exch_ns	the venue's time of the payment
funding_id	the venue's id of it (Binance tranId, Bybit execId)
amount_raw, asset	the payment in the settlement currency: negative paid, positive received
position_qty_raw, position_realized_raw, position_funding_raw	the position it was paid on, and the instrument's realized PnL and funding after it
replayed	1 when it came from the venue's history rather than its private stream

Funding is realized PnL (not a fee), so it is inside every realized_raw below; the funding_raw columns say how much of it.

orders

One row per order, keyed (session_id, cl_ord_id), updated in place: the row holds the last state the session saw and updates counts how many changes it went through. terminal is 1 for Filled, Canceled, Rejected, Expired and Replaced; the open_orders view selects the rest.

Replaced is the store's own state, not an OrderState: a cancel-replace to a new client order id renames the order in place, so the id it superseded never reports a terminal state of its own. Its row keeps the price and quantity it had and is closed when the replacement is acked. Without it every replaced quote would read as still open.

A fill that arrives after its order ended (the cancel ack overtook the execution) updates the row's cum_qty and, when it completes the order, its state: Canceled becomes Filled.

positions

One row per position snapshot, keyed (session_id, seq): qty_raw, avg_px_raw, realized_raw, unrealized_raw, fees_raw, gross_traded_raw, fills, the portfolio totals total_realized_raw, total_unrealized_raw, total_fees_raw, pnl_carry_raw, and (version 4) funding_raw and total_funding_raw.

kill_events

One row per trip, keyed (session_id, seq): scope (global or venue), venue_id (-1 for a global trip), reason (KillReason), kill_flags and the PnL at the time.

pnl_daily

One row per (session_id, day, instrument_id), maintained as position records arrive.

Column	Meaning
realized_raw, fees_raw, gross_traded_raw, fills, funding_raw	the change within that UTC day, not a running total (funding_raw, version 4, is part of realized_raw)
unrealized_raw, qty_raw	the last snapshot of the day: a mark, not a flow
symbol, settlement_ccy	denormalised from instruments

A change that straddles midnight lands on the day of the snapshot that reports it. Realised PnL and fees of instruments that settle in different currencies must not be added: group by settlement_ccy (the risk model says why).

param_schema, param_updates, param_values

Version 8. The strategy parameters over time: what each session started with and every update the engine applied, so the values in effect at any moment can be read back without the journal.

Table	Rows
param_schema	one per (session_id, field): the session's parameter table, name and type (int, double, bool, decimal, bps, ms), by the field index a ParamRecord names
param_updates	one per (session_id, seq): seq 0 is the set the session started with (origin initial, complete 1 when it holds every parameter, defaults included; 0 when the strategy brings its own publisher and only the configured ones are known), then one per update with the record's seq, ts_ns (when the engine applied it), instrument_id and symbol (NULL for every instrument), origin (control or strategy), source (fastmm-ctl param --source), publish_seq, and how many fields it carried and how many changed
param_values	one per (session_id, seq, name): the value as the parameter formats it, its raw value (NULL when not known) and changed against the value before it in that scope

A control update is stored whole, unchanged values included: someone asked for it. An update of the strategy's own publisher keeps only the values that changed, and one that changed nothing writes no row: sessions.param_refreshes counts those. An update to every instrument replaces what single instruments had.

rejects

Version 9. One row per refused order the engine recorded, keyed (session_id, seq): ts_ns, account (the [venues.<name>] the order went to or was routed to), instrument_id, symbol, side, kind (new or replace), reduces (with the open orders on its side it only took the position towards zero), reduce_only, reason (RejectReason), source, count, price_raw, qty_raw, cl_ord_id (a venue's refusal), venue_code and text (the venue's), and the budget the engine saw: budget_known, paused, tokens (the [risk] bucket's, NULL with the limit off), token_wait_us, and orders_10s_used/orders_10s_admits, orders_1m_*, orders_1d_*, weight_* (0 admits: the window is not known).

source	Refused by
risk	a pre-trade check other than the rate limits (reason says which)
risk_bucket	the [risk] orders_per_sec bucket, shared by every account
account_paused	the account's connector is paused (429, 418)
account_orders_10s, account_orders_1m, account_orders_1d	the account's order window at the connector's cap, as the engine counts it (account pools)
venue_local	the connector's own limiter, before sending (text names the window: local rate limit: orders 10s)
venue	the venue (venue_code, text)

Refusals of the same instrument, side, account, reason, source and flags (reduces, reduce_only) within 100 ms of the last row recorded for them are counted, not written one by one: count is 1 plus those folded into the row, so SUM(count) is exact per account. A count no further refusal follows is written with the next once-a-second publication, or at the end of the session. The budget of a folded row is the one at the row's time.

Views

View	Rows
reject_summary	day, account, symbol, side, reduces, reason, source and rejects, the summed count
param_history	param_updates joined with param_values: one row per value of every update, with its session, time, scope, origin and source
pnl_by_day	day, symbol, settlement_ccy, realized_raw, funding_raw, fees_raw, net_raw, gross_traded_raw, fills, summed over sessions
pnl_by_currency	the same by day and settlement_ccy
open_orders	orders with terminal = 0

Writing a backend

A backend implements fastmm::store::Backend (include/fastmm/store/backend.hpp) and, to be queryable, fastmm::store::Reader (include/fastmm/store/reader.hpp), then registers two factories before a session starts:

fastmm::store::StoreRegistry::instance().add("clickhouse", &make_ch_backend, &make_ch_reader);

Nothing registers itself: the linker drops a static library's self-registering object. fastmm::store::register_builtin_backends() adds the ones FastMM ships; an out-of-tree backend calls add() from its own main before run_live. The name none is reserved.

Backend::open receives a BackendOptions holding the [storage] section verbatim, [engine] name and [engine] journal_dir: a backend parses its own keys from GenericSection and nothing is added to the central schema. Everything the interface calls runs off the engine thread and may allocate.

The call order is open, session_open, instruments, then begin / records / commit repeatedly, then session_close and close. A Reader opens the same store read-only and answers sessions, fills, orders, pnl, funding, positions and recovery; every query takes the same QueryFilter and returns string rows.

Recovery at start-up

fastmm-live reads the store before the first session thread starts and logs what the previous session of the same [engine] name left behind: its PnL, whether it shut down cleanly, the kill state, the journal parts, the last position per instrument, and every order that was still open at its last record. fastmm-pnl recover --engine <name> prints the same thing.

With [engine] restore_position the session also carries the last position per venue and symbol over and books what happened while it was down from the venue's trade history (What survives a restart). The position is the last record the store holds for that venue and symbol: a position row or a fill (each fill row carries the position it left, so a crash that kept a fill but not the position row after it loses nothing), from the newest session of the engine that recorded one. A session that crashed before recording anything about a venue leaves the earlier session's record in force. Where each venue's replay starts (Recovery::venue_resume, passed to Venue::resume_executions, and in the attach request behind a gateway):

• In the venue's clock: From exch_ns of the venue's last stored fill or funding payment (in the newest session that stored one), less 1 s, skipping the trade ids and funding ids (as funding:<id>) stored from there on. The funding replay starts there too, so a payment made while nothing ran is booked by the next session and one the store holds is not booked again. Both ends are venue time, so the host's clock does not enter. The 1 s covers a venue publishing executions out of trade-time order (other symbols, a batch), which is milliseconds.
• The ids skipped are those of every session of the engine, not of the session the start comes from: a session killed seconds after it started stores only what its own replay booked, and the next replay's window still reaches what the sessions before it stored.
• The start is where the record is known to be whole: the last fill stored up to the newest session that got past its own replay on the venue (replayed_seq > 0; a session recorded before version 6 counts when it placed an order there, which none does before the venue has reconciled, or shut down cleanly). A session after that one died while its replay was still reading and can hold a fill newer than one the replay had not reached (an order left resting filled meanwhile): its fills are skipped as known, and the start is no later than the earliest of their last ones. When no session ever got past its replay, the start is no later than where the sessions began (the last bullet).
• At most 128 ids a venue: When more stored fills fall in that second, the start moves later, past the oldest millisecond that does not fit whole, and the session logs it: an id left out would be booked twice.
• Binance Spot and USDⓈ-M resume each symbol at the trade id after the highest one stored up to that same session (fromId, Venue::resume_trade_ids), with no overlap, and skip the ids above it that the sessions after it stored (Venue::resume_known_trade_ids). A venue no session got past its replay on has no such id and replays by time. In-process only: a gateway's venue is shared, and it filters another attachment's replay for this one by time and ids.
• A store from before version 3 starts from the engine clock as before: 10 s before the session's last fill, skipping the ids of the 20 s before it.
• A venue with no stored fill replays from the start of the newest session that shut down cleanly, else of the oldest session of the engine, less 10 s (engine clock): a fill of a session that died before storing it, or one made while nothing ran, is booked.

Until every venue that replays executions has finished its first reconciliation (the replay, then the open-order snapshot), the engine sends no order and quoting stays off: the positions are not complete before that, and an order sized, hedged or risk-checked on them could double a hedge or pass a limit. New orders are refused with NotReconciled; the log says every venue has reconciled its orders and executions since the start when it ends.

A store that holds a venue execution or funding payment in more than one session (by venue, symbol, side and the venue's id) was written by a restart that booked it twice. The check reads two indexes on the ids (fills_exec_any, funding_id_any, version 7) in id order and touches no table row unless an id repeats (0.14 s for a million fills; a store not yet migrated is grouped whole, 1.7 s). The start-up summary logs each one as an error (the first 32) with their count, and fastmm-pnl duplicates lists them; the rows are not changed. The positions and PnL stored by the sessions holding the copies count them twice, and a position restored from such a session is wrong until the venue reports its own.

The open orders the recovery lists are the ones FastMM last saw open; the venue may have cancelled, filled or expired them since. The position is FastMM's view at the last record; the venue's view arrives with the reconciliation.

The loss budget [risk] max_loss is carried across restarts by the kill-state file (include/fastmm/core/session_state.hpp), not by the store: it is read before any store is opened, so a session with backend = "none" still carries it.

Tools

• fastmm-pnl ledger reads fills as one row per execution across sessions (by engine, venue name, symbol, side and exec id; a synthetic fill is its own), with its copies and flags; fastmm-pnl order --order <id> joins orders, rejects and fills for one client order (Query what you traded). Both are queries over the tables above, not tables of their own.
• fastmm-pnl (command lines, the daily questions).
• fastmm.open_store(path) returns pandas DataFrames (Python API).
• sqlite3 runs/mm1.db for anything the two do not cover; the schema above is the contract.