Architecture

The engine thread holds the books, orders, positions and risk state. Venue threads send it messages over SPSC rings, and it journals each message it consumes. Live sessions, backtests and replay instantiate Engine<Strategy, Clock, Transport, Feed> with different clock, transport and feed types.

flowchart LR
  subgraph NET["net thread (per venue)"]
    WS["epoll · TLS · WebSocket"] --> P["simdjson parser<br/>book sync FSM"]
    ENC["order encoder"] --> OUT["HTTP / WS API"]
  end
  subgraph ENG["engine thread (pinned, busy-spin)"]
    B["L2/L3 book"] --> S["Strategy (CRTP)"] --> Q["QuoteManager"] --> R["Risk O(1)"] --> O["OMS"]
  end
  P -- "SPSC MsgRing" --> B
  O -- "SPSC MsgRing" --> ENC
  ENG -- "journal ring" --> J["journal thread<br/>.fmj + async log"]
  SIM["SimClock + SimTransport<br/>matching engine"] -. "same Engine template" .-> ENG

Threads

The live application (fastmm-live, src/live/session.cpp) runs a fixed set of threads for the whole session:

Thread	Owns	Waits by
fm-net-<i>, one per venue	net::Reactor: sockets, TLS, WebSocket and HTTP, JSON decoding, book sync, order encoding and signing, the rate limiter, the venue's order latency histograms	epoll_wait or io_uring_enter ([engine] net_backend), or busy
fm-engine	books, strategy, risk, OMS, quote manager, timers, positions, journal sequencing	busy spin, or a futex with spin_mode = adaptive
journal writer	JournalFileWriter: the journal ring into the .fmj file	spin, then waits of 50 µs to 1 ms
fm-store	StoreThread: fill, order, position, kill and funding records into store rows	2 ms sleeps when idle
log sink	formats every thread's log records and writes them	spin, then waits of 50 µs to 10 ms
main	config, reference data, thread start and stop, the control socket, the stats line, the status and kill files, TSC recalibration, signals	50 ms sleeps

With spin_mode = adaptive a network thread polls while active and for 200 µs after, then waits with a 1 ms timeout; the engine writes the wake eventfd only while it waits. The engine spins, then blocks on a futex that the network threads signal after they push events, at most until the next timer (1 ms). The journal writer runs when journaling is on, and fm-store unless [storage] backend = "none". The log sink is woken by flush() and stop().

The main thread posts Venue::on_timer and prints the stats line every second, writes the status file and the kill file every 250 ms, recalibrates the TSC every [engine] tsc_recalibrate_s, and handles SIGINT, SIGTERM and --duration: kill switch, then cancel_all on every venue over an independent REST connection.

All queues are single-producer/single-consumer (MsgRing, byte-oriented, variable-length 64-byte-aligned messages). N producers means N rings; the engine polls them round-robin, at most 64 messages per ring per visit, so one venue cannot starve another. The journal records events in the order the engine consumes them.

With fastmm-live --gateway <socket> there are no fm-net-<i> threads: the venues run in fastmm-gateway, and each venue's md, order-event and outbound rings are ShmRings in shared memory with the same protocol. The engine and its other threads are unchanged (Run behind a gateway).

Run-to-completion

[engine] threading = "single" (one venue; a configuration with more is refused) removes fm-net-0. The fm-engine thread, pinned to [engine] cpu, runs the venue's reactor iteration and Venue::poll() before every engine step (Engine::run_inline), so no event crosses a core between the packet read and the order write:

• Market data: the md sink's drain hook calls Engine::drain() after every commit. The event is journaled, applied and given to the strategy while the venue is still in the receive callback that decoded it; the rest of the datagram or frame is decoded afterwards.
• Orders: LiveTransport::set_direct passes the engine's batch to Venue::send_now, which encodes and writes it with one write, as the network thread's drain of the outbound ring does. T4 to T5 is then encoding plus the write system call, and the engine's tick-to-trade ends with the order on the socket.
• Order events, control messages, parameter rings and timers are taken by the step after each reactor iteration. An event the venue emits while the engine is running (an order it refuses) waits in its ring for that step: drain() does nothing when entered from inside the engine.

The rings stay, as same-thread FIFOs. The journal thread, the log sink and the main thread are unchanged, and the journal records events in the order the engine consumed them, so fastmm-replay works the same way. The costs: while the engine works nobody reads the sockets, and a slow strategy hook delays the venue's heartbeats and timers too. Measured with scripts/bench-e2e.sh: Benchmarks.

Network reactor

net::Reactor (include/fastmm/net/reactor.hpp) is the single-threaded event loop under every connection: descriptors registered with an IoHandler, a timer min-heap, and a mailbox (post() / wake() through an eventfd), the only part that other threads may call. One run_once() waits for I/O (at most until the next timer or max_wait_ms), dispatches it, then runs posted tasks and expired timers, without allocating. The connection, TLS, WebSocket and HTTP code only uses registration (add, modify, remove) and timers, so it runs unchanged on either backend. [engine] net_backend selects the backend for fastmm-live and fastmm-sim-exchange.

	epoll (default)	io_uring
registration	epoll_ctl, edge-triggered, EPOLLRDHUP	one multishot IORING_OP_POLL_ADD per fd with POLLRDHUP; POLLOUT only while the registration asks for Write
modify	EPOLL_CTL_MOD	IORING_POLL_UPDATE_EVENTS on the live poll request
remove	EPOLL_CTL_DEL	IORING_OP_POLL_REMOVE, submitted immediately
wait	epoll_wait, timeout rounded up to whole ms	io_uring_enter with IORING_ENTER_EXT_ARG, timeout in ns
busy poll	epoll_wait with a zero timeout every iteration	reads the completion ring in user space; enters the kernel only to submit, or when the kernel sets IORING_SQ_TASKRUN or IORING_SQ_CQ_OVERFLOW

Both backends map events the same way: POLLERR calls on_error(SO_ERROR), POLLIN, POLLRDHUP or POLLHUP call on_readable(), POLLOUT calls on_writable(). The handler is looked up again for every event, so a handler that an earlier callback in the same batch removed is not called, and one it replaced receives the event instead.

The io_uring backend uses liburing 2.15, linked statically into fastmm_net; SQE preparation, submission and completion reaping are liburing's inline helpers.

• Ring: io_uring_queue_init_params with 512 SQEs, 4096 CQEs, and IORING_SETUP_SUBMIT_ALL, COOP_TASKRUN and TASKRUN_FLAG on 5.19 and newer. EXT_ARG and NODROP are required. Registrations are queued and go out with the next wait, except remove(), which submits at once.
• Support probe: Reactor::io_uring_supported() creates a small ring once and checks that a multishot poll on an eventfd can be updated and then reports IORING_CQE_F_MORE. It is false on ENOSYS, EPERM (kernel.io_uring_disabled, seccomp), ENOMEM and kernels older than 5.13; fastmm-live and the simulator then log a warning and use epoll.
• Stale completions: user_data packs the operation (4 bits), a registration generation (28 bits) and the fd (32 bits). add() bumps the generation, so completions still queued for a removed registration, or for an earlier file that had the same fd number, are dropped.
• Re-arming: A multishot poll that ends without IORING_CQE_F_MORE (CQ overflow, a racing update) is re-armed while its registration exists. An update or remove that races with a completing poll (-EALREADY) is retried. A poll the kernel refuses (for example -EBADF) is reported once through on_error(errno).
• Descriptor lifetime: A poll request holds a reference to its file, so a socket closed while still registered stays open (no FIN, port still bound) until the request is cancelled. Always remove() before close(), as the net classes do. If a socket number is reused while the old registration is still there, add() sees a different inode and cancels the stale request. eventfd and timerfd descriptors share one anonymous inode, so this check cannot tell two of them apart.
• Wake-ups: A multishot poll reports every wake-up of the socket's wait queue, so a handler can be called for readiness it already consumed; it costs one read or write that returns EAGAIN.

Backend latency

bench/bench_reactor.cpp measures 64-byte loopback TCP echo round trips with both backends; they are within measurement noise of each other (bench/README.md). Loopback TCP and the read and write system calls dominate, and both backends make those calls the same way; io_uring saves only the empty epoll_wait of an idle busy-polling loop. The threaded variants need two free cores and refuse to run with fewer in their affinity mask (FASTMM_BENCH_NEEDS_CORES, bench/bench_pin.hpp).

Hot-path rules

1. No heap allocation after warm_up(); enforced by tests/hotpath (global operator new counter).
2. No exceptions; hot functions return Result<T, E> and are noexcept.
3. No double in price/quantity arithmetic: Price/Qty/Notional are int64 with a global 1e-8 scale; products go through __int128.
4. No virtual calls inside Engine::run(): strategies, clocks, transports and feeds are template parameters (CRTP/concepts); virtual dispatch is only used on the control path. Run-to-completion adds one indirect call per market-data event (the drain hook into IEngineRunner::drain()) and one per outbound batch (Venue::send_now).
5. Every cross-thread message is trivially copyable with an explicit size.

Determinism

Clock and Transport are compile-time policies. Live uses TscClock (rdtsc anchored to CLOCK_REALTIME, see below) and LiveTransport (writes into the venue's outbound ring). Backtests use SimClock (virtual time driven by event timestamps) and SimTransport (in-process matching engine plus a seeded latency model). Timers and RNG are virtualised the same way. The fastmm-replay tool feeds a recorded journal through the engine and verifies that the outbound stream is byte-identical. See Determinism and Event flow.

Strategy registration

Strategies are built through the StrategyRegistry, which keeps one factory per transport kind (Sim, Replay, Live). A strategy library registers its strategies with one function that calls register_strategy<S>(r) per strategy and so adds all three factories; the built-in strategies are the module fastmm::register_builtin_strategies in fastmm::strategies. The command lines (fastmm::cli::live, backtest, replay) register the built-in strategies and then the modules the app passes, explicitly at startup; run_backtest(cfg, name), replay_journal, sweeps and the Python module register the built-in strategies through bt::register_builtin_strategies(). The factory templates are declared in strategies/module.hpp and defined in strategies/factory_{sim,replay,live}.hpp, so a registered factory that no file compiles is a link error. See Register a strategy.

Clock calibration

TscClock maps a TSC reading to wall-clock ns with a 32.32 fixed-point rate and an anchor (TscCalibration). calibrate_tsc() takes the anchor from CLOCK_REALTIME and measures the rate against CLOCK_MONOTONIC_RAW over a 50 ms spin, so a wall-clock step inside the window does not change the rate. A 50 ms measurement drifts over a long session, which the recalibration below corrects.

• After startup TscCalibrator takes a fresh anchor (a tight rdtsc bracket around CLOCK_REALTIME and CLOCK_MONOTONIC_RAW) and computes the rate over the whole interval since the previous anchor, against CLOCK_MONOTONIC_RAW. Tens of microseconds of clock_gettime jitter then cost a few ppm instead of thousands, and NTP slews or host clock steps cannot distort the rate.
• The main thread recalibrates every [engine] tsc_recalibrate_s seconds (default 10, 0 turns it off), logs how far the previous calibration had drifted (tsc recalibrated: drift <ns> over <s> (<ppm>) ...) and publishes the result in a Seqlocked<TscCalibration>.
• The main thread writes no other thread's clock. Every user owns a copy: the engine's TscClock is attached to the seqlock (attach_calibration_source) and the engine calls TscClock::refresh() once per loop iteration next to the timer poll (not per event). refresh() is one atomic load when nothing was published; on a new version it copies the calibration. ControlCommand::RecalibrateTsc makes the engine refresh immediately.
• Re-anchoring keeps time continuous: at the refresh point the clock computes the old mapping's time for the current TSC reading and anchors there, so successive now() calls never go backwards. From there it runs at the newly measured rate plus a bounded correction that absorbs the measured offset over one recalibration period (offset × rate ÷ period, at most 500 ppm), so the mapping converges on the measurements instead of accumulating error. If the old mapping disagrees with the fresh measurement by more than 1 ms, the clock steps to the measured anchor instead, counts the step (EngineStats::clock_steps) and the engine logs a warning.
• The calibration follows CLOCK_REALTIME. If the host steps its wall clock (NTP corrections, or WSL2 resynchronising with Windows, which can jump by hundreds of milliseconds), the next recalibration steps the engine clock with it.
• The venues read the latest calibration when they publish their status, to convert their cycle-based latency histograms to ns.
• Without an invariant TSC (/proc/cpuinfo lacks constant_tsc or nonstop_tsc) wall time comes from clock_gettime and there is no recalibration. With constant_tsc alone, as on many KVM cloud instances, the calibration still carries the measured rate (use_tsc = false, has_rate()): TscClock::cycles() stays on the TSC and latency intervals are converted with that rate. Without constant_tsc the intervals are clock_gettime nanoseconds.

Latency instrumentation

Stamps are rdtscp readings (Cycles); intervals go into allocation-free log-linear histograms (LogLinearHistogram, 40 x 16 buckets).

Stamp	Thread	Where
T0	network	the frame was received (carried as EventHeader::t0_cycles)
T1	network	decoded into an engine message (t1_delta)
T2	engine	book (or position/OMS) updated
T3	engine	strategy hook returned
T4, T5	engine	around transport.send() of the outbound batch; live, that is the push into the venue's outbound ring, or with threading = "single" the encoding and write

The engine's LatencyTracker (decode, book apply, strategy, serialize = T3 to T4, send = T4 to T5, tick-to-trade = T0 to T5, wire-to-book = T0 to T2) is published through a seqlock every latency_publish_ms. The engine copies the triggering event's t0_cycles into every outbound message header, and the network thread measures the rest in venues::WireLatencyRecorder: for each outbound order message it stamps before encoding, after encoding and signing, and after the WebSocket write (or REST request call) returned, and records

• encode: JSON encoding and signing,
• send: the WebSocket/REST send call,
• wire tick-to-trade: after-send minus t0_cycles, for orders triggered by an inbound event (t0_cycles != 0; orders from timers are not counted).

These histograms stay in cycles on the network thread and are converted to ns with the published calibration when the venue publishes its VenueStatus (once per stats interval). fastmm-live prints the wire tick-to-trade p50/p99 and the encode/send p50 per venue in its stats line and in the final summary.

Failure handling

Failure	Detection	Action
Market-data disconnect	EPOLLRDHUP, read of 0 bytes, TLS error	Quotes pulled, reconnect with backoff, fresh snapshot
Sequence gap	Book sync state machine	Resync state, re-snapshot (rate limited), quotes pulled meanwhile
Stale feed	No traffic for the connector's stale_ms	Stale event, quotes pulled; dead_ms without traffic forces a reconnect (Venue connectors)
Order channel loss	Connection state machine	Cancel-all through REST (cancel_on_order_channel_loss); after reconnect, replay the executions since the watermark, then reconcile open orders
Kill switch	max_loss, a full outbound or journal ring, every venue killed, a strategy error, the client order ids used up, SIGINT/SIGTERM, --duration, an order-event ring overflow, a journal write failure	Stop quoting and cancel working orders (cancels stay allowed); at shutdown each venue also cancels all over REST (5 s timeout per request). See Risk model and Kill switch and shutdown
Venue unusable	The venue's error map: Fatal (bad key, signature or permission, failed authentication) or HardStop (Binance HTTP 418 IP ban); a dead man's switch that could not be refreshed; a multicast feed that cannot rebuild its books	That venue's kill switch: its quotes are pulled, its orders cancelled and new orders to it refused. Every venue killed is a global kill
Rate limit	Response headers and error codes	Cool down until the limit resets; after HTTP 418 REST stays stopped until restart
Clock skew	Timestamp rejection codes	Re-measure the offset from the venue's time endpoint
Ring overflow	Push fails	Market data: drop the delta and resync. Order events are never dropped; an overflow trips the kill switch