Errors and exit codes¶
Exit codes, reject reasons, kill reasons and venue error actions. Log messages with their causes and fixes are in Troubleshooting.
Exit codes¶
Each program defines its own exit codes. 1 from fastmm-live means the process crashed.
fastmm-live¶
"Orders cancelled" is the cancel-all the control thread runs over REST at shutdown; cancel_all FAILED turns any code into 5 (Kill switch and shutdown).
| Code | Meaning | Orders cancelled | Restart without a human |
|---|---|---|---|
| 0 | stopped by --duration, SIGINT, SIGTERM or fastmm-ctl stop with cancel_all ok; also after a kill with on_kill = "stay"; --help, --version, --list-strategies |
yes | yes |
| 2 | bad command line; an unset ${VAR} in [venues.*] |
nothing started | no: fix the invocation |
| 3 | the configuration, strategy or a parameter does not load, a [storage] backend is unknown or cannot be opened, or a venue refuses a setting of the account (Bybit linear in hedge mode) |
nothing started | no: fix the config or the account |
| 4 | a venue's reference data failed to load, or --gateway could not attach |
nothing started | retry once; a repeat means the venue or the network |
| 5 | cancel_all FAILED, the journal cannot be opened or written (a full filesystem trips the kill switch), an order-event ring overflowed, the gateway closed the attachment, or an uncaught error |
not certain | no: check the venue for open orders first |
| 6 | the engine tripped the kill switch itself, on_kill = "exit", cancel_all ok; or, at start, a max_loss trip is latched in [engine] kill_file (nothing started) |
yes | no: find the kill reason in the log; a latched trip needs --clear-kill |
| 7 | a Python strategy's slow tier failed, cancel_all ok (python -m fastmm run and fastmm.run_live; fastmm-live never returns it) |
yes | no: fix the slow method |
| 8 | another process holds the instance lock ([engine] instance_lock), or --takeover failed: the holder refused or did not answer the handoff, or kept the lock past handoff_timeout_ms |
nothing started | no: another session trades this engine (Hand over a running session) |
A restart after 5, 6 or 7 is a decision for a person; the systemd unit does not make it (Deploy). What a restarted session carries over: Run in production.
fastmm-gateway uses the same codes: 0, 2, 3 (also an invalid [gateway] limit or a refused account setting), 4 (reference data), 5 (cancel_all FAILED or an uncaught error), and 6 when its kill file holds a latched [gateway] max_loss trip at start.
The other programs¶
| Code | fastmm-backtest |
fastmm-replay |
fastmm-sim-exchange |
fastmm-sim-itch |
fastmm-top |
|---|---|---|---|---|---|
| 0 | ran | matched, or no --verify |
stopped | stopped | quit, or --once printed a frame |
| 1 | — | verification mismatch | — | — | — |
| 2 | bad command line | bad command line | bad command line | bad command line | bad command line |
| 3 | bad config, strategy or parameter | unreadable config or journal, unknown strategy | bad config | bad config, CPU pinning, or the summary file | --once with no status file, or one from another build |
| 4 | the market data cannot be read | — | cannot listen on a port | cannot open a socket | — |
| 5 | the run failed | — | — | — | — |
fastmm-data: 2 bad command line, 3 bad config, 4 the market data cannot be read, 5 the output cannot be written. fastmm-pnl: 2 bad command line or a store that cannot be opened, 3 recover found no session. fastmm-ctl: 1 the session answered error, 2 bad command line, 3 no session answered.
Reject reasons¶
RejectReason (include/fastmm/core/enums.hpp). A reject means the order was not sent, or the venue refused it; nothing else changed. The strategy sees the reason in the Result of ctx.send; the engine counts them per reason and shows them in fastmm-top and the shutdown summary.
Pre-trade checks (engine)¶
Checked in the order listed, which is not numeric order; the first failure decides. A limit of 0 or omitted turns its check off (Risk model). These rejects never reach the venue.
| # | Name | Setting | Clears when |
|---|---|---|---|
| 1 | KillSwitch |
— | fastmm-ctl unkill or SIGHUP (with on_kill = "stay"), or a restart |
| 2 | VenueKilled |
— | as KillSwitch |
| 3 | InstrumentDisabled |
[[instruments]] enabled |
the config changes, or the venue re-enables the symbol |
| 4 | InvalidTick |
tick |
the strategy rounds prices with inst.round_price |
| 5 | InvalidLot |
lot, min_qty, max_qty |
the strategy rounds quantities with inst.round_qty |
| 6 | BelowMinNotional |
min_notional |
the order is larger |
| 7 | StaleMarketData |
[risk] stale_md_ms |
the book updates (its age is measured on the engine clock from when the engine applied it) |
| 20 | FeedLag |
[risk] max_feed_lag_ms |
100 ms pass without a market-data message of the venue later than its baseline by more than the limit; IOC, FOK and market orders and orders that only reduce the position are not refused (Feed-lag gate) |
| 8 | PriceCollar |
[risk] price_collar_bps |
the mid moves to the price, or the price moves to the mid |
| 9 | FatFinger |
[risk] fat_finger_bps |
a trade prints nearer the price |
| 10 | MaxOrderQty |
[risk] max_order_qty |
the order is smaller |
| 11 | MaxOrderNotional |
[risk] max_order_notional |
the order is smaller |
| 12 | MaxPosition |
[risk] max_position |
the position or the same-side open orders shrink |
| 19 | FxRateUnknown |
[accounting] |
the source of the order's settlement currency has a valid book that updated within [accounting] stale_fx_ms; an order that reduces its instrument's position is not refused |
| 17 | MaxGrossNotional |
[risk] max_gross_notional |
the portfolio's |position| shrinks, or the order reduces its instrument's position |
| 18 | MaxNetNotional |
[risk] max_net_notional |
the signed sum moves back, or the order reduces its instrument's position |
| 22 | UnderlyingMarkUnknown |
[risk.underlying] |
every inverse contract of the underlying with a position or open orders (and the order's own, if inverse) has a valid book no older than stale_md_ms |
| 21 | MaxUnderlyingNet |
[risk.underlying.<BASE>] max_net |
the underlying's position or same-side open orders shrink, or the order brings the net towards zero |
| 23 | BalanceShort |
[risk] check_balance |
the venue reports more free balance, or our orders on that asset end or fill (Balance check) |
| 13 | MaxOpenOrders |
[risk] max_open_orders |
an order of that instrument terminates |
| 14 | SelfTradePrevention |
[risk] stp |
our resting order on the other side moves or is cancelled |
| 15 | RateLimit |
[risk] orders_per_sec, burst |
the token bucket refills |
| 16 | MaxLoss |
— | never produced: a max-loss breach arrives as KillSwitch with KillReason::MaxLoss |
Cancels skip every check, including the kill switch.
OMS and transport¶
| Value | Name | Cause |
|---|---|---|
| 32 | PoolExhausted |
the order pool or the id map is full: more than 4096 working orders (kMaxOpenOrders, include/fastmm/core/oms.hpp) |
| 33 | UnknownOrder |
a cancel or replace for an id the OMS does not hold |
| 34 | InvalidState |
a cancel or replace for an order that is no longer working |
| 35 | DuplicateId |
the client order id is already in use, locally or at the venue |
| 36 | TransportFull |
the connector's outbound or REST queue was full; the message was dropped |
| 37 | NotReconciled |
never produced by the current code |
| 38 | InvalidTag |
ctx.send used a user_tag inside the quote manager's reserved range |
| 39 | OrderTableFull |
the connector's table of the orders it has sent (7168 working orders and replaces in flight, over every strategy behind a gateway) was full; the order was not sent |
| 40 | InvalidAccount |
NewOrderRequest::account named a venue that is neither the instrument's nor a member of its pool ([venues.<x>] pool_of); refused locally |
Gateway¶
A strategy attached to fastmm-gateway gets these back from the gateway's account guards (Run behind a gateway); the order never reached the connector.
| Value | Name | Cause |
|---|---|---|
| 48 | GatewayRateLimit |
[gateway] orders_per_sec, burst: the venue's order rate, shared by every attached strategy |
| 49 | GatewayOpenNotional |
[gateway] max_open_notional: the notional working at the venue, over every attached strategy |
| 50 | GatewayNotOwner |
the order's instrument is not one this strategy claimed when it attached |
| 51 | GatewayAccountKilled |
[gateway] max_loss tripped the account's kill switch |
| 52 | GatewayGrossNotional |
[gateway] max_gross_notional: the account's positions at the marks plus this order; an order that reduces its instrument's position passes |
| 53 | GatewayNetNotional |
[gateway] max_net_notional: the same for the signed sum |
| 54 | GatewayFxRateUnknown |
[accounting]: the order adds to exposure in a currency whose source has no valid book in the gateway, or none that updated within [accounting] stale_fx_ms |
| 55 | GatewayUnderlyingNet |
[gateway.underlying.<BASE>] max_net: the account's net position in the base asset over every venue, with the orders working on the order's side and this one, would move further past the limit |
| 56 | GatewayUnderlyingMarkUnknown |
[gateway.underlying]: an inverse contract of the underlying with a position or working orders has no valid book in the gateway, or none newer than [risk] stale_md_ms |
| 57 | GatewaySelfTrade |
[gateway.shared]: the order would trade with a resting order of another attached strategy on the same instrument |
| 58 | GatewayBalanceShort |
[gateway] check_balance: the account's balance on the venue, less the orders of every attached strategy, does not cover the order |
Venue-originated¶
The connectors map each venue's error codes onto these (*_error_map.hpp per venue; the code tables are on Venue connectors).
| Value | Name | Transient | What to do |
|---|---|---|---|
| 64 | VenueReject |
unknown | the code was not in the venue's table; read the message in the log |
| 65 | PostOnlyWouldCross |
yes | the quote would have taken; expected when quoting at the touch, and excluded from [engine] reject_backoff_ms |
| 66 | InsufficientBalance |
no | fund the account or shrink the quote |
| 67 | VenueRateLimit |
yes | the connector cools down; lower the request rate |
| 68 | VenueUnknownOrder |
no | the order is gone at the venue; most codes also trigger a reconciliation |
Kill reasons¶
KillReason (include/fastmm/core/enums.hpp), the first reason each kill-switch bit was set for. fastmm-top shows it next to the state and per venue; the status file carries it in kill_reason.
| Value | Name | Scope | on_kill applies |
First thing to check |
|---|---|---|---|---|
| 0 | None |
— | — | not killed |
| 1 | Requested |
global | no, it always shuts down | why the shutdown started: signal, --duration, slow tier |
| 2 | MaxLoss |
global | yes | the PnL and the market before restarting |
| 3 | TransportFull |
global | yes | [engine] order_ring_bytes, and whether the network thread is starved |
| 4 | JournalOverflow |
global | yes | [engine] journal_ring_bytes, and disk write throughput |
| 5 | AllVenuesKilled |
global | yes | each venue's own kill reason |
| 6 | VenueFatal |
one venue | when it is the last venue | the key, its permissions and its environment |
| 7 | VenueHardStop |
one venue | when it is the last venue | a Binance 418 IP ban; terminal until the process restarts |
| 8 | OrderRingOverflow |
global | no, it always shuts down (exit 5) | [engine] order_ring_bytes, engine thread pinning |
| 9 | StrategyError |
global | yes | a Python hot hook raised, called ctx.fail or produced a non-finite value |
| 10 | FeedLost |
one venue | when it is the last venue | a nasdaq_itch feed that cannot rebuild its books |
| 11 | OrderIdsExhausted |
global | yes | the session's 32-bit order id sequence ran out; restart |
| 12 | DeadMansSwitchLost |
one venue | when it is the last venue | the venue refused the countdown refresh for a whole window |
| 13 | GatewayMaxLoss |
every venue | yes | the gateway's account: [gateway] max_loss over every strategy |
| 14 | GatewayOperator |
every venue | yes | an operator's kill on the gateway's control socket |
A running session's kill switch is cleared by fastmm-ctl unkill or SIGHUP; with on_kill = "exit" the process has already exited. A latched max_loss trip refuses the next start (exit 6) until --clear-kill. What each reason does to quoting and orders: Kill switch and shutdown.
Venue actions¶
VenueAction (include/fastmm/venues/error_action.hpp) is what the connector does beyond reporting the reject. Every venue's error map returns one.
| Value | Name | Connector behaviour |
|---|---|---|
| 0 | None |
reports the reject and nothing else |
| 1 | Backoff |
retries later |
| 2 | RateLimit |
enters the rate limiter's cooldown, honouring Retry-After when the venue sends one, else 10 s |
| 3 | ResyncClock |
fetches the venue's server time and recomputes the offset (a no-op on Deribit) |
| 4 | Reconcile |
requests the venue's open orders |
| 5 | DisableInstrument |
the configured tick, lot or minimum is wrong for that symbol |
| 6 | HardStop |
stops REST until the process restarts, and trips the venue's kill switch with VenueHardStop |
| 7 | Fatal |
disables order entry and trips the venue's kill switch with VenueFatal |
Journal errors¶
tools/journal_dump.py and fastmm-replay report these:
| Symptom | Meaning |
|---|---|
trailer MISSING |
the session did not shut down cleanly; events before the damaged block are still readable. fastmm-replay refuses such a file without --allow-incomplete |
| a CRC failure on a block | the block is damaged; --no-crc reads past it |
replay MISMATCH |
the replayed order stream differs from the recorded one; only the first difference means anything (Determinism) |
| a what-if warning | --config or --strategy differs from the recording, so a match is not expected |
Related¶
- Troubleshooting: every log message with its cause and fix.
- Status file: where the counters and reasons are published.
- Storage:
sessions.clean_shutdown,journal_completeandrecords_droppedsay what a session's record is missing. - Run in production: restarts, venue-side switches, monitoring and accounting.