Measurement, comparison identity and spend
cost_per_success = total measured economic value across ALL attempts
/ successful operationsFailed-attempt spend remains in the numerator. Unknown is not zero. Configured economic values are not invoices; Exa estimates retain their provenance. Exact decimal values determine the gate before display rounding.
| Result | Meaning | Exit |
|---|---|---|
| PASS | Every required declared case has certifiable measurement, both sides have successes, baseline value is positive, finite measured regression ≤ threshold | 0 |
| FAIL | Same complete comparison, measured regression > threshold | 1 |
| INCONCLUSIVE | Measurement cannot establish a trustworthy economic comparison | 2 |
| ERROR | Configuration, source/dependency construction, command or execution precondition invalid | 3 |
Only PASS approves an economic test. init and doctor can exit zero for their own setup operations; they are not economic gates. Zero successes, zero baseline value, undefined/unrepresentable ratios, missing cases, unknown paid value, unfinished work and budget-limited runs are INCONCLUSIVE. A complete failed operation is an observed failure, not automatically missing measurement.
The scope is the finite declared workload. Success semantics and quality remain application/user responsibilities. A PASS describes the cases actually executed; it cannot establish the cost of unseen production traffic. Include representative inputs and costly edge cases in your workload. No confidence setting turns unobserved production modes into evidence.
What is compared
| Input | Policy / identity |
|---|---|
| Baseline application | --against, else operation compare.against, else local origin/HEAD; resolved to immutable commit and blob-verified Git export |
| Candidate application | Current HEAD plus SHA-256 inventory of tracked and nonignored untracked files, including dirty edits/deletions; no overlay into baseline |
| Config, case IDs, inline inputs, provider endpoints, rate/estimate choice, budget, limits, success protocol, dependency mode | Effective candidate configuration frozen once and applied to both sides; config_sha256, saved config.json |
| Command string | Shared, including CLI override; npm/pnpm scripts resolve from each side’s package.json; both argv arrays recorded |
| Wrapper, success helper and external workload files | Revision-owned source by existing policy; explicitly declared comparison_files must be committed, nonignored, unchanged and present on both sides |
| Dependencies / TS compiler config | Revision-owned; locks/manifests included in source-files.json; isolated frozen installation; supported TS configuration read in that snapshot |
| Environment / external state | Inherited for both; secret values not recorded or hashed. Application must arrange comparable state, honor cases and await paid work |
The default comparison_files: [] preserves existing direct-application integrations; such entries/success logic may change with revision-owned source. Their source hashes and resolved commands disclose changes. The recommended wrapper workflow freezes the wrapper and every separately stored workload or success helper. MarginCI cannot infer which imported module is a workload versus application code: declare those dependencies explicitly. Candidate-only/untracked/ignored config remains the existing shared-input policy; commit config in production. Ignored application inputs are not snapshotted and cannot be supplied by copying the current working tree into baseline.
comparison_identity records baseline/candidate commits, candidate content digest and dirty flag, effective config hash, resolved argv, frozen-file hashes, success protocol and environment policy. source-files.json records SHA-256 of every pre-install source file on each side; snapshots and config.json preserve bytes. A clean candidate still uses the legacy HEAD+working:digest identity; the separate candidate_commit is the immutable Git identity. Detached HEAD works. Missing baseline is actionable ERROR. No hardcoded main/master fallback or automatic network fetch occurs.
Spend guard and cancellation
max_spend_usd spans both revisions, every selected case, SDK requests and whole-operation retries. Before dispatch, each billable request reserves max_request_usd against remaining known spend plus outstanding reservations. Firecrawl creation retains its reservation until terminal owned-resource valuation; nonbillable status observations still require complete responses. Requests exceeding available reservation capacity are refused. A check stops on incomplete measurement; known failed spend is retained.
This is an economic-certification guard, not a guaranteed provider invoice cap. Request bounds are user assertions. An underestimated bound may incur excess spend before completion reveals it; REQUEST_BOUND_EXCEEDED makes the check INCONCLUSIVE. Already-dispatched concurrent requests and remote resources can continue. Unknown final charges remain unknown, never a certified zero. Use native token/page/result/concurrency limits and provider-enforced account controls for a cash cap.
An operation timeout or SIGINT/SIGTERM during measurement kills the owned local process group on supported systems and returns INCONCLUSIVE with preserved dispatch evidence. No new operation is dispatched after cancellation. Remote work is not canceled by killing a local process; unresolved provider work/final charges are disclosed. Forced SIGKILL of MarginCI or machine failure cannot produce a final report; absence of a completed PASS must never approve a gate. Dependency installers have separate bounded timeouts; cancellation before measurement is ERROR and spends no provider budget.
The homepage’s synthetic example uses unrounded totals $1.96 /20 and $3.39 /20. Actual regression is 143/196 = 72.95918367…%, displayed 72.96%. These illustrative values are not provider prices or customer savings.