Cost calculation and results
The result tells you whether the cost per successful operation increased beyond your threshold for the test cases you ran. MarginCI calculates each revision separately:
cost_per_success = total calculated cost across all attempts
/ successful operations
measured_regression = candidate cost_per_success / baseline cost_per_success - 1A test case supplies an ID and input. An attempt runs the operation for that case in a new process. Whole-operation retries add attempts, but each case counts as at most one successful operation per revision. Your success check determines whether the operation succeeded.
Total cost includes every attempt, including retries and failures. Costs use configured rates or supported provider estimates, not settled invoice charges. Unknown usage is never treated as zero. MarginCI compares exact decimal values before rounding them for display.
Read the verdict
| Result | Meaning | Exit code |
|---|---|---|
| PASS | Measurements are complete and the increase in cost per successful operation is at or below your threshold | 0 |
| FAIL | Measurements are complete and the increase exceeds your threshold | 1 |
| INCONCLUSIVE | Required measurements are incomplete or the comparison cannot be calculated | 2 |
| ERROR | Configuration, repository setup, dependencies or execution prerequisites are invalid | 3 |
Both revisions must complete the declared test workload, each must have at least one successful operation, and the baseline cost must be positive before PASS or FAIL is possible. Equality with the threshold passes. For example, max_regression: 20% allows candidate cost per successful operation up to 1.2 times the baseline.
Choose test cases that reflect the work you care about. The result covers those executions, not unseen production traffic. Review the success checks separately: a cost PASS does not establish application quality.
Only a completed test with PASS approves the cost check. init and doctor can exit zero after checking setup without performing an economic comparison.
Resolve an incomplete or failed check
Read the report’s reasons before rerunning; test may incur further provider charges.
| Situation | Next action |
|---|---|
| FAIL | Inspect provider totals and attempts to locate the increase. Check application changes, retries and success counts before changing the threshold. |
| No successes or zero baseline cost | Review test inputs, success checks and rates. Either condition makes the ratio undefined and returns INCONCLUSIVE. |
| Missing usage, unfinished work or incomplete cases | Check the provider requirements, consume all responses and await paid work. |
| Spend guard or timeout reached | Inspect retained attempts and any remote jobs. Adjust request limits or bounds only after checking the possible cost. |
| Undefined or unrepresentable ratio | Inspect costs and success counts; MarginCI returns INCONCLUSIVE rather than rounding an invalid ratio into a verdict. |
| ERROR | Fix the reported field, dependency or repository problem, then run doctor before test. |
An operation that fails its success check can still have complete cost measurements. Its cost remains included; the failed outcome alone does not make the comparison INCONCLUSIVE.
What is compared
| Input | How MarginCI uses it |
|---|---|
| Baseline | --against, then compare.against, then local origin/HEAD; resolved to an immutable commit |
| Candidate | Current HEAD plus tracked and nonignored untracked files, including dirty edits and deletions |
| Test cases, rates, provider endpoints, limits and success protocol | Candidate configuration is frozen once and applied to both revisions; saved in config.json with config_sha256 |
| Command | Shared command string, including any CLI override; npm/pnpm scripts resolve from each revision’s package.json |
comparison_files |
Listed wrappers, input files and success helpers must be committed, nonignored, present and identical on both revisions |
| Application code, dependencies and TypeScript configuration | Taken from each revision; dependencies installed separately from frozen locks |
| Environment and external state | Environment inherited by both; credential values are not recorded or hashed. You must arrange comparable external state. |
Declare the wrapper and its input/success helpers in comparison_files. The default is an empty list; without it, those files can change with application code. MarginCI cannot infer which imported modules define test inputs or success. See Repository setup for the recommended sequence.
comparison_identity records commits, the candidate content digest and dirty flag, config hash, resolved commands, comparison-file hashes and success protocol. source-files.json records each source file’s hash before dependency installation. Snapshots preserve the files used. Ignored application inputs are not copied into either snapshot.
Spend guard and cancellation
max_spend_usd covers both revisions, all selected test cases, SDK requests and whole-operation retries. Before a billable request starts, MarginCI reserves its configured max_request_usd against the remaining budget, including outstanding requests. A Firecrawl batch keeps its reservation until its final status and credit usage are known. Requests that exceed available reservation capacity are refused.
The spend guard uses configured values and request bounds. It cannot guarantee a cap on your provider bill. If a request costs more than its bound, the charge can occur before MarginCI detects it; REQUEST_BOUND_EXCEEDED returns INCONCLUSIVE. Already dispatched requests and remote jobs may continue. Use provider token, page, result and concurrency limits, plus provider-enforced account controls where available.
A timeout or SIGINT/SIGTERM during measurement stops the local operation process group and returns INCONCLUSIVE with retained evidence. No new operation starts after cancellation. Stopping a local process does not cancel remote provider work. Review unresolved jobs before retrying.
SIGKILL or machine failure can prevent a final report. An absent completed PASS must never approve the check. Cancellation during dependency installation returns ERROR before provider work begins.
Synthetic example
The homepage uses calculated totals of $1.96 and $3.39, with 20 successful operations on each revision. Cost per successful operation is $0.098 and $0.1695. The increase is 72.95918367…%, displayed as 72.96%, so it fails a 20% threshold. These are illustrative values, not provider prices or customer results.