OpenAI
OpenAI-compatible chat and text completions, including streams with final usage data.
Costs use the model rates you configure.
COST REGRESSION TESTING / PRIVATE ALPHA
Catch API cost regressions before merge.
Run the same test cases on two revisions. MarginCI measures supported paid API usage and checks the increase in cost per successful operation against your threshold.
For Node.js and TypeScript operations that run in one process, locally or in CI.
Request alpha access
See example Email hello@marginci.com
FAIL · EXIT 1
+72.96%Measured regression
20.00%Allowed regression
| Same workload | MAIN | PR |
|---|---|---|
| Cost per successful operation | $0.098000 | $0.169500 |
| Successful operations | 20/20 | 20/20 |
| OpenAI | $1.820000 | $3.110000 |
| Exa | $0.140000 | $0.280000 |
| Total calculated cost | $1.960000 | $3.390000 |
Synthetic example. Calculated costs are illustrative and may differ from provider charges.
View CLI output
MarginCI — research
Declared workload: 20 cases
Baseline: MAIN
Candidate: PR
BASELINE CANDIDATE
successful 20/20 20/20
openai $1.820000 $3.110000
exa $0.140000 $0.280000
known value $1.960000 $3.390000
cost / success $0.098000 $0.169500
measured regression 72.96%
allowed regression 20.00%
Values follow frozen configured rates / allowed provider estimates; not settled invoice charges.
FAIL
Verdict applies to the declared MarginCI workload. It is not an estimate of unseen production traffic.
Exit: 1
01 / HOW IT WORKS
One workload. Two revisions.
Test a change to your application with the same inputs, success checks and rates on both sides.
- 01
Define the work
Choose test inputs and the checks that decide whether an operation succeeded.
- 02
Run both revisions
Use the same workload in isolated copies of the baseline and candidate.
- 03
Compare cost per success
Include spending from failures and retries, then check the increase against your threshold.
- Failed attempts count.
- Total measured cost includes failed attempts and retries.
- PASS and FAIL require a complete comparison.
- Both revisions must complete the declared workload, have a successful operation and produce a defined ratio with a positive baseline cost. PASS means the increase is within your threshold; FAIL means it exceeds it.
- Missing measurements cannot pass.
- Incomplete or undefined comparisons return INCONCLUSIVE. Invalid configuration or execution prerequisites return ERROR.
- The result covers your test cases.
- The comparison describes the workload you ran. Costs use configured rates or supported provider estimates; production costs and provider charges may differ.
Runs in your environment. MarginCI runs on your machine or CI runner. Your application calls its configured API providers. Security and data handling ↗
03 / COMPATIBILITY
Start with a supported API.
Check the provider documentation for supported requests, required usage data and configuration.
Firecrawl
v2 batch scraping.
Costs use final reported credits and your configured credit rate.
Exa
Supported search requests.
Costs use your configured rate or Exa’s estimate when enabled.
Anthropic
Text generation through the Messages API, including streaming.
Costs use configured token and cache rates.
Gemini
Text generation through the Gemini Developer API, including streaming.
Costs use configured token and cache rates.
Pinecone
Serverless vector queries.
Costs use reported read units and response size at your configured rates.
Run MarginCI with Node.js 24.21.0 on macOS (Apple silicon) or Ubuntu 24.04 (x64). See Compatibility for exact supported configurations.
The spend guard uses configured costs and request bounds. Provider charges can exceed its limit.
PRIVATE ALPHA
Start with one
real operation.
Request access by email. Include a brief description of the operation and the paid APIs you would like to test.