Configuration and CLI

Use marginci.yml to define the operation, test cases, provider rates and limits. Schema version 1 is strict: unknown fields, duplicate keys, ambiguous numbers and unsupported options return ERROR with the affected field paths.

Configure one operation

This configuration shows the required structure for an OpenAI-compatible operation. Replace your-model-id with the exact requested and returned model ID, and supply your own rates and request bound. These numbers are not provider prices. Configure your application endpoint and model separately; running the operation can incur provider charges.

version: 1
providers:
  openai:
    base_url: https://api.openai.com/v1
    credential_env: OPENAI_API_KEY
    max_request_usd: 0.01
    models:
      your-model-id:
        input_per_million: 0.1
        output_per_million: 0.2
        cached_input_per_million: 0.05
operations:
  research:
    command: node marginci/research.cjs
    comparison_files: [marginci/research.cjs]
    providers: [openai]
    cases:
      - id: company-a
        input: {query: example.com}
    compare: {against: HEAD}
    gate:
      metric: cost_per_success
      max_regression: 20%
      max_spend_usd: 1
    limits:
      max_cases: 50
      max_attempts: 1
      max_requests: 50
      timeout_ms: 30000
      max_response_bytes: 2097152

The example sets compare.against: HEAD for an initial comparison. Remove or change it after setup so later checks use your intended baseline.

Credential values belong in environment variables, not YAML. credential_env names the variable required before dispatch; it does not reconfigure your SDK or choose a model. If omitted, your application remains responsible for providing credentials. Missing provider pricing configuration returns ERROR before dispatch: OpenAI needs a model-rate table, Firecrawl a credit rate, and Exa a rate or enabled estimate. Requests naming unpriced models are blocked before dispatch; an unexpected response model or missing cache rate still invalidates measurement.

Fields and defaults

Additional provider configuration

Add only providers your operation uses to both the top-level providers mapping and the operation’s providers list. Anthropic, Gemini, Pinecone and experimental Twilio require explicit currency: USD and their exact API version. Anthropic, Gemini and Pinecone also require rate_source: a description or URL identifying the frozen tariff, platform, region and tier being modeled. There is no automatic rate lookup. Rates are nonnegative YAML numbers with the units shown below; max_request_usd remains mandatory.

Use credential_env for each credential’s environment-variable name. Default direct bases are https://api.anthropic.com/v1 and https://generativelanguage.googleapis.com/v1beta. A Bedrock runtime base is https://bedrock-runtime.REGION.amazonaws.com/anthropic/v1; replace REGION with the authorized region and configure the SDK to use the same native Messages route. Pinecone has no default index. Request output/thinking bounds are set in application calls; declaring rates does not constrain those calls automatically. See provider contracts for formulas, completion and unsupported features.

Experimental twilio additionally requires experimental: true, api_version: '2010-04-01', account_sid_env, price_source: terminal_message_price, and account_addons: none. Only assert none after establishing that separately billed messaging add-ons are disabled. Its default base is https://api.twilio.com/2010-04-01. max_status_reads defaults to 12 (maximum 12); observation_timeout_ms defaults to 120000 (maximum 120000). The application must await its own status reads within these limits and the operation timeout. Twilio is excluded from the live-validated provider list; the template-only trial does not exercise the implemented custom-message path.

CLI commands

Numeric syntax is nonnegative decimal (including supported scientific notation). Empty, negative, nonfinite, hexadecimal and coercive values fail. max_spend_usd is a number; the CLI equivalent is --max-spend 1.

After installation adds marginci to PATH, use init and doctor for nonpaid setup checks. Every test command below can make paid API calls; review the configuration and spending limits first:

marginci init
marginci doctor research --against HEAD
marginci test research --against HEAD --json
marginci test research --against HEAD --max-regression 20% --max-spend 1 --cases company-a --verbose
marginci test research --against HEAD -- node marginci/research.cjs

The final command overrides the configured command for both revisions. --cases selects the test cases for this run; that subset becomes the declared test workload. All overrides are recorded with the effective configuration. --json always emits one JSON document; verbose progress goes to stderr. --version and --help are available.

Success checks and attempts

By default, the operation’s exit status determines success. To report success explicitly, set results: {format: jsonl, path: .marginci/results.jsonl}. The runner deletes stale results and supplies MARGINCI_RESULTS_PATH. Each process writes exactly one { "case_id": "company-a", "success": true } line with no extra fields or duplicate keys. A nonzero exit or signal overrides true. Missing or ambiguous output makes the measurement incomplete.

Test cases receive MARGINCI_CASE_ID, MARGINCI_CASE_JSON and MARGINCI_ATTEMPT_ID. Whole-operation retries run in new processes, include failed attempts in total cost and count at most one successful operation per test case. SDK retries count as additional observed requests. MarginCI adds no provider retries.

Requests with no variable cost

free_egress optionally lists exact nonprovider origins and a reason of at least 8 characters explaining why they have no variable cost. MarginCI relies on your assertion. It cannot bypass measurement of a configured provider host. Do not exclude paid downstream work.

See Cost calculation and results for PASS (0), FAIL (1), INCONCLUSIVE (2) and ERROR (3). Setup commands exiting zero do not approve a cost comparison.