Inference & routing
How a real request should move through eligibility, execution, metering, and failure handling.
In this chapter
The current integration boundary#
An operator-only adapter can be configured for an HTTPS OpenAI-compatible endpoint. It uses a fixed server-configured model and credential, bounded input/output, a timeout, and no automatic redirect or retry. It rejects missing configuration rather than returning a simulated answer.
That adapter is not the customer marketplace. Its mock-transport tests establish handling of configuration, limits, errors, and response shape—not that a GPU has executed a production request or that a customer has been billed correctly.
Intended request path#
Buyer or agent
→ authentication and request validation
→ eligible model/provider selection
→ price quote and spending authorization
→ durable reservation
→ provider execution
→ usage reconciliation
→ charge, supplier payable, and receiptRouting must filter before optimizing. A provider that violates the buyer's model, context, region, privacy, or feature requirements is not a valid fallback just because it is cheaper. Among eligible providers, the platform can evaluate price and measured service quality.
Streaming and failures#
| Situation | Required behavior before paid launch |
|---|---|
| Provider rejects before execution | Release the appropriate reservation and report the failure accurately. |
| Timeout with unknown outcome | Reconcile before retrying or charging again; a timeout is not proof that no work occurred. |
| Partial streamed response | Use an explicit policy for delivered tokens, upstream costs, cancellation, and refunds. |
| Client disconnects | Propagate cancellation where supported and reconcile any remaining provider work. |
| Provider reports unexpected usage | Bound, verify, and investigate it; do not accept arbitrary charges blindly. |
| Privacy-qualified provider is unavailable | Fail or ask for permission; never silently downgrade privacy. |
The first supply: OpenRouter#
The initial production supply is OpenRouter's aggregation API, activated as operator-owned offers covering its published model catalog. It is labeled as operator supply, not independent marketplace liquidity, and buyer-facing prices carry a disclosed markup over upstream cost.
This is deliberately a bootstrap. It proves the full transaction loop — reserve, upstream execution, reconciliation, capture, receipt — without provisioning GPUs. It does not make IdleAI cheaper than buying the same upstream directly; that requires direct suppliers pricing below aggregator rates. Rented-GPU supply (e.g. Runpod) and third-party operators remain the path to genuinely cheaper inference.