Back to app MetaLens360Help Center

AI provider health and the self-heal doctor

Platform administrators can see the health of every AI provider on the platform key, which model each one uses and why, and let the AI doctor repair retired models, skip broken providers and raise issues it cannot fix.

Updated in Administration & security

Overview

AI providers change without notice: a model is retired, a free model stops being free, a key expires, credits run out. The AI provider health panel shows how each provider on the platform key is doing, and the AI doctor repairs what it can on its own. The doctor follows fixed rules and uses no AI.

  • Health per provider — worked out from the last 7 days of AI calls: last success, last error, success rate over 24 hours and 7 days, and failures in a row.
  • No fixed model per provider — the configured models are preferences. Each provider's live model list is read, and a model that is no longer offered is skipped in favour of the provider's next eligible model. If the model list cannot be read, the configured models are used as they are.
  • Billing is checked per model first — a "payment required" or "prepayment credits are depleted" answer on one model usually means that model isn't free for the key. That model is set aside and the provider's next model is tried straight away. The provider itself is set aside only when every one of its models asks for payment.
  • Cool-down — a provider that fails with a permanent error (key rejected, payment required on every model, model gone or no longer free) is skipped for 6 hours instead of being tried on every request. Rate limits, overload and timeouts never start a cool-down.
  • The doctor — repairs providers by itself, without anyone clicking anything. It runs:
    • right after a failure, in the background, as soon as the model in use fails with a permanent error (model gone, not found, not free, payment required, no access, key rejected). It runs at most once per provider every 15 minutes. Rate limits, overload, server errors and timeouts never start it, because they pass on their own.
    • after a burst of failures: when one provider fails 5 calls within 10 minutes, for any reason (rate limits, timeouts and server errors included), the doctor checks that provider at once. It tests the model in use once, switches to another model if that one is the problem, opens an issue if a person must act, and otherwise retries later. A rate-limited provider is not tested until its limit resets. At most once per provider every 15 minutes.
    • every 2 hours on a schedule.
    • after every update (deploy), a few minutes after the new version starts. A server that starts for more traffic (not a new version) skips this when the doctor ran in the last 30 minutes.
    • and whenever you click Run doctor now.
  • Free-tier analysis for SambaNova, Cerebras and OpenRouter — on every run the doctor checks each model of these providers, decides whether it is free from the provider's own signals, ranks your preferred order first and the newest free models after it, verifies the top one with a tiny test request and uses it. A paid model is never chosen. See Read the free-tier model analysis below.
  • Issues only when you must act — the doctor opens an issue only for something a person has to fix: a rejected or missing key, payment required on every model it tried, a suspended account, or a pinned model that fails. Everything it can fix or retry by itself is written to the audit log only. Every issue closes itself as soon as the provider answers again.

The panel covers the platform key only. A workspace or a person using their own key is never skipped because the platform key is failing. Only switching a provider off (below) affects their keys too.

Before you start

  • You need platform administrator access. Other users don't see the panel.
  • Open Settings, then the AI Providers tab. The panel is at the top.

Steps

Read the provider table

  1. Find the provider in the AI provider health table. Providers without a platform key are not listed.
  2. The Health badge reads Healthy, Degraded, Failing, Cooling down (with the time it ends) or Unknown (no calls in the last 7 days). Hover over it for the last error, the success rates and the cool-down reason.
  3. The Model and why column shows the model in use, where it came from — Auto (chosen and verified by the doctor), Pinned (chosen by an administrator) or Configured (the deployment's preference list) — and a short reason, for example "doctor: first live preferred id", "auto: newest live chat model — preferred ids gone" or "pinned by owner".
  4. If a pin or an allowed-models list has no live models, the row says "your allowed list has no live models; using auto selection". The doctor and the AI then ignore that list and pick from the live models, without changing what you saved. Change or clear the list when you're ready.
  5. The Endpoint column shows the address and protocol each provider is called on. Blue means an administrator set it (see Edit a provider's connection settings).
  6. The last column shows the last repair, and Open issue when the doctor raised one.

Run the doctor now

You rarely need to: the doctor already runs after a failure, after a burst of failures, every 2 hours and after each update. Use the button to check straight away, for example after you replace a key.

  1. Click Run doctor now above the table.
  2. For each failing provider the doctor:
    • Model retired, not found, no access or no longer free — reads the live model list, keeps only the free chat models (no previews, image, speech or paid-only models), puts the newest first and checks them one by one with a tiny test request, up to 5 per run. A model that answers "not found", "payment required" or times out is skipped and not tried again for 24 hours. The first model that answers is saved and an audit entry is written, for example "Groq: model-a → model-b, verified". No issue is opened. If no model answers yet, that is written to the audit log and the doctor tries again on its next run.
    • Model not available — a provider answering 410, or "The requested model … is (currently) not available" (model_not_available), is saying something about that one model, not about the provider. The model is skipped (24 hours when it is gone, 1 hour when it is "currently not available"), live requests move straight on to the provider's next model, and the provider itself is never marked down or put in a cool-down. The doctor then picks a replacement the same way as for a retired model.
    • Payment required — billing is about the provider account, not one model. The provider is skipped for 6 hours and the next provider answers. The doctor tests the model that failed once more (a topped-up account recovers at once and every rest ends); otherwise one payment issue stays open with the exact error.
    • Run doctor now tests several models — for a failing provider it tries the model in use first, then the provider's other models in order, up to 5, until one answers. The result shows the model that answered (and saves it as the provider's model unless a model is pinned), or every model's exact error when none did.
    • Key rejected or missing, paid provider out of credit, or account suspended — code can't fix these. The doctor opens one issue with the provider, the exact error, what it tried and how to fix it. A later run updates the same open issue instead of opening another.
    • Unreachable, overloaded or timing out — even for a long time, this is not something you can fix, so no issue is opened. The doctor writes it to the audit log and keeps retrying.
  3. A key rejected during a repair stops it at once, because no other model would answer either.
  4. Each provider is repaired at most once per run. The automatic runs wait 30 minutes before testing the same provider again, so free quotas are not used up. Run doctor now tests it every time you click.
  5. The results appear under the button. Healthy providers show OK. Click Open issue to jump to an issue.

A repair sticks: failures recorded before a verified repair no longer count, and neither do failures on a model the provider no longer uses. On its scheduled runs the doctor also moves a healthy provider to a newer free release of the same model (for example a new Flash-Lite version) after one successful test request. Pinned models are never moved.

Read the free-tier model analysis

The Free-tier model analysis section under the table has one panel per free-tier provider with a platform key.

  1. Chosen is the model in use, with the evidence that it is free (for example "probe answered on the free-tier key") and when it was checked.
  2. Quota / credits shows what the provider reports:
    • SambaNova — requests left today and this minute, read from each reply. When the day's requests are used up, that model is skipped until the daily reset and the next model or provider answers.
    • OpenRouter — free requests left today, credit left on the key and usage, read from OpenRouter's key endpoint.
    • Cerebras — Cerebras doesn't publish usage or limits to apps. Check Limits and Usage in the Cerebras console.
  3. The table lists each model with a Verdict and its Evidence:
    • Free — a test request or a live request answered on the free-tier key, or (OpenRouter) the model is listed at a price of 0.
    • Paid / not free — the model asked for payment or credits, is priced in OpenRouter's list, or tripped the paid guard. It is never chosen.
    • Retired — the provider no longer lists it for this key, or it answered 410 / "model not available".
    • Quota spent — free, but its free requests are used up for now.
    • Not verified / Probe failed — not proven free yet; the doctor tests it on a later run.
  4. The doctor re-checks on every run. It moves to a newer free model when one appears and drops a model that becomes paid or disappears. A free result is re-tested after 12 hours. Each automatic run sends at most 3 test requests per provider, and none within 30 minutes of the last; Run doctor now may send up to 5.
  5. SambaNova: the live model list decides. The preferred models (DeepSeek, gpt-oss, Llama) are only preferences. The doctor reads SambaNova's model list for your key and uses the preferred models that are actually on it, then other DeepSeek, gpt-oss or Llama models on it, then the other chat models on it, newest first. A preferred model that isn't on the list, or that answered "not available", is skipped. The panel shows it under Skipped preferred models with the reason, and it appears as Retired in the table.

The paid guard protects OpenRouter: only free models are ever requested, every request is capped at a price of 0, and if a reply still reports a cost or comes from a priced model, that model is blocked, the provider moves on to the next free model, and an issue is opened. The issue stays open until you clear the block.

Switch a provider off or on for the platform

  1. In the AI provider health table, use the switch next to the provider's name.
  2. Off means the provider is not used anywhere: not by the AI assistant, the copilots, knowledge search or screenshot reading, and not by the doctor. This includes every workspace's own key and every person's own key for that provider. The platform key is kept, so switching it back on needs nothing else.
  3. Switching a provider off closes the doctor's open issues about it.
  4. If it is the last free provider still on, you're asked to confirm first, because free AI would stop.

Every switch is written to the audit log.

A workspace that still wants the provider can turn it back on for itself, on its own key: see Configure AI providers. The platform key stays off for that workspace too. When you switch the provider back on, those workspace choices no longer matter, but they are kept in case you switch it off again. The doctor only looks at the platform key and never tests a workspace's key.

Set the preferred models

  1. Click the model badge on the provider's row, or Preferred models / pin on its free-tier panel.
  2. Under Preferred models (in order), type the model ids, best first, separated by commas, and click Save order. Leave the field empty and save to go back to the deployment's list.
  3. The doctor still skips a model the provider no longer lists and still verifies the choice with a test request.

To unblock a model the paid guard blocked, check in the provider's console that it is free again, then click Clear paid-guard blocks in the same dialog.

Pin a model

  1. Click the Auto, Pinned or Configured badge on the provider's row.
  2. Choose a model from the list, or Other — type a model id…, and click Pin model.
  3. To remove the pin, choose Automatic (recommended) and click Use automatic.

A pinned model is used on its own and the doctor never overrides it or upgrades it. If it fails, the doctor opens an issue instead of swapping it. The issue closes when the pinned model answers again or when you change or remove the pin.

Edit a provider's connection settings

  1. Click the provider's address in the Endpoint column.
  2. Change what you need: Base URL, Protocol (OpenAI compatible, Anthropic or Gemini), Models endpoint path, Timeout, Max output tokens, Temperature, Extra headers (one "Name: value" per line), Rate limits (requests per minute, tokens per minute, requests per day) and Free / paid.
  3. Click Save. The change applies on the next request, on every server within about a minute. No update or deploy is needed.
  4. Each field shows where its value comes from: Admin (set here), Env (the deployment) or Default (built in). Click Reset next to a field, or Reset all to default, to go back to the deployment's value.

Rules that keep this safe:

  • The address must use https and a public host name. Internal, private and local addresses are refused, and the check runs again on every Test and every call.
  • Headers can't carry keys, tokens, secrets, cookies or authorization. API keys are never shown or changed here.
  • For a free provider, OpenRouter's maximum price stays 0. Marking a free provider as paid needs a confirmation.
  • Every change is written to the audit log (AI_PROVIDER_CONNECTION) with the old and new values.

Add your own provider

Below the health table, Added providers lets a platform administrator bring any provider that speaks the OpenAI, Anthropic or Gemini protocol.

  1. Click Add provider.
  2. Enter a Name, pick the Protocol and enter the Base URL (required for OpenAI compatible, for example https://api.example.com/v1; optional for Anthropic and Gemini).
  3. Optionally list Models, best first. Leave it empty on a free provider and the doctor picks from the provider's live model list. A paid provider needs at least one model.
  4. Tick or untick Free provider. For a paid provider, enter the prices per million tokens.
  5. Optionally set the Request parameters (models path, timeout, max output tokens, temperature) and paste the API key.
  6. Click Add provider.

The provider joins the end of the provider order and then behaves like a built-in one: switch it on or off in the health table, use Test, change its connection settings, and let the doctor repair it. The key is stored encrypted as the platform key and is never shown again; to replace it, click Edit and type a new one. The address passes the same https and public-host check on save and on every call. Adding, editing and removing are written to the audit log (AI_PROVIDER_ADD, AI_PROVIDER_EDIT, AI_PROVIDER_REMOVE) without the key.

To remove an added provider, click Remove. You first see what happens: where it sits in the provider order, how many workspace orders list it, which stored keys are deleted and whether it is the last free provider on. Tick the confirmation and click Remove. To pause a provider instead, switch it off: its key and settings are kept. Built-in providers can't be removed, only switched off.

Fix an open issue

  1. Read How to fix in the issue under the table.
  2. For a rejected key, create a new key in the provider's console and save it as the platform key on the AI Providers tab.
  3. For billing or a suspended account, sort it out with the provider, or switch the provider off.
  4. You don't need to close the issue. As soon as the provider answers again (a live request, the doctor's next test or a repair), the issue is marked Resolved with the note "Resolved automatically: provider answered on model at time". Click Run doctor now if you don't want to wait.

Example

Groq retires the model a deployment listed first. Calls on that model fail with "model does not exist", so the next eligible Groq model is used at once. The failure also starts the doctor in the background: it reads Groq's model list, tests the replacement once and saves it, usually within a minute. The row shows Auto and "✓ Groq: model-a → model-b, verified", and no issue is opened.

Gemini answers "402 — prepayment credits are depleted" on the model in use, because the key's project has no free access to it. That request moves on to the next Gemini model straight away and Gemini stays in service. The doctor starts at once, tests the model once more, then tests the newest free Gemini model, for example a Flash-Lite "latest" model, and saves it: "Google Gemini: model-a → model-c, verified". If every free Gemini model asks for payment, one payment issue lists each model and its error.

FAQs

Does the doctor ever switch to a paid model?

No. When OpenRouter says a free model is "unavailable for free" and suggests a paid one instead, that free model is set aside and the next free model is used, ending with OpenRouter's free-models router. The suggested paid model is never used. On free providers it chooses only among free chat models. On a paid provider it chooses only among the models the deployment lists for that provider.

Can SambaNova, Cerebras or OpenRouter start charging?

Not through MetaLens360. OpenRouter is only ever sent free models with a price cap of 0, and a reply that reports a cost blocks that model. SambaNova stays free as long as no payment method is linked to the SambaNova account, and the Cerebras trial credits are never topped up automatically. Keep it that way: don't add a payment method or buy credits, and set the OpenRouter key's credit limit to $0. When the free quota or trial runs out, the provider is skipped and the next free provider answers.

Does the doctor spend money?

Each test request asks for a few words. A repair tests at most 5 models, at most once per provider every 30 minutes. Healthy providers get at most one test request every 2 hours, and only when a newer free release of their model is listed.

Is every change audited?

Yes. Repairs, repairs that have to wait for a later run, pins, connection-setting changes (old and new values), new issues and issues that closed themselves are written to the platform audit log.

Do old issues close by themselves?

Yes. When the doctor repairs or verifies a provider, every open issue it raised about that provider closes, including issues from older versions of the doctor. Issues people created are never closed automatically.

Who is told about a new issue?

The platform administrators get an email when an issue is opened (not on every update), if email is set up. The issue also stays on this panel.

Troubleshooting

A provider shows Cooling down after I replaced its key

Click Run doctor now. The doctor sends a real test request straight away, even during the cool-down and even if it tested the provider a few minutes ago. If the provider answers, the cool-down ends on every server within about 15 seconds, and the audit log records AI_PROVIDER_REST_CLEARED. If it still fails, the cool-down stays and the result shows the provider's own error.

The model list in the pin dialog is empty

The list appears once the provider's models have been read. You can still type a model id.

Was this article helpful?

Related articles