Skip to main content

Model Groups and Routing

A model group is an entry in the Model Library that has no model of its own. Each turn, it sends the request to one of its member models, and if that member hits a capacity problem it fails over to another. Agents use a group exactly like a model.

Use a group to:

  • Spread traffic across several deployments of the same model (for example two Azure regions).
  • Prefer one model and fall back to another when it's rate-limited.
  • Keep agents running through a provider's short outages.

Create a group​

  1. Go to Resources → Models.
  2. Click the New model group button at the top of the Model Library.
  3. Enter a Group Name and click Create.
  4. Open the group's Members tab.

Add members​

  1. In Add member model, choose a model and click Add. Members must be ordinary models, not folders or other groups.
  2. Set each member's:
    • Weight: relative share of traffic for the weighted strategy (default 1).
    • Priority: preference for the priority strategy; lower numbers are tried first (default 100).
    • Enabled: disabled members are skipped.
  3. Add at least two members. A group with one member can't fail over, and the page warns you.

Configure routing​

SettingOptions and defaultEffect
StrategyWeighted (random, by weight) (default), Priority (lowest number first), Round-robinHow a member is picked for each turn.
Fail over onRate limit (429), Server error (5xx), Timeout (all on by default)Which errors move the turn to another member. Authentication errors, bad requests and tool-call errors never fail over, because another member wouldn't fix them.
Max failover hops / turn1–20, default 3How many times one turn may fail over.
Default cooldown (s)0–3600, default 60How long a failed member is skipped when the provider doesn't say.
Sticky (stay on last-good member)On by defaultKeep using the member that last worked until it fails.
Honor Retry-AfterOn by defaultUse the provider's Retry-After header for the cooldown.
Aggressive mid-stream failoverOff by defaultWhen on, a turn whose member errors part-way through a streamed response is restarted on the next member. When off, the error is shown and failover happens on the next turn.

Click Save Group.

Check the group​

The group's Definition tab shows Routes to: the members in order with their deployment, credential mode, priority and weight, and the routing strategy. A group with no members is flagged: every agent using it would fail. Use Run preflight to check that the configuration loads, and Consumers to see which agents use it.

Use the group​

Select the group in an agent's LLM Configuration field, just as you would a single model.

tip

Each member keeps its own credential and context window, so give members similar context lengths if an agent's prompts are large.