feature request: virtual models

[iggy]

It would be nice to support virtual models that routes to multiple different models.

A few examples:

• openrouter's free model
• opus-4.6 that routes to anthropic/opus-4.6 -> openrouter/opus-4.6 -> bedrock/opus-4.6
• opus-latest that you can switch to Opus 4.8 when it comes out

[SantiagoDePolonia]

Check out the aliases feature (Dashboard->Models->Right-Top corner). You can’t do load balancing with aliases, but everything else is possible.

Also, failover/fallback feature might be helpful here. Check the docs.

Is that enough for your use cases?

[iggy]

The alias feature doesn't really do what I'm after. Unless I'm missing something, it's 1:1 and I'm looking for 1:many. It would technically solve the last example I mentioned, but that's not the one I'm most interested in.

[SantiagoDePolonia]

What do you mean by 1:many? What response do you expect when multiple models are called? Which one should be returned? Should they be called one-by-one or parallel?

By 1:many, do you mean “when model A is not responding or is no longer available, make a call to model B”?

If so, the failover and alias features might be helpful here: https://gomodel.enterpilot.io/docs/features/failover. Am I correct?

Failover is not configurable from the UI right now. Would UI-based configuration be helpful for you?

[iggy]

I tried the failover/fallback functionality to see if it could do what I wanted, but I may be doing it wrong.

I tried a fallback.json that looks like below and neither of the top-level model names (free-tiny/opus-latest) show up in the /models output. Not sure if that is intended or not, but some tools learn the available models from that output. I tried to set it up in one of my tools that doesn't introspect /models and it says unsupported model: free-tiny.

{
  "free-tiny": [
    "fwd/google/Gemma4-2b-Q4",
    "openrouter/liquid/lfm-2.5-1.2b-thinking:free",
    "openrouter/nvidia/nemotron-3-nano-30b-a3b:free",
    "openrouter/openrouter/free"
  ],
  "opus-latest": [
    "mantle-use1/anthropic.claude-opus-4-8",
    "openrouter/~anthropic/claude-opus-latest"
  ]
}

What do you mean by 1:many? What response do you expect when multiple models are called? Which one should be returned? Should they be called one-by-one or parallel?

For my needs going through them in order until one fails would be preferable (because I generally put the cheaper ones first), but I could see where having some options there would be useful (i.e. round-robin, random, budget driven, etc). Parallel is an interesting thought, but I can't see how that would work with the existing openai api spec.

By 1:many, do you mean “when model A is not responding or is no longer available, make a call to model B”?

Yeah, for my use case, that's what I'm after, although I touched on some other options in my previous answer.

Failover is not configurable from the UI right now. Would UI-based configuration be helpful for you?

If it works, I'm fine with the config file. Although anything that doesn't require restarting the service after a change gets a 👍 in my book.

[iggy]

This didn't exist when I opened this issue, but I think #352 covers all the use cases I mentioned above. If that gets merged, I think it'll solve this issue.

[SantiagoDePolonia]

@iggy
Good news!

Today I've shipped load balancing and improved the newly introduced virtual models feature. Please check it out. You can configure it in the dashboard OR in the config level.

One caveat - the feature you (and a few folks) requested, was implemented as load balancing with Round robin algorithm. It means that the requests are redirected in order. In the case of 2 models
1st request -> 1st model
2nd request -> 2nd model
3rd request -> 1st model etc

It also means that you might lose the benefits related to prompt caching.

Let me know if that's what you meant.

[SantiagoDePolonia]

At the latest version 0.1.47, also failover feature was improved. Now you can set up a few models which should be used once the requested model is unavailable.

I'm closing the issue. Let me know if you have any other feature request or suggestion.