Make it so when a model is overloaded enter a queue rather than error?

When a model is overloaded Venice will display an error stating the model is overloaded try again later or something like that.ย 

trying a few seconds later usually the prompt gets thru but sometimes will see the error many times requiring the user to try and submit the prompt again or switching model to one that is not overloaded.ย 

Could the behavior instead be that if a model is overloaded show a warning and enter a prompt queue that is sequential so the prompt eventual gets processed if the user waits instead of playing a game of roulette where many users try spamming their prompt to the overloaded model to get it processed?

Have the option to exit this queue and switch models as well

Please authenticate to join the conversation.

Upvoters
Status

New Submission

Board
๐Ÿ’ก

Feature Requests

Tags

New Model

Date

9 days ago

Subscribe to post

Get notified by email when there are changes.