Skip to main content

Make it so when a model is overloaded enter a queue rather than error?

When a model is overloaded Venice will display an error stating the model is overloaded try again later or something like that. 

trying a few seconds later usually the prompt gets thru but sometimes will see the error many times requiring the user to try and submit the prompt again or switching model to one that is not overloaded. 

Could the behavior instead be that if a model is overloaded show a warning and enter a prompt queue that is sequential so the prompt eventual gets processed if the user waits instead of playing a game of roulette where many users try spamming their prompt to the overloaded model to get it processed?

Have the option to exit this queue and switch models as well

Log in to comment and vote

No comments yet

Be the first to share your thoughts.