At our company, we expect to run multiple projects (and teams) concurrently, for example, product A and Product B, run evals at the same time. Some of the Eval SMEs are involved in both projects/products. At some point we want to add an Eval for project C. The key is that we can quick setup a new product/project, assign (existing) SMEs, without having to redeploy/operate a lot of new software. What is the best practice to support this using Prodigy?
In particular, deploying a new server for each Prodigy instance would require a lot of approvals, so we ideally want to operate a single server that directs annotators to the appropriate evaluation run.
It's possible to run multiple Prodigy instances on the same server by specifying a different port for each.
I think in your case the easiest solution would be to implement a simple service (e.g. FastAPI) with a web interface (could be as simple as rendering a jinja template) that would act as router/manager for all Prodigy instances. It would have endpoints for:
starting and stopping Prodigy services (this should make adding new service - like your service C - simple if you program it to accept configuration in yml, via web form or similar). You might want to run each prodigy service in a separate subprocess or container for isolation.
redirecting SMEs to the right Prodigy service (when SMEs click on a project from their assigned projects, they would be redirected to a correct Prodigy instance)
you could also add an extra "project manager" endpoints for viewing progress, handling SMEs assignment etc.
The details will of course depend on your stack and infrastructure but we're happy to support as you go!