Monitoring multiple user sessions

Hi! I have several prodigy annotation tasks. Each unique task is on a separate port. For each task there can be several annotators.

Can I somehow monitor all users sessions? I would like to see which tasks run on which ports. For each port I would like to see users that annotate this task. And for each user I would like to see how much data they annotated and how much is left.

If you're running separate REST APIs on different ports, there's not reall an easy way for them to know about each other – but you can easily log this information somewhere when you start the individual processes. For example, log the port and the full command used to start the server.

Named sessions can be created and added at runtime, and Prodigy doesn't track user behaviour in the browser. But if you know the allowed session names and you're limiting the allowed session names (via the PRODIGY_ALLOWED_SESSIONS environment variable), you can always check the database for the session dataset to retrieve all annotations created by the named session. For example, dataset_name-user will include all annotations created by the named session user for a task writing to the dataset dataset_name.

This also lets you calculate how much is left – how you do that depends on how the goal is defined. If your goal is to annotate all examples in your source data, and you know the length, that's pretty easy – if you're working with an infinite stream and/or an active learning strategy that selects and skips exmples, you probably want to calculate this differently (e.g. based on the loss after updating your model from the annotations).

1 Like