Saving annotations, measuring time taken per instance and remove skip option

, ,
1

There are a couple of topics I had problems with:

  1. Saving the annotations if the user quits from the browser
  2. Measuring user statistics such as time taken per instance and total time per session.
  3. How to keep only reject and accept and removed skip from the given options

TIA

2

Before exiting the browser, the user should hit "Save". Prodigy will keep the most recent annotations in the browser only and sends the back to the server in batches. This allows an easy way of undoing / reannotating, because the server doesn't have to reconcile different versions. But it also means that if the user just closes the browser, the last annotations may not be saved. Once the browser is closed, it obviously can't communicate with Prodigy anymore.

You can use the batch_size setting to set a lower batch size and send the answers back more frequently. Or you set instant_submit to always send on answer (and not let the user undo).

You could log the current timestamps when the server starts, in the recipe's update callback (when batch of answers is received) and also add it to each outgoing example so you know exactly when it's sent out. Then you could also write timestamps to the annotation tasks on the client using the window.prodigy.update function (for instance, every time the prodigyanswer event is fired) – e.g. as the "client_timestamp" field. This should give you a rough estimate of what was done when.

(Just make sure to account for differences in the server time and client time – otherwise you'll end up with inaccurate stats!)

You can always hide a button using the global_css setting if you want to. I wrote up an example here: Custom multilabel categorization recipe - #9 by ines

3

Thanks for the prompt response Ines, much appreciated! These are solid answers, also should I use instant_submit in the return dict object?

4

Thanks and sorry if that was unclear. The config settings should all go in the "config" returned by the recipe – for instance:

return {
    "dataset": dataset,
    "stream": stream,
    # etc...
    "config": {
        "instant_submit": True,
        # and so on
    }
}
5

Great! That was really helpful :slight_smile: