Annotators Performance Tracker

Hi, Prodigy team :). I wonder if there is out of a box feature that tracks annotators' performance over time, i.e. by month. Basically, we want to have analytical tools to see the annotators' progress of the tasks so that we can decide additional annotators are needed.

If this is not provided, can you suggest how we can extend prodigy so that having such analytical tools can be implemented? can we somehow save and retrieve the timestamp when an annotation is saved?

Thank you :smiley:

Hi! Timestamps for each annotation as it's submitted in the UI is a feature coming to Prodigy v1.11 – we'll be releasing a new nightly soon that will already have it :tada:

In the meantime, here are some ideas and code snippets you can use to add it yourself: Feature request: timestamps for data entry - #5 by ines Quick note: you probably want to use Math.floor(new Date().getTime() / 1000) for the timestamp so you can easily use and convert it in Python.

There's no direct feature that outputs the progress over time, but if you have the timestamps, you should be able to easily calculate that, depending on your task, the datasets you're tracking and how you define "progress". Using the Database API in Python you can load all examples from one or more datasets, get their timestamps and then group them by timestamp rounded to week, month, year or whatever else you're interested in. You can then output the number of annotations created in the given time period, or a cumulative sum of all annotations, which is more like a classic "progress" diagram that's always going up. Or you could look for other patterns if you want, like most active day of the week, most active hour of the day etc :smiley:

Just released Prodigy v1.11, which now automatically adds a _timestamp key to all examples! We've also added a new progress command that calculates the annotation progress over time for different intervals, using the _timestamp if available (and the dataset creation time as a fallback): https://prodi.gy/docs/recipes#progress