How to review all the annotated tasks?

1

I want to see all the tasks that have been annotated instead of only ten history tasks. How can I achieve this?

1 Like
2

Hi! There are several ways you can do this, depending on what your goal is:

  1. Use the review recipe. This is especially useful if you want to review and reannotate data and resolve conflicts, e.g. if the same examples have been annotated multiple times and the annotators disagreed. At the end of it, you'll get one final master dataset with the merged annotations.
  2. Use db-out to export all annotations in the data to a file. This give you a JSONL file with one annotation per line that you can use or any other process, or load back into Prodigy.
  3. Use print-dataset to show all annotations in a given dataset in your terminal, with nice formatting. This is especially useful if you want to get a quick overview of the data.
1 Like
3

All these strategies can only be performed after annotating. However, what I want is to review all the previous annotation while I was annotating. Because some annotators might recall that the previous annotation has flaws when he/she is annotating on the current task.

4

The history is limited to 10 examples, because those are the answers that have not yet been sent to the server. They're kept on the client so the annotator can easily hit undo, and without requiring Prodigy to reconcile two annotations later (the original one and the edited answer). It also means that datasets can be "append-only" (see here for some background on this).

We could introduce a setting to change the number of examples kept in the history, but this would mean that you'd delay sending the answers, which isn't that useful. So during a single annotation session, you can currently only undo the most recent 10 examples.

1 Like