Bug when trying to review previous work (Prodigy 1.10.5)
Startup command was
prodigy review X X --label SOMETHING --view-id dep
Hi! Can you share a bit more about the dataset and data format, and how it was created (e.g. which recipe you used and what the JSON looks like in the dataset)?
Good morning Ines.
The dataset is very basic. I annotated a few items similar to the following using rel.manual blank:en. I have since dropped the data set from the db as I could not find a good way to review annotations from before the session started. Perhaps I am missing something there (new user)?
The dataset was minimal, maybe 10 items in JSONL format.
{"text": "Some text of a few words."}
...
Some items I completed annotating and I skipped a couple.
- The History in the side panel is good to have but it would be nice to have groups that could be recalled based on an executing session of the tool (ie, I open Prodigy, annotate 10 items, and close. Then next time I open I look at a review tab which has the group (named or timestamped group) of 10 from the previous session, and sessions further back. They could be lazily loaded. This would be good for when I am still forming an idea of how I want to annotate the data and may decide the labels should change.
Ah, so one problem here is that you created the annotation with the relations interface, but set --view-id dep during review, so Prodigy was trying to render the annotations with a different interface that expects a different data format. (It could have failed more gracefully, though!) For the standard workflow, you shouldn't have to set a --view-id for reviewing – that information is stored with each task, so Prodigy knows how to render it again.
The review recipe is designed for reviewing annotations created by multiple annotators and to help resolve conflicts to create one final annotation per example. If you just want to go through your own annotations again, it's probably easier to load in your existing dataset again and save the results to a new set. For example:
prodigy rel.manual reviewed_dataset blank:en dataset:previous_dataset [...]
The history in the sidebar currently shows you the latest 10 annotations from the current session that haven't yet been saved to the database and that can still be edited in the UI as you annotate. Datasets are append-only so if you load back annotations from a previous session and save them to the same dataset, you'll end up with two data points. (Or maybe I'm misunderstanding your question?)
Append-only. Got it. I see how that would be difficult then.
The use case I was thinking of was if I had labeled 250 statements but realized say 20 of the entity categorizations could be adjusted to something more appropriate (say I had 50 items labeled PLACE and determined instead they should be CITY or COUNTRY), it would be nice to be able to quickly find the annotations with PLACE and cycle through only those for reannotation. Perhaps there could be a method to copy the db to a temporary one, save all annotations except ones with CITY, and serve the CITY ones (starting with previous markup) for updating.
Maybe something like:
prodigy --redo PLACE ner.manual my_dataset blank:en dataset:my_dataset