Is there a way we can check point datasets and roll back if new annotations have actually deteriorated the accuracy? Currently the only method is to copy the dataset (db-out and db-in) and then keep moving between datasets. It will help to have a checkpoint and rollback feature while carefully building models.
You’ll probably want to write a recipe to give you the exact commands you need, but it should be quite easy to do. Remember that the annotations are always linked to sessions — so you should find it fairly easy to drop a session from a dataset.
Got it! It is there in the readme.
When you start annotating, Prodigy will also create a session dataset, using the timestamp as the dataset ID. This lets you view, export or discard annotations of a specific session. To see all dataset and session IDs, use the stats command with the flag -ls