Did they mix those two types of annotations in the same dataset? The
training.jsonl file will only include the data used in that particular training session. Ideally, you wouldn't want to mix binary and manual annotations in the same set, because you'd want to use the data and train / evaluate the model differently depending on the data.
There are some clues in the data that can tell you whether annotations are binary questions about the model's predictions, or manual annotations created by a human or semiautomatically by a human and a model.
Binary annotations: Only ever have one entry in the
"spans" (the entity to collect feedback on) and if the suggestion comes from the model, the task's
"meta" typically contains the
"score". In Prodigy v1.8+, tasks also have a
"_view_id" storing the name of the annotation interface that was used. This should be
Manual annotations: Can have any number of
"spans" and also contain
"tokens" (because the text is pre-tokenized for easier highlighting). The
"_view_id", if present, would typically be