rel.manual not accepting entities because of tokenization


I'm trying to relate some pre-annotated entities. However, when I feed a dataset with them to rel.manual:
prodigy rel.manual test_dataset_rel en_core_web_sm dataset:test_dataset -l SUBJECT
it throws a bunch of few exceptions like this:
⚠ Skipped 27 span(s) that were already present in the input data because the tokenization didn't match.

And indeed, it drops most of the labels, especially (but not only) the ones that are several tokens long, e.g.:

I wonder what might be the cause.

Hi! In general, the idea here is that annotations should always map to valid token boundaries because you'll be making predictions over tokens later, and any mismatch means that the model can't easily be updated with the annotation.

However, it looks like in this case, the token boundaries are fine, so I wonder if this is related to whitespace? Maybe double-check the underlying JSON and check that the entity span doesn't include the trailing space? It could be as simple as an off-by-one character index on the last span.