No start and end of span using data-to-spacy after rel.manual

, , ,
2

Hi! Do you know which version of Prodigy you created the annotations with? The same version you're currently using? It seems like for some reason, you ended up with an invalid span here.

The easiest way to find and exclude it would be to just go over your data, check the spans and if they include a start/end and only keep the valid spans in a new dataset:

from prodigy.components.db import connect

db = connect()
examples = db.get_dataset("myDataset2")
filtered_examples = []
for eg in examples:
    if "spans" in eg:
        new_spans = []
        for span in eg["spans"]:
            if "start" not in span or "end" not in span:
                print("Found bad span:", span)
            else:
                new_spans.append(span)
        eg["spans"] = new_spans
    filtered_examples.append(eg)

# Add filtered examples to new dataset
db.add_dataset("myDataset2_filtered")
db.add_examples(filtered_examples, ["myDataset2_filtered"])
2 Likes