Segmenting examples with long spans as NERs

[honnibal (Matthew Honnibal)]

Hi @pvcastro,

As I mentioned in the last thread, I'm suspicious of using the entity recognised for these long spans. I think you should try applying sentence labels, and perhaps also marking words which are important for the category you're interested in. Then you can use the dependency parse to find the claim boundaries. You can find documentation about the dependency parser here: Linguistic Features · spaCy Usage Documentation

@pvcastro

But my actual problem is that if I don’t use the ‘unsegmented’ parameter, the recipe is losing the spans for each example after segmenting them. My dataset of 400 examples goes to a few thousand sentences, but only about 11 of these have the original spans, all the other thousands have empty spans. Do you know what could be the problem?

There should be as many spans, whether you set unsegmented or not --- unless the spans cross segmentation boundaries. This sounds like it might be a bug; we'll look into it. At first glance the segmentation function looks correct, and it's passing our tests. But I'll play around with your sample and see if I can find the problem.