spancat with really large spans? (Identify sections in text)

[koaning (Vincent D Warmerdam)]

I found a dialogue on the forum that might be inspirational here.

Combining NER with text classification

@wpm If you segment the document into paragraphs, you can run the NER over all the paragraphs though, right? I think it makes good sense to use the text classifier during training to find paragraphs with a high enough density of entities to make your annotation effort productive. But at runtime, where you just want the tool to extract entities, you may as well run it over the whole text. As far as doing joint learning goes: there are a few ways you can do this. One solution would be to share t…

It's a different problem, but it highlights another two-step approach to rethinking spans.

That said, reading your reply still makes me think that textcat might be the simplest way forward, albeit on paragraphs instead of sentences. While I like your idea of using NER to detect the start of a section, I wonder if you might be able to leverage that this always starts on a newline, which suggests a heuristic might be better than a ML model.