Size of the raw text in the source file

[ines (Ines Montani)]

Hi! I just answered a similar question here:

Best annotation strategy for NER

Hi! Most NER implementations, including spaCy's default NER model, typically look at a very narrow context window, e.g. a few surrounding words on either side. So there's usually not an advantage in labelling the whole text at once as opposed to sentence by sentence or paragraph by paragraph. In fact, it can sometimes be counterproductive: if you design your annotation scheme so that it needs context that's very far away, you could collect data that your model might not be able to learn from. S…

The main thing that's important is that your training and runtime inputs should match. So if you're training on single pargraphs, your model should also be run on single paragraphs. But if you have control over the preprocessing, that's usually no problem.