Preserve preceding whitespaces at the beginning of a line

[ines (Ines Montani)]

Hi! Ultimately, this comes down to how the incoming texts and whitespace are tokenized: by default, spaCy splits on whitespace but preserves multiple whitespace characters as whitespace tokens. Prodigy itself doesn't really have an opinion on whitespace and will just render whatever it gets via the tokens.

A similar question came up in this thread recently and I shared some background and suggestions for custom whitespace tokens:

whitespaces at the beginning of a line

Hi! I just tested this and I think the underlying problem here is that the default tokenizer currently produces a single whitespace token for \n and trailing spaces. So the tokens you end up with under the hood look like this: nlp = spacy.blank("en") doc = nlp("hello dear\n world") print([t.text for t in doc]) # ['hello', ' ', 'dear', '\n ', 'world'] There are a few different ways you could work around it and it kinda depends on the rest of your process: Prov…