Preserve preceding whitespaces at the beginning of a line

[kerenya (keren)]

Hi, sorry if this is a duplicate but I couldnt find any answer to what I look for
How can I preserve spaces at the beginning of a new line , my text contains lines that begin in different positions within the line and I need to preserve the positions (for correctness)
Is there a way to preserve those sapces?
Thanks
Keren

[ines (Ines Montani)]

Hi! Ultimately, this comes down to how the incoming texts and whitespace are tokenized: by default, spaCy splits on whitespace but preserves multiple whitespace characters as whitespace tokens. Prodigy itself doesn't really have an opinion on whitespace and will just render whatever it gets via the tokens.

A similar question came up in this thread recently and I shared some background and suggestions for custom whitespace tokens:

whitespaces at the beginning of a line

Hi! I just tested this and I think the underlying problem here is that the default tokenizer currently produces a single whitespace token for \n and trailing spaces. So the tokens you end up with under the hood look like this: nlp = spacy.blank("en") doc = nlp("hello dear\n world") print([t.text for t in doc]) # ['hello', ' ', 'dear', '\n ', 'world'] There are a few different ways you could work around it and it kinda depends on the rest of your process: Prov…