# whitespaces at the beginning of a line

**URL:** <https://support.prodi.gy/t/whitespaces-at-the-beginning-of-a-line/4739>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy\
**Created:** [September 29, 2021, 9:36am UTC](https://support.prodi.gy/t/whitespaces-at-the-beginning-of-a-line/4739 "2021-09-29T09:36:18Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 4, 2021, 9:37am UTC](https://support.prodi.gy/t/whitespaces-at-the-beginning-of-a-line/4739/2 "2021-10-04T09:37:21Z")

</div>

Hi! I just tested this and I think the underlying problem here is that the default tokenizer currently produces a single whitespace token for `\n` and trailing spaces. So the tokens you end up with under the hood look like this:

```python
nlp = spacy.blank("en")
doc = nlp("hello dear\n world")
print([t.text for t in doc])
# ['hello', ' ', 'dear', '\n ', 'world']

```

There are a few different ways you could work around it and it kinda depends on the rest of your process:

- Provide a list of `"tokens"` yourself that includes two separate tokens for `\n` and the preceeding spaces. You could use Prodigy's `add_tokens` preprocessor to add the default tokens and then adjust them automatically in a postprocessing step (just make sure to adjust the token indices when you split the existing token).
- Probably a bit more elegant: add a rule/component that runs after the tokenizer and uses `doc.retokenize` to split all tokens consisting of `\n` + spaces into two: [https://spacy.io/api/doc#retokenizer.split](https://spacy.io/api/doc#retokenizer.split)
- Hacky: Add an invisible/zero-width space (`\u200b`) in between. You'd probably just want to replace it again afterwards with a simple search and replace over the data.

---

_[View the full topic](https://support.prodi.gy/t/whitespaces-at-the-beginning-of-a-line/4739)._