Hi Vincent and thank you for your quick response.
I had looked at the section mentioned, that is why I thought there might be a simpler way .
Thanks for confirming it. So If I understood correctly the tokenization will be automatically transformed during train given that I am using spacy models am I right?
PS: I am not interested in non-spacy models at least for the moment.
Thanks again for reaching out.
Regards.