Use custom tokenizer in data-to-spacy recipe

[alvaro.marlo (Álvaro Martínez López)]

I am developing a model with spaCy using data from prodigy. I'm using a custom tokenizer and I want this new spaCy data with specific tokenization. Is there any chance to run the data-to-spacy command specifying the tokenizer from an external file?

[ryanwesslen]

Hi @alvaro.marlo!

@alvaro.marlo

I am developing a model with spaCy using data from prodigy. I'm using a custom tokenizer and I want this new spaCy data with specific tokenization. Is there any chance to run the data-to-spacy command specifying the tokenizer from an external file?

Yes, it seems like you can pass the -F for a script with a tokenizer function. It seems to work even though it wasn't originally designed to do this:

Use custom tokenizer in data-to-spacy

Ok, thanks! That explains it. Like Ines said, the -F wasn't originally designed for this, but it does work in this context. I think it'll be useful/consistent to also support the --code flag for these cases, as spaCy does.

Hope this helps!