Custom Tokenization Support for Spacy (and by extension Prodigy).

[ines (Ines Montani)]

For reference, I think this might have already been solved here:

Off-track use of Prodigy/Spacy - Custom Regex Pattern Matching and Modeling

Slightly modified your suffixes to include the , in the character class (between [ and ]) and it now splits the , off correctly: suffix_re = re.compile(r'''[\]\)"',]$''')