Wrong tokenization on commas preceded by a special character

,
1

Hi!
I've noticed that when using the default tokenizer (from the en_core_web_sm model) I got wrong tokenization in sentences like the following:

Required languages: java,c#,javascript.

This gives the following tokens: Required languages : java , c#,javascript .

This does not happen with the following case:

Required languages: java,python,javascript.

Tokens: Required languages : java , python , javascript .

The problem seems to occur with any comma preceded by a special characte, e.g.:

java,c++,javascript -> java , c++,javascript
java,F#,javascript -> java , F#,javascript
java,XyZ@,javascript -> java , XyZ@,javascript

I've tried to add special cases to the tokenizer but it didn't make any difference:

import spacy
nlp = spacy.load("en_core_web_sm")
nlp.tokenizer.add_special_case(u"C#", [{ORTH: u"C#"}])
nlp.tokenizer.add_special_case(u"c#", [{ORTH: u"c#"}])
doc = nlp("Required languages: java,c#,javascript.")
print([t.text for t in doc])

Any advice on how to fix this tokenization issue?

2

Hi! I think what you want to do here is add the , to the infixes, since you want the tokenizer to additionally split on commas within a string. (Your special case rules only tell it to preserve strings, which it already does). I just tired it out locally and the following worked for me:

infixes = nlp.Defaults.suffixes + (r",",)
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer
print([t.text for t in nlp("Required languages: java,python,javascript.")]                                                        
# ['Required', 'languages', ':', 'java', ',', 'python', ',', 'javascript', '.']
1 Like
3

Thanks! I've tried with your solution, now it splits correctly on subtokens commas, but still the tokenization is incorrect because it treats the '#' character as a unique token:

58
Screenshot 2019-10-04 at 13.44.58.png1802×476 65.4 KB

I've tried, according to the documentation, to add a token_match pattern for the programming languages like C# and F# that end with a # character in order to override the punctuation rules, but still no effect:

39
Screenshot 2019-10-04 at 13.48.39.png1714×500 76.8 KB
Screenshot 2019-10-04 at 13.48.39

What am I missing?

4

I guess you could remove the # from the infixes?

infixes = nlp.Defaults.infixes + (r",",)
infixes = tuple([i for i in infixes if i != "#"])
infix_regex = spacy.util.compile_infix_regex(infixes)
nlp.tokenizer.infix_finditer = infix_regex.finditer
print([t.text for t in nlp("Required languages: java,python,c#,javascript")])
# ['Required', 'languages', ':', 'java', ',', 'python', ',', 'c#', ',', 'javascript']
1 Like
5

That did the trick, thanks Ines!

Just a curiosity: why we are loading the default suffixes as the new infixes and not doing something like:

infixes = nlp.Defaults.infixes + (r",",)

The above didn't work, only the one with nlp.Defaults.suffixes worked, just wondering why :grin:

Thank you again for the help!

6

Sorry, that was a typo! :woman_facepalming: I just fixed it above.