# UnicodeDecodeError while training japanese model **URL:** \ **Category:** Uncategorized\ **Tags:** usage, spacy\ **Created:** [January 26, 2019, 8:57am UTC](https://support.prodi.gy/t/unicodedecodeerror-while-training-japanese-model/1158 "2019-01-26T08:57:11Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![KavyaGujjala](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/kavyagujjala/32/606_2.png) [@KavyaGujjala](https://support.prodi.gy/u/KavyaGujjala)\ **Post date:** [January 26, 2019, 8:57am UTC](https://support.prodi.gy/t/unicodedecodeerror-while-training-japanese-model/1158/1 "2019-01-26T08:57:11Z")
Hi, I wanted to train a japanese blank model using ner.manual command. But I am getting encoding error . Does anything have to exported like you have mentioned for english language model? command used `prodigy ner.manual test jap_model_vm_1 out1.txt --label ORG` out1.txt file have japanese text Error looks like ``` Using 1 labels: ORG Traceback (most recent call last): File "/usr/local/conda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main " __main__", mod_spec) File "/usr/local/conda3/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/conda3/lib/python3.6/site-packages/prodigy/ __main__.py", line 259, in controller = recipe(*args, use_plac=True) File "cython_src/prodigy/core.pyx", line 178, in prodigy.core.recipe.recipe_decorator.recipe_proxy File "cython_src/prodigy/core.pyx", line 55, in prodigy.core.Controller. __init__ File "/usr/local/conda3/lib/python3.6/site-packages/toolz/itertoolz.py", line 368, in first return next(iter(seq)) File "cython_src/prodigy/core.pyx", line 84, in iter_tasks File "cython_src/prodigy/components/preprocess.pyx", line 107, in add_tokens File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 117, in make_doc return self.tokenizer(text) File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 79, in __call__ dtokens = detailed_tokens(self.tokenizer, text) File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 60, in detailed_tokens parts = node.feature.split(',') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte ``` What can be the reason? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [January 26, 2019, 11:53pm UTC](https://support.prodi.gy/t/unicodedecodeerror-while-training-japanese-model/1158/2 "2019-01-26T23:53:28Z")
This usually indicates that the encoding of the text file isn’t valid utf-8 (unicode). Could you try explicitly chaning the encoding to `utf-8`? For example from the command line using a tool like [iconv](https://en.wikipedia.org/wiki/Iconv) or in your text editor (e.g. in Visual Studio Code: Change encoding \> UTF-8). ---
**Author:** ![KavyaGujjala](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/kavyagujjala/32/606_2.png) [@KavyaGujjala](https://support.prodi.gy/u/KavyaGujjala)\ **Post date:** [February 6, 2019, 5:03am UTC](https://support.prodi.gy/t/unicodedecodeerror-while-training-japanese-model/1158/3 "2019-02-06T05:03:37Z")
Hi, Thanks for the reply. Tried using iconv as you said even then the same error is coming up. I have trained with 1500 sentences of japanese text on blank model and using it in prodigy as base model. command used: `prodigy ner.teach jap_dataset_try_1 japanese/models/jap_model_5_iter_drop_0.1 japanese/train/jap_sentences.txt` ``` Traceback (most recent call last): File "/usr/local/conda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main " __main__", mod_spec) File "/usr/local/conda3/lib/python3.6/runpy.py", line 85, in _run_code exec(code, run_globals) File "/usr/local/conda3/lib/python3.6/site-packages/prodigy/ __main__.py", line 259, in controller = recipe(*args, use_plac=True) File "cython_src/prodigy/core.pyx", line 178, in prodigy.core.recipe.recipe_decorator.recipe_proxy File "cython_src/prodigy/core.pyx", line 55, in prodigy.core.Controller. __init__ File "/usr/local/conda3/lib/python3.6/site-packages/toolz/itertoolz.py", line 368, in first return next(iter(seq)) File "cython_src/prodigy/core.pyx", line 84, in iter_tasks File "cython_src/prodigy/components/sorters.pyx", line 136, in __iter__ File "cython_src/prodigy/components/sorters.pyx", line 51, in genexpr File "cython_src/prodigy/models/ner.pyx", line 265, in __call__ File "cython_src/prodigy/models/ner.pyx", line 233, in get_tasks File "cytoolz/itertoolz.pyx", line 1047, in cytoolz.itertoolz.partition_all. __next__ File "cython_src/prodigy/models/ner.pyx", line 192, in predict_spans File "cytoolz/itertoolz.pyx", line 1047, in cytoolz.itertoolz.partition_all. __next__ File "cython_src/prodigy/components/preprocess.pyx", line 36, in split_sentences File "/usr/local/conda3/lib/python3.6/site-packages/spacy/language.py", line 548, in pipe for doc, context in izip(docs, contexts): File "/usr/local/conda3/lib/python3.6/site-packages/spacy/language.py", line 572, in pipe for doc in docs: File "nn_parser.pyx", line 367, in pipe File "cytoolz/itertoolz.pyx", line 1047, in cytoolz.itertoolz.partition_all. __next__ File "/usr/local/conda3/lib/python3.6/site-packages/spacy/language.py", line 746, in _pipe for doc in docs: File "/usr/local/conda3/lib/python3.6/site-packages/spacy/language.py", line 551, in docs = (self.make_doc(text) for text in texts) File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 117, in make_doc return self.tokenizer(text) File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 79, in __call__ dtokens = detailed_tokens(self.tokenizer, text) File "/usr/local/conda3/lib/python3.6/site-packages/spacy/lang/ja/ __init__.py", line 60, in detailed_tokens parts = node.feature.split(',') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 2: invalid start byte Exception ignored in: SystemError: returned a result with an error set ``` I got this error. Can you please help me with this? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 6, 2019, 10:24am UTC](https://support.prodi.gy/t/unicodedecodeerror-while-training-japanese-model/1158/4 "2019-02-06T10:24:15Z")
It looks like there's a problem with pickling the external library that's optional for Japanese tokenization. So this is more something in spaCy. See here for details: > > > I am facing this Japanese NER issue: > I am using japanese blank model with Mecab > i used ner.manual recipe > prodigy ner.manual dataset\_test\_1 path/to/txt/file --label...