Seeing the same issue installing spacy 2.0.0a15 on a clean install of python 3.6, reverting to 2.0.0a13 also fixed the problem.
We just pushed a new spaCy update yesterday, v2.0.0a16, which uses Thincv6.9.0 and should fix the compatibility issues. It requires the latest version of Prodigy that was released yesterday, v0.3.0, so if you install that cleanly, it should use all the latest dependencies!
Hi @ines, thank for your answer.
Do I have to install prodigy to use spacy v2.0.0a16?
Right now
pip install spacy-nightly
in a clean (conda) python 3.6 environment on MacOS install spacy v2.0.0a16 with thinc v6.9.0 but
spacy download en
still fail as above.
End of log:
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 cymem-1.31.2 cytoolz-0.8.2 dill-0.2.7.1 ftfy-4.4.3 html5lib-0.999999999 idna-2.6 msgpack-numpy-0.4.1 msgpack-python-0.4.8 murmurhash-0.28.0 numpy-1.13.3 pathlib-1.0.1 plac-0.9.6 preshed-1.0.0 regex-2017.4.5 requests-2.18.4 six-1.11.0 spacy-nightly-2.0.0a16 termcolor-1.1.0 thinc-6.9.0 toolz-0.8.2 tqdm-4.19.1.post1 ujson-1.35 urllib3-1.22 wcwidth-0.1.7 webencodings-0.5.1 wrapt-1.10.11
(analytics) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ spacy download en
Traceback (most recent call last):
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/__init__.py", line 4, in <module>
from .cli.info import info as cli_info
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/__init__.py", line 6, in <module>
from .train import train
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/train.py", line 17, in <module>
from ..tokens.doc import Doc
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/tokens/__init__.py", line 1, in <module>
from .doc import Doc
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
Ah, no – but you need spaCy v2.0.0a16 to use the latest version of Prodigy.
And this is strange – just as a sanity check, do you get the same error when you run:
python -m spacy download en
Or compare the output of both:
spacy info
python -m spacy info
If this does the trick, there might be an issue with the spacy executable that should actually only alias spacy to python -m spacy. For example, it could have ended up on the wrong Python path. (So maybe we should update the docs again to always recommend python -m as we used to, since it still seems to be the safest option, especially when working with virtual environments etc.)
Hmm all these commands gives me the same error.
Actually opening python (3.6) and trying import spacy (before the model is downloaded) also fails as
(analytics-test) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/__init__.py", line 4, in <module>
from .cli.info import info as cli_info
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/__init__.py", line 6, in <module>
from .train import train
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/train.py", line 17, in <module>
from ..tokens.doc import Doc
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/tokens/__init__.py", line 1, in <module>
from .doc import Doc
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
Tried in a clean conda environment again.
@ines, I tried in a clean env on ubuntu and it worked without any problem so I am suspecting it has something to do with MacOS (maybe in the line of this Cython issue: https://github.com/cython/cython/issues/1551 (I am a Cython newbie) ?
Do you have any recommendation in terms of MacOS setup? I use Miniconda env.
Hmm! So to summarize the chain here, we go:
.pyx source files -> Cython = .cpp files
.cpp files -> GCC/clang = .so extensions
.so extensions + python = import spacy
The .pyx to .cpp step happens on our server, and you download the generated .cpp files in the spaCy package. When you do pip install spacy-nightly, your compiler then turns them into then .so extensions, so you can import the library.
Until recently, we’ve been using a fairly old version of Cython to generate the .cpp files. I suspect that the newer version of Cython is generating code that your compiler is unhappy with. The suggestion in that issue looks like a likely solution — if we can change the compiler flag used when installing the module, that might work.
Thanks @honnibal for the Cython introduction!
So I tried to install spaCy from source by compiling the develop branch of the repo.
I switched the COMPILE_OPTIONS in setup.py from -O3 to -O0 as advised in the above github issue and also tried to remove the open_mp flags.
But that didn’t change the error messages which is still
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
Here is the part of the pip install log that relates to the compilation of morphology.pxd (I can post the full log if you want)
building 'spacy.morphology' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -I/private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -c spacy/morphology.cpp -o build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -O0 -Wno-strict-prototypes -Wno-unused-function
In file included from spacy/morphology.cpp:505:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/arrayobject.h:15:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarrayobject.h:17:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarraytypes.h:1728:
/private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/npy_deprecated_api.h:11:2: warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
^
1 warning generated.
g++ -bundle -undefined dynamic_lookup -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -o build/lib.macosx-10.7-x86_64-3.6/spacy/morphology.cpython-36m-darwin.so -Wl,-rpath,@loader_path/../spacy/platform/darwin/lib
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9
I am on MacOSX Sierra 10.12.4.
Shall I revert to your previsously used versino of cython ?
I am also having the issue with Spacy-Nightly V2.0.0A16 as reported above.
I recently attempted to upgrade to the newest version, which resulted in the errors described above. Now I can not reinstall spacy and prodigy without the following error thrown by prodigy:
/Users/crivera5/.virtual_envs/spacy_python3/bin/python: Error while finding spec for 'prodigy.__main__' (<class 'ValueError'>: thinc.extra.search.MaxViolation has the wrong size, try recompiling); 'prodigy' is a package and cannot be directly executed
I have the issue when installing prodigy 0.4.0 with spacy-nightly v.2.0.17
clean environment. python 3.6. macos 10.11.6
I get the SystemError: initialization of morphology failed without raising an exception error when executing prodigy or executing spacy or importing spacy.
We’ve got this fixed in spaCy now, so the next update will take care of the problem.
The Morphology file had a a long enum, listing out the morphological features in the Universal Dependencies schema. Recent versions of Cython cause a lot of code to be generated from this construction, which causes the error on some platforms. I’ve found an easy workaround, that also reduces our compile times a bit.
How do we fix that problem? I am having it too. Just updated spacy to current and downloaded newest version of Prodigy. It downloaded nightly. When I run that it gives me the error.
@crivera The current version of Prodigy is not yet compatible with the latest spaCy – only with spacy-nightly==v2.0.0a17. But we already have the new Prodigy wheels ready – just need to test them, then we’ll send out another update!
See this thread for more details: Installing problem with prodigy-0.4.0 on Ubuntu 16.0.4 LTS
Thanks!
I tried reinstalling with a fresh environment. I still get
mod_name, mod_spec, code = _get_module_details(mod_name)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 118, in _get_module_details
return _get_module_details(pkg_main_name)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/runpy.py", line 104, in _get_module_details
spec = importlib.util.find_spec(mod_name)
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/importlib/util.py", line 88, in find_spec
parent = __import__(parent_name, fromlist=['__path__'])
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/prodigy/__init__.py", line 4, in <module>
from . import recipes, about # noqa
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/prodigy/recipes/__init__.py", line 4, in <module>
from . import ner, textcat, compare, terms, generic # noqa
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/prodigy/recipes/ner.py", line 7, in <module>
import spacy
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/spacy/__init__.py", line 4, in <module>
from .cli.info import info as cli_info
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/spacy/cli/__init__.py", line 6, in <module>
from .train import train
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/spacy/cli/train.py", line 17, in <module>
from ..tokens.doc import Doc
File "/Users/crivera5/.virtual_envs/spacy_python3/lib/python3.5/site-packages/spacy/tokens/__init__.py", line 1, in <module>
from .doc import Doc
File "morphology.pxd", line 25, in init spacy.tokens.doc```@crivera This is a very strange error indeed 
We just sent out a new version that makes Prodigy compatible with spaCy v2.0. Could you download this one and try again?
Which did you update? Prodigy or Spacy? I will try tomorrow. Thanks!!!
@crivera Both – and Thinc 
But as I said, you don’t have to worry about any dependencies. Just download the new Prodigy wheel, install it in a fresh environment, and everything else will be installed automatically. This is also the best way to avoid installation issues and dependency mismatches.
Installation finally worked again with the newest version. Thanks!
1 Like