What version of SpaCy does one need to use to run prodigy. I’ve been trying to install the newest version of SpaCy-nightly in a clean python 3.5 environment.
When importing I get the following in ipython:
In [1]: import spacy
---------------------------------------------------------------------------
SystemError Traceback (most recent call last)
<ipython-input-1-76a01d9c502b> in <module>()
----> 1 import spacy
~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/__init__.py in <module>()
2 from __future__ import unicode_literals
3
----> 4 from .cli.info import info as cli_info
5 from .glossary import explain
6 from .deprecated import resolve_load_name
~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/cli/__init__.py in <module>()
4 from .package import package
5 from .profile import profile
----> 6 from .train import train
7 from .convert import convert
8 from .model import model
~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/cli/train.py in <module>()
12 from timeit import default_timer as timer
13
---> 14 from ..tokens.doc import Doc
15 from ..scorer import Scorer
16 from ..gold import GoldParse, merge_sents
~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/tokens/__init__.py in <module>()
----> 1 from .doc import Doc
2 from .token import Token
3 from .span import Span
4
5 __all__ = [Doc, Token, Span]
morphology.pxd in init spacy.tokens.doc()
SystemError: initialization of morphology did not return an extension module
I’ve tried to remake a virtual environment several times.
Thanks
Thanks for the report – this is strange. Will investigate!
In the meantime, could you try and see if downgrading to spacy-nightly v2.0.0a13 helps? (This version is also compatible with the latest version of Prodigy).
We just pushed a new spaCy update yesterday, v2.0.0a16, which uses Thincv6.9.0 and should fix the compatibility issues. It requires the latest version of Prodigy that was released yesterday, v0.3.0, so if you install that cleanly, it should use all the latest dependencies!
Do I have to install prodigy to use spacy v2.0.0a16?
Right now
pip install spacy-nightly
in a clean (conda) python 3.6 environment on MacOS install spacy v2.0.0a16 with thinc v6.9.0 but
spacy download en
still fail as above.
End of log:
Successfully installed certifi-2017.7.27.1 chardet-3.0.4 cymem-1.31.2 cytoolz-0.8.2 dill-0.2.7.1 ftfy-4.4.3 html5lib-0.999999999 idna-2.6 msgpack-numpy-0.4.1 msgpack-python-0.4.8 murmurhash-0.28.0 numpy-1.13.3 pathlib-1.0.1 plac-0.9.6 preshed-1.0.0 regex-2017.4.5 requests-2.18.4 six-1.11.0 spacy-nightly-2.0.0a16 termcolor-1.1.0 thinc-6.9.0 toolz-0.8.2 tqdm-4.19.1.post1 ujson-1.35 urllib3-1.22 wcwidth-0.1.7 webencodings-0.5.1 wrapt-1.10.11
(analytics) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ spacy download en
Traceback (most recent call last):
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 183, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 142, in _get_module_details
return _get_module_details(pkg_main_name, error)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/runpy.py", line 109, in _get_module_details
__import__(pkg_name)
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/__init__.py", line 4, in <module>
from .cli.info import info as cli_info
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/__init__.py", line 6, in <module>
from .train import train
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/train.py", line 17, in <module>
from ..tokens.doc import Doc
File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/tokens/__init__.py", line 1, in <module>
from .doc import Doc
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
Ah, no – but you need spaCy v2.0.0a16 to use the latest version of Prodigy.
And this is strange – just as a sanity check, do you get the same error when you run:
python -m spacy download en
Or compare the output of both:
spacy info
python -m spacy info
If this does the trick, there might be an issue with the spacy executable that should actually only alias spacy to python -m spacy. For example, it could have ended up on the wrong Python path. (So maybe we should update the docs again to always recommend python -m as we used to, since it still seems to be the safest option, especially when working with virtual environments etc.)
Hmm all these commands gives me the same error.
Actually opening python (3.6) and trying import spacy (before the model is downloaded) also fails as
(analytics-test) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/__init__.py", line 4, in <module>
from .cli.info import info as cli_info
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/__init__.py", line 6, in <module>
from .train import train
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/train.py", line 17, in <module>
from ..tokens.doc import Doc
File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/tokens/__init__.py", line 1, in <module>
from .doc import Doc
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
@ines, I tried in a clean env on ubuntu and it worked without any problem so I am suspecting it has something to do with MacOS (maybe in the line of this Cython issue: https://github.com/cython/cython/issues/1551 (I am a Cython newbie) ?
Do you have any recommendation in terms of MacOS setup? I use Miniconda env.
The .pyx to .cpp step happens on our server, and you download the generated .cpp files in the spaCy package. When you do pip install spacy-nightly, your compiler then turns them into then .so extensions, so you can import the library.
Until recently, we’ve been using a fairly old version of Cython to generate the .cpp files. I suspect that the newer version of Cython is generating code that your compiler is unhappy with. The suggestion in that issue looks like a likely solution — if we can change the compiler flag used when installing the module, that might work.
So I tried to install spaCy from source by compiling the develop branch of the repo.
I switched the COMPILE_OPTIONS in setup.py from -O3 to -O0 as advised in the above github issue and also tried to remove the open_mp flags.
But that didn’t change the error messages which is still
File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception
Here is the part of the pip install log that relates to the compilation of morphology.pxd (I can post the full log if you want)
building 'spacy.morphology' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -I/private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -c spacy/morphology.cpp -o build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -O0 -Wno-strict-prototypes -Wno-unused-function
In file included from spacy/morphology.cpp:505:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/arrayobject.h:15:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarrayobject.h:17:
In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarraytypes.h:1728:
/private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/npy_deprecated_api.h:11:2: warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
^
1 warning generated.
g++ -bundle -undefined dynamic_lookup -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -o build/lib.macosx-10.7-x86_64-3.6/spacy/morphology.cpython-36m-darwin.so -Wl,-rpath,@loader_path/../spacy/platform/darwin/lib
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9
I am on MacOSX Sierra 10.12.4.
Shall I revert to your previsously used versino of cython ?
I recently attempted to upgrade to the newest version, which resulted in the errors described above. Now I can not reinstall spacy and prodigy without the following error thrown by prodigy:
/Users/crivera5/.virtual_envs/spacy_python3/bin/python: Error while finding spec for 'prodigy.__main__' (<class 'ValueError'>: thinc.extra.search.MaxViolation has the wrong size, try recompiling); 'prodigy' is a package and cannot be directly executed
I have the issue when installing prodigy 0.4.0 with spacy-nightly v.2.0.17
clean environment. python 3.6. macos 10.11.6
I get the SystemError: initialization of morphology failed without raising an exception error when executing prodigy or executing spacy or importing spacy.
We’ve got this fixed in spaCy now, so the next update will take care of the problem.
The Morphology file had a a long enum, listing out the morphological features in the Universal Dependencies schema. Recent versions of Cython cause a lot of code to be generated from this construction, which causes the error on some platforms. I’ve found an easy workaround, that also reduces our compile times a bit.
How do we fix that problem? I am having it too. Just updated spacy to current and downloaded newest version of Prodigy. It downloaded nightly. When I run that it gives me the error.
@crivera The current version of Prodigy is not yet compatible with the latest spaCy – only with spacy-nightly==v2.0.0a17. But we already have the new Prodigy wheels ready – just need to test them, then we’ll send out another update!