Installation problem with spacy-nightly v2.0.0a14

What version of SpaCy does one need to use to run prodigy. I’ve been trying to install the newest version of SpaCy-nightly in a clean python 3.5 environment.

When importing I get the following in ipython:

In [1]: import spacy
SystemError                               Traceback (most recent call last)
<ipython-input-1-76a01d9c502b> in <module>()
----> 1 import spacy

~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/ in <module>()
      2 from __future__ import unicode_literals
----> 4 from import info as cli_info
      5 from .glossary import explain
      6 from .deprecated import resolve_load_name

~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/cli/ in <module>()
      4 from .package import package
      5 from .profile import profile
----> 6 from .train import train
      7 from .convert import convert
      8 from .model import model

~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/cli/ in <module>()
     12 from timeit import default_timer as timer
---> 14 from ..tokens.doc import Doc
     15 from ..scorer import Scorer
     16 from import GoldParse, merge_sents

~/.virtual_envs/spacy_tools_python3/lib/python3.5/site-packages/spacy/tokens/ in <module>()
----> 1 from .doc import Doc
      2 from .token import Token
      3 from .span import Span
      5 __all__ = [Doc, Token, Span]

morphology.pxd in init spacy.tokens.doc()

SystemError: initialization of morphology did not return an extension module

I’ve tried to remake a virtual environment several times.

Thanks for the report – this is strange. Will investigate!

In the meantime, could you try and see if downgrading to spacy-nightly v2.0.0a13 helps? (This version is also compatible with the latest version of Prodigy).

pip install spacy-nightly==2.0.0a13

I will try this. Thanks!!!

I was able to install that build with successful import.

The underlying problem of this will be fixed in the upcoming Prodigy v0.3.0 btw, so you should be able to use it with the latest spaCy version then.

Seeing the same issue installing spacy 2.0.0a15 on a clean install of python 3.6, reverting to 2.0.0a13 also fixed the problem.

We just pushed a new spaCy update yesterday, v2.0.0a16, which uses Thincv6.9.0 and should fix the compatibility issues. It requires the latest version of Prodigy that was released yesterday, v0.3.0, so if you install that cleanly, it should use all the latest dependencies!

Hi @ines, thank for your answer.

Do I have to install prodigy to use spacy v2.0.0a16?

Right now

pip install spacy-nightly

in a clean (conda) python 3.6 environment on MacOS install spacy v2.0.0a16 with thinc v6.9.0 but

spacy download en

still fail as above.

End of log:

Successfully installed certifi-2017.7.27.1 chardet-3.0.4 cymem-1.31.2 cytoolz-0.8.2 dill- ftfy-4.4.3 html5lib-0.999999999 idna-2.6 msgpack-numpy-0.4.1 msgpack-python-0.4.8 murmurhash-0.28.0 numpy-1.13.3 pathlib-1.0.1 plac-0.9.6 preshed-1.0.0 regex-2017.4.5 requests-2.18.4 six-1.11.0 spacy-nightly-2.0.0a16 termcolor-1.1.0 thinc-6.9.0 toolz-0.8.2 tqdm-4.19.1.post1 ujson-1.35 urllib3-1.22 wcwidth-0.1.7 webencodings-0.5.1 wrapt-1.10.11
(analytics) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ spacy download en
Traceback (most recent call last):
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/", line 109, in _get_module_details
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/", line 4, in <module>
    from import info as cli_info
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/", line 6, in <module>
    from .train import train
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/cli/", line 17, in <module>
    from ..tokens.doc import Doc
  File "/Users/thomaswolf/miniconda3/envs/analytics/lib/python3.6/site-packages/spacy/tokens/", line 1, in <module>
    from .doc import Doc
  File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception

Ah, no – but you need spaCy v2.0.0a16 to use the latest version of Prodigy.

And this is strange – just as a sanity check, do you get the same error when you run:

python -m spacy download en

Or compare the output of both:

spacy info
python -m spacy info

If this does the trick, there might be an issue with the spacy executable that should actually only alias spacy to python -m spacy. For example, it could have ended up on the wrong Python path. (So maybe we should update the docs again to always recommend python -m as we used to, since it still seems to be the safest option, especially when working with virtual environments etc.)

Hmm all these commands gives me the same error.
Actually opening python (3.6) and trying import spacy (before the model is downloaded) also fails as

(analytics-test) MacBook-Pro-de-Thomas:hf_analytics thomaswolf$ python
Python 3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:14:59) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import spacy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/", line 4, in <module>
    from import info as cli_info
  File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/", line 6, in <module>
    from .train import train
  File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/cli/", line 17, in <module>
    from ..tokens.doc import Doc
  File "/Users/thomaswolf/miniconda3/envs/analytics-test/lib/python3.6/site-packages/spacy/tokens/", line 1, in <module>
    from .doc import Doc
  File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception

Tried in a clean conda environment again.

@ines, I tried in a clean env on ubuntu and it worked without any problem so I am suspecting it has something to do with MacOS (maybe in the line of this Cython issue: (I am a Cython newbie) ?

Do you have any recommendation in terms of MacOS setup? I use Miniconda env.

Hmm! So to summarize the chain here, we go:

.pyx source files -> Cython = .cpp files
.cpp files -> GCC/clang = .so extensions
.so extensions + python = import spacy

The .pyx to .cpp step happens on our server, and you download the generated .cpp files in the spaCy package. When you do pip install spacy-nightly, your compiler then turns them into then .so extensions, so you can import the library.

Until recently, we’ve been using a fairly old version of Cython to generate the .cpp files. I suspect that the newer version of Cython is generating code that your compiler is unhappy with. The suggestion in that issue looks like a likely solution — if we can change the compiler flag used when installing the module, that might work.

Thanks @honnibal for the Cython introduction!

So I tried to install spaCy from source by compiling the develop branch of the repo.

I switched the COMPILE_OPTIONS in from -O3 to -O0 as advised in the above github issue and also tried to remove the open_mp flags.

But that didn’t change the error messages which is still

  File "morphology.pxd", line 25, in init spacy.tokens.doc
SystemError: initialization of morphology failed without raising an exception

Here is the part of the pip install log that relates to the compilation of morphology.pxd (I can post the full log if you want)

    building 'spacy.morphology' extension
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include -arch x86_64 -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -I/private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include -I/Users/thomaswolf/miniconda3/envs/analytics-test/include/python3.6m -c spacy/morphology.cpp -o build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -O0 -Wno-strict-prototypes -Wno-unused-function
    In file included from spacy/morphology.cpp:505:
    In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/arrayobject.h:15:
    In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarrayobject.h:17:
    In file included from /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/ndarraytypes.h:1728:
    /private/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/pip-pp89oel9-build/include/numpy/npy_deprecated_api.h:11:2: warning: "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
    #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
    1 warning generated.
    g++ -bundle -undefined dynamic_lookup -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/spacy/morphology.o -L/Users/thomaswolf/miniconda3/envs/analytics-test/lib -o build/lib.macosx-10.7-x86_64-3.6/spacy/ -Wl,-rpath,@loader_path/../spacy/platform/darwin/lib
    clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9

I am on MacOSX Sierra 10.12.4.

Shall I revert to your previsously used versino of cython ?

I am also having the issue with Spacy-Nightly V2.0.0A16 as reported above.

I recently attempted to upgrade to the newest version, which resulted in the errors described above. Now I can not reinstall spacy and prodigy without the following error thrown by prodigy:

/Users/crivera5/.virtual_envs/spacy_python3/bin/python: Error while finding spec for 'prodigy.__main__' (<class 'ValueError'>: has the wrong size, try recompiling); 'prodigy' is a package and cannot be directly executed

I have the issue when installing prodigy 0.4.0 with spacy-nightly v.2.0.17

clean environment. python 3.6. macos 10.11.6

I get the SystemError: initialization of morphology failed without raising an exception error when executing prodigy or executing spacy or importing spacy.


We’ve got this fixed in spaCy now, so the next update will take care of the problem.

The Morphology file had a a long enum, listing out the morphological features in the Universal Dependencies schema. Recent versions of Cython cause a lot of code to be generated from this construction, which causes the error on some platforms. I’ve found an easy workaround, that also reduces our compile times a bit.

How do we fix that problem? I am having it too. Just updated spacy to current and downloaded newest version of Prodigy. It downloaded nightly. When I run that it gives me the error.

@crivera The current version of Prodigy is not yet compatible with the latest spaCy – only with spacy-nightly==v2.0.0a17. But we already have the new Prodigy wheels ready – just need to test them, then we’ll send out another update!

See this thread for more details: Installing problem with prodigy-0.4.0 on Ubuntu 16.0.4 LTS