TypeError: can not serialize 'cupy._core.core.ndarray' object

, ,
1

Hi, @ines, I got a question about using GPU when training a model for text classification by using Spacy. What's weird is that this error disappeared when I deleted --gpu-id 0.
Do you have any suggestions about this question? I searched here but found no related questions and answers. Thanks!

(venv) C:\Users\Jayshow\PycharmProjects\spacy_transform>python -m spacy train textcatcorpus\config.cfg --output ./textcatoutput --paths.train tex
tcatcorpus\train.spacy --paths.dev textcatcorpus\dev.spacy --gpu-id 0
ℹ Saving to output directory: textcatoutput
ℹ Using GPU: 0

=========================== Initializing pipeline ===========================
[2022-01-16 02:20:07,666] [INFO] Set up nlp object from config
[2022-01-16 02:20:07,677] [INFO] Pipeline: ['transformer', 'textcat_multilabel']
[2022-01-16 02:20:07,681] [INFO] Created vocabulary
Traceback (most recent call last):
  File "C:\Python37\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\__main__.py", line 4, in <module>
    setup_cli()
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\cli\_util.py", line 71, in setup_cli
    command(prog_name=COMMAND)
  File "C:\Python37\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "C:\Python37\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "C:\Python37\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Python37\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Python37\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "C:\Python37\lib\site-packages\typer\main.py", line 497, in wrapper
    return callback(**use_params)  # type: ignore
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\cli\train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\cli\train.py", line 72, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\training\initialize.py", line 84, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\language.py", line 1266, in initialize
    self, data=I["vocab_data"], lookups=I["lookups"], vectors=I["vectors"]
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\training\initialize.py", line 135, in init_vocab
    vectors_hash = hash(nlp.vocab.vectors.to_bytes())
  File "spacy\vectors.pyx", line 453, in spacy.vectors.Vectors.to_bytes
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\util.py", line 1171, in to_bytes
    return srsly.msgpack_dumps(to_dict(getters, exclude))
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\spacy\util.py", line 1189, in to_dict
    serialized[key] = getter()
  File "spacy\vectors.pyx", line 447, in spacy.vectors.Vectors.to_bytes.serialize_weights
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\srsly\_msgpack_api.py", line 14, in msgpack_dumps
    return msgpack.dumps(data, use_bin_type=True)
  File "C:\Users\Jayshow\PycharmProjects\spacy_transform\venv\lib\site-packages\srsly\msgpack\__init__.py", line 55, in packb
    return Packer(**kwargs).pack(o)
  File "srsly\msgpack\_packer.pyx", line 285, in srsly.msgpack._packer.Packer.pack
  File "srsly\msgpack\_packer.pyx", line 291, in srsly.msgpack._packer.Packer.pack
  File "srsly\msgpack\_packer.pyx", line 288, in srsly.msgpack._packer.Packer.pack
  File "srsly\msgpack\_packer.pyx", line 282, in srsly.msgpack._packer.Packer._pack
TypeError: can not serialize 'cupy._core.core.ndarray' object


(venv) C:\Users\Jayshow\PycharmProjects\spacy_transform>pip list
Package              Version
-------------------- ----------------
aiofiles             0.5.0
altair               4.1.0
argon2-cffi          20.1.0
astor                0.8.1
async-generator      1.10
attrs                21.2.0
backcall             0.2.0
base58               2.1.0
beautifulsoup4       4.9.3
bleach               3.3.1
blinker              1.4
blis                 0.4.1
bs4                  0.0.1
cachetools           4.1.1
catalogue            2.0.6
certifi              2020.6.20
cffi                 1.14.6
chardet              3.0.4
click                7.1.2
colorama             0.4.4
contextualSpellCheck 0.4.1
cPython              0.0.6
cupy-cuda110         10.0.0
cycler               0.10.0
cymem                2.0.3
debugpy              1.3.0
decorator            5.0.9
defusedxml           0.7.1
editdistance         0.5.3
en-core-sci-lg       0.2.5
en-core-sci-sm       0.2.5
en-core-web-sm       3.1.0
en-core-web-trf      3.1.0
entrypoints          0.3
et-xmlfile           1.1.0
fastapi              0.68.0
fastrlock            0.6
filelock             3.0.12
gitdb                4.0.7
GitPython            3.1.18
grobid-client-python 0.0.2
h11                  0.9.0
httplib2             0.19.1
huggingface-hub      0.0.8
idna                 2.10
importlib-metadata   1.7.0
ipykernel            6.0.2
ipython              7.25.0
ipython-genutils     0.2.0
ipywidgets           7.6.3
jedi                 0.18.0
jiagu                0.2.3
Jinja2               3.0.1
joblib               0.16.0
jsonschema           3.2.0
jupyter              1.0.0
jupyter-client       6.2.0
jupyter-console      6.4.0
jupyter-core         4.7.1
jupyterlab-pygments  0.1.2
jupyterlab-widgets   1.0.2
kiwisolver           1.3.1
lxml                 4.6.4
MarkupSafe           2.0.1
matplotlib           3.4.2
matplotlib-inline    0.1.2
metaknowledge        3.4.1
mistune              0.8.4
murmurhash           1.0.2
nbclient             0.5.3
nbconvert            6.1.0
nbformat             5.1.3
nest-asyncio         1.5.1
networkx             2.6.2
nltk                 3.6.5
nmslib               2.0.6
notebook             6.4.0
numpy                1.21.5
openpyxl             3.0.7
packaging            21.0
pandas               1.3.0
pandocfilters        1.4.3
parso                0.8.2
pathy                0.6.0
peewee               3.13.3
pickleshare          0.7.5
Pillow               8.3.1
pip                  21.3.1
plac                 1.1.3
plotext              4.1.5
preshed              3.0.2
prodigy              1.11.2
prometheus-client    0.11.0
prompt-toolkit       3.0.19
protobuf             3.17.3
psutil               5.7.2
pyarrow              4.0.1
pybind11             2.5.0
pycparser            2.20
pydantic             1.8.2
pydeck               0.6.2
Pygments             2.9.0
PyJWT                2.1.0
pymongo              3.12.1
pyparsing            2.4.7
pyrsistent           0.18.0
pysbd                0.3.2
python-dateutil      2.8.2
pytz                 2021.1
pywin32              301
pywinpty             1.1.3
pyzmq                22.1.0
qtconsole            5.1.1
QtPy                 1.9.0
regex                2021.8.3
requests             2.24.0
sacremoses           0.0.45
scikit-learn         0.23.2
scipy                1.5.2
scispacy             0.2.5-unreleased
seaborn              0.11.1
selenium             3.141.0
Send2Trash           1.7.1
sentencepiece        0.1.91
setuptools           57.4.0
six                  1.16.0
smart-open           5.1.0
smmap                4.0.0
soupsieve            2.2.1
spacy                3.1.4
spacy-alignments     0.8.3
spacy-legacy         3.0.8
spacy-lookups-data   1.0.2
spacy-transformers   1.0.3
srsly                2.4.2
starlette            0.14.2
streamlit            0.84.1
terminado            0.10.1
testpath             0.5.0
thinc                8.0.13
threadpoolctl        2.1.0
tokenizers           0.10.3
toml                 0.10.2
toolz                0.10.0
torch                1.7.1+cu110
torchaudio           0.7.2
torchvision          0.8.2+cu110
tornado              6.1
tqdm                 4.48.2
traitlets            5.0.5
transformers         4.6.1
typer                0.3.2
typing-extensions    3.10.0.0
tzlocal              2.1
urllib3              1.25.10
uvicorn              0.13.4
validators           0.18.2
wasabi               0.8.2
watchdog             2.1.3
wcwidth              0.2.5
webencodings         0.5.1
websockets           8.1
wheel                0.37.0
widgetsnbextension   3.5.1
xlrd                 2.0.1
xlwt                 1.3.0
zipp                 3.1.0
2

Not sure exactly what's going on, but here are a couple of things that come to mind.

Your versions of spaCy and spacy-transformers are slightly old, and you might be missing a bugfix. Can you upgrade to the latest (pip install -U spacy spacy-transformers) and try again? Note you may have to re-download models.

It looks like you are using word vectors with a Transformer. That's not recommended - word vectors typically don't add much if you have a Transformer. Could you share your config and try not using word vectors? (Word vectors should technically work with a Transformer I think, but this exact error has been caused by using them a few years ago, see here.)

3

Hi, @polm, many thanks for your quick and kind reply. Sorry, I am new to Spacy and Prodigy. Yes, I prefer the Transformer-based model to the normal Word Vector for my text classification experiments. Just wanna see whether there are some differences in the outputs.

I annotated my data by using textcat.manual and converted it to .spacy format by using data-to-spacy. See the command line below.

python -m prodigy data-to-spacy ./corpus --lang en --textcat-multilabel my_dataset

Then, I got train. spacy and eva. spacy as well as config.cfg file. See the details information in config.cfg file below. You probably see the Word vector that will be used is spacy.TextCatBOW.v2. Next, I executed the spacy train below with GPU, and it worked well.

[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = null
seed = 0

[nlp]
lang = "en"
pipeline = ["textcat_multilabel"]
batch_size = 1000
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.textcat_multilabel]
factory = "textcat_multilabel"
threshold = 0.5

[components.textcat_multilabel.model]
@architectures = "spacy.TextCatBOW.v2"
exclusive_classes = false
ngram_size = 1
no_output_layer = false
nO = null

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
accumulate_gradient = 1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2
get_length = null

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001
t = 0.0

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001
learn_rate = 0.001

[training.score_weights]
cats_score = null
cats_score_desc = null
cats_micro_p = null
cats_micro_r = null
cats_micro_f = 0.2
cats_macro_p = null
cats_macro_r = null
cats_macro_f = 0.2
cats_macro_auc = 0.3
cats_f_per_type = null
cats_macro_auc_per_type = 0.3

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.components.textcat_multilabel]

[initialize.components.textcat_multilabel.labels]
@readers = "spacy.read_labels.v1"
path = "textcatcorpus\\labels\\textcat_multilabel.json"

[initialize.tokenizer]

python -m spacy train textcatcorpus\config.cfg --output ./textcatoutput --paths.train tex
tcatcorpus\train.spacy --paths.dev textcatcorpus\dev.spacy --gpu-id 0

However, I just wanna try a Transformer-based model in my experiment. So I created a new config.cfg file at https://spacy.io/usage/training, see the details in the figure below. Then, I got the error presented in the title when executing the spacy train. That's so weird, right? :broken_heart:

image
image901×632 30.7 KB

4

Hi @ines, could you please help me with this question? I appreciate it.

5

Hi, so far we haven't been able to reproduce this and aren't sure what might be going on. I thought that this particular error with srsly and cupy was fixed almost two years ago in v1.0.2.

This is a long shot, but can you try uninstalling and reinstalling srsly with pip just to double-check?