Running on CPU

, ,
1

Hi,

Is it possible to train in Prodigy on cpu without gpu?
I am currently getting the following warning with the train recipe, running on an Amazon ec2 instance (cpu only):

UserWarning: User provided device_type of 'cuda', but CUDA is not available.

After which the process is immediately killed.

2

Hi! Prodigy's train workflow is a little wrapper around spaCy's train command, which will train on CPU by default. If you're not providing a --gpu-id on the command line, it should train on CPU.

Can you share the full command you're running and the full traceback that includes where the warning happened?

3

Thank you for your quick response!
I am running the command:

prodigy train my_model --ner my_dataset -m en_core_web_trf

and getting back:

========================= Generating Prodigy config =========================
:information_source: Auto-generating config with spaCy
:information_source: Using config from base model
:heavy_check_mark: Generated training config

=========================== Initializing pipeline ===========================
[2022-02-03 12:44:09,141] [INFO] Set up nlp object from config
Components: ner
Merging training and evaluation data for 1 components

  • [ner] Training: 392 | Evaluation: 97 (20% split)
    Training: 338 | Evaluation: 92
    Labels: ner (5)
    [2022-02-03 12:44:09,243] [INFO] Pipeline: ['tok2vec', 'transformer', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']
    [2022-02-03 12:44:09,243] [INFO] Resuming training for: ['ner', 'transformer']
    [2022-02-03 12:44:09,251] [INFO] Created vocabulary
    [2022-02-03 12:44:09,252] [INFO] Finished initializing nlp object
    [2022-02-03 12:44:09,628] [INFO] Initialized pipeline components: ['tok2vec']
    :heavy_check_mark: Initialized pipeline

============================= Training pipeline =============================
Components: ner
Merging training and evaluation data for 1 components

  • [ner] Training: 392 | Evaluation: 97 (20% split)
    Training: 338 | Evaluation: 92
    Labels: ner (5)
    :information_source: Pipeline: ['tok2vec', 'transformer', 'tagger', 'parser',
    'attribute_ruler', 'lemmatizer', 'ner']
    :information_source: Frozen components: ['tagger', 'parser', 'attribute_ruler',
    'lemmatizer']
    :information_source: Initial learn rate: 0.0
    E # LOSS TOK2VEC LOSS TRANS... LOSS NER ENTS_F ENTS_P ENTS_R SPEED SCORE

Token indices sequence length is longer than the specified maximum sequence length for this model (776 > 512). Running this sequence through the model will result in indexing errors
/usr/local/lib64/python3.7/site-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
warnings.warn('User provided device_type of 'cuda', but CUDA is not available. Disabling')
0 0 0.00 143.00 226.19 17.08 20.00 14.91 59.10 0.17
Killed

What do you think could be the problem?

4

Thanks for the details! And I think the problem here might just be that you're running out of memory trying to train a transformer-based pipeline on CPU. For that, it typically makes more sense to train on a GPU.

If you just want to run a quick training experiment to see if your data is good and your model is learning, you can just use one of the non-transformer pipelines instead, e.g. en_core_web_lg (with word vectors) or even en_core_web_sm (without word vectors). This should give you similar signals and insights – and if the results look promising, you can then export your data and train a transformer-based pipeline on a GPU, which should give you another boost in accuracy. (And if your model isn't learning or there are problems, you usually want to go and work on the data first.)

5

Thank you! We'll definitely try experimenting with one of the other models first and then take it from there. This was very helpful, thanks again!