Running on CPU

Hi,

Is it possible to train in Prodigy on cpu without gpu?
I am currently getting the following warning with the train recipe, running on an Amazon ec2 instance (cpu only):

UserWarning: User provided device_type of 'cuda', but CUDA is not available.

After which the process is immediately killed.

Hi! Prodigy's train workflow is a little wrapper around spaCy's train command, which will train on CPU by default. If you're not providing a --gpu-id on the command line, it should train on CPU.

Can you share the full command you're running and the full traceback that includes where the warning happened?

Thank you for your quick response!
I am running the command:

prodigy train my_model --ner my_dataset -m en_core_web_trf

and getting back:

========================= Generating Prodigy config =========================
:information_source: Auto-generating config with spaCy
:information_source: Using config from base model
:heavy_check_mark: Generated training config

=========================== Initializing pipeline ===========================
[2022-02-03 12:44:09,141] [INFO] Set up nlp object from config
Components: ner
Merging training and evaluation data for 1 components

  • [ner] Training: 392 | Evaluation: 97 (20% split)
    Training: 338 | Evaluation: 92
    Labels: ner (5)
    [2022-02-03 12:44:09,243] [INFO] Pipeline: ['tok2vec', 'transformer', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer', 'ner']
    [2022-02-03 12:44:09,243] [INFO] Resuming training for: ['ner', 'transformer']
    [2022-02-03 12:44:09,251] [INFO] Created vocabulary
    [2022-02-03 12:44:09,252] [INFO] Finished initializing nlp object
    [2022-02-03 12:44:09,628] [INFO] Initialized pipeline components: ['tok2vec']
    :heavy_check_mark: Initialized pipeline

============================= Training pipeline =============================
Components: ner
Merging training and evaluation data for 1 components

  • [ner] Training: 392 | Evaluation: 97 (20% split)
    Training: 338 | Evaluation: 92
    Labels: ner (5)
    :information_source: Pipeline: ['tok2vec', 'transformer', 'tagger', 'parser',
    'attribute_ruler', 'lemmatizer', 'ner']
    :information_source: Frozen components: ['tagger', 'parser', 'attribute_ruler',
    'lemmatizer']
    :information_source: Initial learn rate: 0.0
    E # LOSS TOK2VEC LOSS TRANS... LOSS NER ENTS_F ENTS_P ENTS_R SPEED SCORE

Token indices sequence length is longer than the specified maximum sequence length for this model (776 > 512). Running this sequence through the model will result in indexing errors
/usr/local/lib64/python3.7/site-packages/torch/autocast_mode.py:141: UserWarning: User provided device_type of 'cuda', but CUDA is not available. Disabling
warnings.warn('User provided device_type of 'cuda', but CUDA is not available. Disabling')
0 0 0.00 143.00 226.19 17.08 20.00 14.91 59.10 0.17
Killed

What do you think could be the problem?

Thanks for the details! And I think the problem here might just be that you're running out of memory trying to train a transformer-based pipeline on CPU. For that, it typically makes more sense to train on a GPU.

If you just want to run a quick training experiment to see if your data is good and your model is learning, you can just use one of the non-transformer pipelines instead, e.g. en_core_web_lg (with word vectors) or even en_core_web_sm (without word vectors). This should give you similar signals and insights – and if the results look promising, you can then export your data and train a transformer-based pipeline on a GPU, which should give you another boost in accuracy. (And if your model isn't learning or there are problems, you usually want to go and work on the data first.)

Thank you! We'll definitely try experimenting with one of the other models first and then take it from there. This was very helpful, thanks again!