# Train Curve not using GPU even tho -g is set.

**URL:** <https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068>\
**Category:** Uncategorized\
**Tags:** bug, to-be-released, transformers, training\
**Created:** [November 3, 2022, 11:32am UTC](https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068 "2022-11-03T11:32:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![turulix](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/turulix/32/3388_2.png) [@turulix](https://support.prodi.gy/u/turulix)\
**Post date:** [November 3, 2022, 11:32am UTC](https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068/1 "2022-11-03T11:32:38Z")

</div>

Hey!

I'm not sure if i'm just simply missing something or if this is an actual bug.  
When I run `prodigy train-curve -g 0 --spancat Dataset -c .\config.cfg` it will still train on the CPU. `prodigy train -g 0 --spancat Dataset -c .\config.cfg` runs on the GPU as intended

Thanks in advance,  
Turulix

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [November 4, 2022, 2:59pm UTC](https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068/2 "2022-11-04T14:59:45Z")

</div>

hi @turulix!

Thanks for your message and welcome to the Prodigy community 👋

I just checked the code and `train-curve` simply loops through the same `train` recipe so no obvious problems stand out.

Can you provide your `config.cfg` file? The GPU setting used while training is not in `config.cfg` so I don't think it will be the cause but this will allow us to try to replicate.

---

<div class="post-metadata">

**Author:** ![turulix](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/turulix/32/3388_2.png) [@turulix](https://support.prodi.gy/u/turulix)\
**Post date:** [November 4, 2022, 10:39pm UTC](https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068/3 "2022-11-04T22:39:54Z")

</div>

Hey! Sure:

```python
[paths]
train = null
dev = null
vectors = null
init_tok2vec = null

[system]
gpu_allocator = "pytorch"
seed = 0

[nlp]
lang = "de"
pipeline = ["transformer","spancat"]
batch_size = 128
disabled = []
before_creation = null
after_creation = null
after_pipeline_creation = null
tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[components]

[components.spancat]
factory = "spancat"
max_positive = null
scorer = {"@scorers":"spacy.spancat_scorer.v1"}
spans_key = "sc"
threshold = 0.5

[components.spancat.model]
@architectures = "spacy.SpanCategorizer.v1"

[components.spancat.model.reducer]
@layers = "spacy.mean_max_reducer.v1"
hidden_size = 128

[components.spancat.model.scorer]
@layers = "spacy.LinearLogistic.v1"
nO = null
nI = null

[components.spancat.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0
pooling = {"@layers":"reduce_mean.v1"}
upstream = "*"

[components.spancat.suggester]
@misc = "spacy.ngram_suggester.v1"
sizes = [1,2,3]

[components.transformer]
factory = "transformer"
max_batch_items = 4096
set_extra_annotations = {"@annotation_setters":"spacy-transformers.null_annotation_setter.v1"}

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "bert-base-german-cased"
mixed_precision = false

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.transformer.model.grad_scaler_config]

[components.transformer.model.tokenizer_config]
use_fast = true

[components.transformer.model.transformer_config]

[corpora]

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0
gold_preproc = false
limit = 0
augmenter = null

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"
seed = ${system.seed}
gpu_allocator = ${system.gpu_allocator}
dropout = 0.1
patience = 1600
max_epochs = 0
max_steps = 20000
eval_frequency = 200
frozen_components = []
annotating_components = []
before_to_disk = null

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256
get_length = null

[training.logger]
@loggers = "spacy.ConsoleLogger.v1"
progress_bar = false

[training.optimizer]
@optimizers = "Adam.v1"
beta1 = 0.9
beta2 = 0.999
L2_is_weight_decay = true
L2 = 0.01
grad_clip = 1.0
use_averages = false
eps = 0.00000001

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 0.00005

[training.score_weights]
spans_sc_f = 1.0
spans_sc_p = 0.0
spans_sc_r = 0.0

[pretraining]

[initialize]
vectors = ${paths.vectors}
init_tok2vec = ${paths.init_tok2vec}
vocab_data = null
lookups = null
before_init = null
after_init = null

[initialize.components]

[initialize.tokenizer]

```

Console output from `prodigy train-curve -g 0 --spancat Dataset -c .\config.cfg`:

```python
========================= Generating Prodigy config =========================
✔ Generated training config

=========================== Train curve diagnostic ===========================
Training 4 times with 25%, 50%, 75%, 100% of the data

% Score spancat
---- ------ ------
Some weights of the model checkpoint at bert-base-german-cased were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).     
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

```

Console output from `prodigy train -g 0 --spancat Dataset -c .\config.cfg`:

```python
ℹ Using GPU: 0

========================= Generating Prodigy config =========================
✔ Generated training config

=========================== Initializing pipeline ===========================
[2022-11-04 23:38:04,069] [INFO] Set up nlp object from config
Components: spancat
Merging training and evaluation data for 1 components
  - [spancat] Training: 873 | Evaluation: 220 (20% split)
Training: 863 | Evaluation: 216
Labels: spancat (4)
[2022-11-04 23:38:04,960] [INFO] Pipeline: ['transformer', 'spancat']
[2022-11-04 23:38:04,963] [INFO] Created vocabulary
[2022-11-04 23:38:04,964] [INFO] Finished initializing nlp object
Some weights of the model checkpoint at bert-base-german-cased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).     
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

```

maybe these help out aswell!

---

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [March 14, 2023, 1:44pm UTC](https://support.prodi.gy/t/train-curve-not-using-gpu-even-tho-g-is-set/6068/4 "2023-03-14T13:44:23Z")

</div>

hi @turulix!

Apologies for the delayed response. We've just added a new fix for the next Prodigy release that adds `setup_gpu(gpu_id)` to `train-curve`. I've added a "to-be-released" tag and will post back when that update is released.

This was in `train` but was not included in `train-curve`. `setup_gpu()` is from [spaCy](https://github.com/explosion/spaCy/blob/e8cab4625c12666ef599f19eb60403500af2a385/spacy/cli/_util.py#L573) but here's what it does:

```python
def setup_gpu(use_gpu: int, silent=None) -> None:
    """Configure the GPU and log info."""
    if silent is None:
        local_msg = Printer()
    else:
        local_msg = Printer(no_print=silent, pretty=not silent)
    if use_gpu >= 0:
        local_msg.info(f"Using GPU: {use_gpu}")
        require_gpu(use_gpu)
    else:
        local_msg.info("Using CPU")
        if gpu_is_available():
            local_msg.info("To switch to GPU 0, use the option: --gpu-id 0")

```

Without this it wasn't printing the same message `Using GPU: {use_gpu}` to console. It also uses [`require_gpu()`](https://spacy.io/api/top-level#spacy.require_gpu). Thanks again for reporting this problem and hope this resolves the issues!