Training BERT on prodigy

[Mohammad (Mismail)]

After I did the annotation on prodigy how I can run training on prodigy for BERT

python3 -m prodigy bert.ner.manual correctd_drsv1 ./model_drsv1 ./CV_json_English_Format.jsonl --label Skill,MaritalStatus,University,Designation,Degree,Experience --tokenizer-vocab ./bert-base-uncased-vocab.txt --lowercase --hide-wp-prefix -F transformers_tokenizers.py

[ryanwesslen]

Hi @Mohammad!

@Mohammad

how I can run training on prodigy for BERT

Technically, you can train however you want (e.g., write a script in PyTorch, TensorFlow, etc.). The annotated data is in a SQLite database (by default) and you can simply export out. Many Prodigy users who use Prodigy's BERT recipe already have their own training workflows.

I suspect what you're looking for a simple way to train with Prodigy. Prodigy's default training recipe train, is really a wrapper for spacy train with simple defaults. The problem is training transformers (BERT) needs a lot of customization, therefore, if you want to train with spaCy, you'll need to familiarize yourself with spaCy training (e.g., config, GPU).

Training a NER model with en_trf_robertabase_lg

See my comment on this thread for more details on training a text classifier with transformer weights. TL;DR: You can do it, but you should probably use a separate training script, since the transformer models need pretty specific settings.

I would recommend reading up on spaCy's documentation on transformer training. You'll likely need to already have knowledge about spaCy training.

Also, you'll likely find more resources on spaCy's GitHub discussions forum like

Training with BERT Transformers · explosion/spaCy · Discussion #12115

Please post there if you have spaCy specific questions as the spaCy core team answers questions there.

This FAQ may be helpful too to familiarize yourself with handling GPU's, which is another major thing to learn before training transformers (spaCy or anything other framework).

GPU Support Troubleshooting FAQ · explosion/spaCy · Discussion #11436

Ultimately, it's important you take a step back and consider: do you really need transformers (BERT)?

Will using transformers improve textcat_multilabel score/accuracy?

hi @joebuckle! There aren't guarantees transformers will always lead to better performance for GPU. I would say on average, yes, we would expect them to do better but at what cost? The only likely guarantee is that productionalizing your model will be harder (e.g., longer time to score, more memory). Vincent mentioned this previously: The spaCy team has done some benchmarks using transformers in spaCy based on speed and it's clear transformers slow end-to-end processing speed. But there a…

For many projects (especially in industry) the answer is that the performance gains may not justify the additional work that's required, especially if you're starting out from scratch. However, I do think if you spend enough time and have the right expectations, it could be a valuable skill set to develop. But it'll take time!

[Mohammad (Mismail)]

The task what I want to do to is extracting relationships in CV between skills and employment and what I understand spacy can not fit for this task without help with BERT as spacy can perform relations extraction.

[ryanwesslen]

@Mohammad

The task what I want to do to is extracting relationships in CV between skills and employment and what I understand spacy can not fit for this task without help with BERT as spacy can perform relations extraction.

Thanks for the background!

Not necessarily. You can train with CPU (non-transformers) or with GPU (transformers, aka BERT-like models).

Sofie has a great video where she outlines both approaches:

SPACY v3: Custom trainable relation extraction component

She also has the repo as a spaCy project, which include both a command for cpu_train and one for gpu_train:

I would recommend watching her video carefully and following her project and modifying it as you need.