# Training BERT on prodigy **URL:** \ **Category:** Uncategorized\ **Tags:** transformers, relations\ **Created:** [February 2, 2023, 12:35pm UTC](https://support.prodi.gy/t/training-bert-on-prodigy/6339 "2023-02-02T12:35:06Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Mohammad](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Mohammad](https://support.prodi.gy/u/Mohammad)\ **Post date:** [February 2, 2023, 12:35pm UTC](https://support.prodi.gy/t/training-bert-on-prodigy/6339/1 "2023-02-02T12:35:06Z")
After I did the annotation on prodigy how I can run training on prodigy for BERT python3 -m prodigy bert.ner.manual correctd\_drsv1 ./model\_drsv1 ./CV\_json\_English\_Format.jsonl --label Skill,MaritalStatus,University,Designation,Degree,Experience --tokenizer-vocab ./bert-base-uncased-vocab.txt --lowercase --hide-wp-prefix -F transformers\_tokenizers.py ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [February 2, 2023, 1:55pm UTC](https://support.prodi.gy/t/training-bert-on-prodigy/6339/2 "2023-02-02T13:55:23Z")
Hi @Mohammad! > [@Mohammad](#): > > how I can run training on prodigy for BERT Technically, you can train however you want (e.g., write a script in PyTorch, TensorFlow, etc.). The annotated data is in a SQLite database (by default) and you can simply export out. Many Prodigy users who use Prodigy's BERT recipe already have their own training workflows. I suspect what you're looking for a simple way to train with Prodigy. Prodigy's default training recipe [`train`](https://prodi.gy/docs/recipes/#train), is really a wrapper for `spacy train` with simple defaults. The problem is training transformers (BERT) needs a lot of customization, therefore, if you want to train with spaCy, you'll need to familiarize yourself with spaCy training (e.g., `config`, GPU). > [@Training a NER model with en\_trf\_robertabase\_lg](https://support.prodi.gy/t/training-a-ner-model-with-en-trf-robertabase-lg/2433/4): > > See my comment on [this thread](https://support.prodi.gy/t/error-using-xlnet-for-text-classification-no-component-trf-tok2vec-found-in-pipeline/2416/2) for more details on training a text classifier with transformer weights. TL;DR: You can do it, but you should probably use a separate training script, since the transformer models need pretty specific settings. I would recommend reading up on spaCy's documentation on [transformer training](https://spacy.io/usage/embeddings-transformers#transformers-training). You'll likely need to already have knowledge about [spaCy training](https://spacy.io/usage/training). Also, you'll likely find more resources on [spaCy's GitHub discussions forum](https://github.com/explosion/spaCy/discussions) like > **[Training with BERT Transformers · explosion/spaCy · Discussion #12115](https://github.com/explosion/spaCy/discussions/12115)** > > Hi, I am developing a spaCy model and I want to use BERT transformers. Starting from the bottom, I have an annotations file from Prodigy that we are converting to spaCy data with the data-to-spacy ... Please post there if you have spaCy specific questions as the spaCy core team answers questions there. This FAQ may be helpful too to familiarize yourself with handling GPU's, which is another major thing to learn before training transformers (spaCy or anything other framework). > **[GPU Support Troubleshooting FAQ · explosion/spaCy · Discussion #11436](https://github.com/explosion/spaCy/discussions/11436)** > > GPU libraries are developed at a rapid pace, and with a number of versions available it can be hard to get a functioning environment. This FAQ post covers common questions and issues related to get... Ultimately, it's important you take a step back and consider: do you really need transformers (BERT)? > [@Will using transformers improve textcat\_multilabel score/accuracy?](https://support.prodi.gy/t/will-using-transformers-improve-textcat-multilabel-score-accuracy/6199/2): > > hi @joebuckle! There aren't guarantees transformers will always lead to better performance for GPU. I would say on average, yes, we would expect them to do better but at what cost? The only likely guarantee is that productionalizing your model will be harder (e.g., longer time to score, more memory). Vincent mentioned this previously: The spaCy team has done some benchmarks using transformers in spaCy based on [speed](https://spacy.io/usage/facts-figures#benchmarks-speed) and it's clear transformers slow end-to-end processing speed. But there a… For many projects (especially in industry) the answer is that the performance gains may not justify the additional work that's required, especially if you're starting out from scratch. However, I do think if you spend enough time and have the right expectations, it could be a valuable skill set to develop. But it'll take time! ---
**Author:** ![Mohammad](https://avatars.discourse-cdn.com/v4/letter/m/e274bd/32.png) [@Mohammad](https://support.prodi.gy/u/Mohammad)\ **Post date:** [February 2, 2023, 2:56pm UTC](https://support.prodi.gy/t/training-bert-on-prodigy/6339/3 "2023-02-02T14:56:29Z")
The task what I want to do to is extracting relationships in CV between skills and employment and what I understand spacy can not fit for this task without help with BERT as spacy can perform relations extraction. ---
**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\ **Post date:** [February 2, 2023, 3:09pm UTC](https://support.prodi.gy/t/training-bert-on-prodigy/6339/4 "2023-02-02T15:09:24Z")
> [@Mohammad](#): > > The task what I want to do to is extracting relationships in CV between skills and employment and what I understand spacy can not fit for this task without help with BERT as spacy can perform relations extraction. Thanks for the background! Not necessarily. You can train with CPU (non-transformers) or with GPU (transformers, aka BERT-like models). Sofie has a great video where she outlines both approaches: [![](https://img.youtube.com/vi/8HL-Ap5_Axo/hqdefault.jpg "SPACY v3: Custom trainable relation extraction component") ](https://www.youtube.com/watch?v=8HL-Ap5_Axo) She also has the repo as a spaCy project, which include both a command for `cpu_train` and one for `gpu_train`: > > > //github.com/explosion/projects/tree/v3/tutorials/rel\_component I would recommend watching her video carefully and following her project and modifying it as you need.