# Using transformer models inside prodigy and finetuning **URL:** \ **Category:** Uncategorized\ **Tags:** enhancement, usage, transformers\ **Created:** [February 28, 2020, 7:47am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604 "2020-02-28T07:47:06Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![RohitRanga](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@RohitRanga](https://support.prodi.gy/u/RohitRanga)\ **Post date:** [February 28, 2020, 7:47am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/1 "2020-02-28T07:47:06Z")
Hi, Does Prodigy support usage of Transformer models like BERT or XLNET for performing text classification? Currently I am able to use en\_core\_web\_lg, sm, md, etc. only. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 28, 2020, 9:36am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/2 "2020-02-28T09:36:29Z")
Hi! Prodigy's training recipes currently use spaCy's regular `textcat` model – but you can always export the data and use the training scripts provided in [`spacy-transformers`](https://github.com/explosion/spacy-transformers). I've been experimenting a lot with workflows for annotating with transformers _in the loop_, but it's been pretty tricky so far. The models typically require larger batch sizes and don't always respond that well to small single updates. They're also a bit slow, especially on CPU, so you can't really avoid having to wait around for the model to finish updating. A more promising approach IMO is to use Prodigy to collect a small dataset with `textcat.manual`, use it to fine-tune a large transformer model (ideally to very high accuracy), and then use that to semi-automatically create large volumes of training data for a more efficient (and just as accurate) runtime model. I've been working on some recipes for this that I'll hopefully be able to share soon 🙂 ---
**Author:** ![RohitRanga](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@RohitRanga](https://support.prodi.gy/u/RohitRanga)\ **Post date:** [February 28, 2020, 9:52am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/3 "2020-02-28T09:52:28Z")
Sure, thanks Ines! ---
**Author:** ![RohitRanga](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@RohitRanga](https://support.prodi.gy/u/RohitRanga)\ **Post date:** [February 28, 2020, 12:00pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/4 "2020-02-28T12:00:53Z")
Hi, I have my jsonl file which has been exported from prodigy dataset after manually annotating about 2500 examples. I would like to use this to finetune a BERT transformer from inside spaCy for text classification. Can you please guide me on how I can do this? Thanks! I came across this but it looks too complicated for a starter. [https://github.com/explosion/spacy-transformers/blob/master/examples/train\_textcat.py](https://github.com/explosion/spacy-transformers/blob/master/examples/train_textcat.py) ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [February 28, 2020, 2:06pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/5 "2020-02-28T14:06:59Z")
Merged the two transformers threads to keep things in one place 🙂 > [@RohitRanga](#): > > I came across this but it looks too complicated for a starter. [https://github.com/explosion/spacy-transformers/blob/master/examples/train\_textcat.py](https://github.com/explosion/spacy-transformers/blob/master/examples/train_textcat.py) Well, this is the full end-to-end training loop with preprocessing, various settings, evaluation logic, early stopping, pretty output and so on. Usage is pretty straightforward – a single command. You can even run it without data to test it on example data (IMDB dataset): ```bash python train_textcat.py en_trf_bertbaseuncased_lg ``` You can run `python train_textcxat.py --help` to see the available arguments. To run it on your own data, create a directory with two files `training.jsonl` and `evaluation.jsonl` that look like this, and then pass it in as the input directory. ```json ["This is a text", {"cats": {"LABEL_ONE": 0.0, "LABEL_TWO": 1.0}}] ``` This should be very easy to create from Prodigy annotations collected with `textcat.manual` – the `"accept"` list of each annotation contains the correct labels. All other labels are zero. ---
**Author:** ![RohitRanga](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@RohitRanga](https://support.prodi.gy/u/RohitRanga)\ **Post date:** [February 29, 2020, 4:21pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/6 "2020-02-29T16:21:58Z")
Great! Thank you very much for the support Ines! ---
**Author:** ![RohitRanga](https://avatars.discourse-cdn.com/v4/letter/r/9d8465/32.png) [@RohitRanga](https://support.prodi.gy/u/RohitRanga)\ **Post date:** [March 1, 2020, 5:05pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/7 "2020-03-01T17:05:47Z")
Hi Ines, What about those annotations where "answer" was "reject" instead of "accept"? Those will not have "accept" field. Please correct me if I am wrong, we cannot use those annotations for training or evaluation and hence they need not be added to training.jsonl or evaluation.jsonl. Also, can you please explain how evaluation is done using evaluation.jsonl file? When I run the command, by default, it takes positive label for evaluation as one of the possible labels. ---
**Author:** ![Simpan](https://avatars.discourse-cdn.com/v4/letter/s/dfb087/32.png) [@Simpan](https://support.prodi.gy/u/Simpan)\ **Post date:** [April 29, 2020, 4:48pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/9 "2020-04-29T16:48:09Z")
Hi Ines! Just stumbled upon this post. I'm currently collecting data to fine-tune a transformers model for text classification. Would you mind elaborating on the third step you are suggesting: _"to semi-automatically create large volumes of training data for a more efficient (and just as accurate) runtime model."_ . - How would this be done? Is this some kind of data augmentation technique you are talking about? Any pointers or resources would be really helpful! 🙂 Best regards, Simon ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [April 30, 2020, 10:01am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/10 "2020-04-30T10:01:48Z")
> [@Simpan](#): > > How would this be done? Is this some kind of data augmentation technique you are talking about? I think people also refer to this as "uptraining". Basically, a workflow could look like this: 1. Manually label a small set of examples from scratch and train a large transformer-based model. That model may be pretty accurate already, but possibly very slow and unwieldy. 2. Use that model to predict categories in your text and stream those in for annotation. - If the score is _above_ a certain threshold, automatically add the example to your dataset and assume it's correcct. - If the score is _below_ a certain threshold, skip it entirely. - If the score is _between_ the low and high threshold, send it out for annotation again and correct the model if it's wrong. 3. Train a new and more efficient model on _all_ collected data. This model now ideally has a lot of data to learn from and may achieve the same accuracy as a much larger model trained on fewer examples. But you didn't have to spend time and create _all_ those annotations from scratch. A workflow like this should be pretty straightforward to implement with Prodigy and a [custom recipe](https://prodi.gy/docs/custom-recipes) that processes the incoming texts and only yields out selected examples and uses the [database API](https://prodi.gy/docs/api-database#database) to auto-add annotations to the dataset. ---
**Author:** ![Simpan](https://avatars.discourse-cdn.com/v4/letter/s/dfb087/32.png) [@Simpan](https://support.prodi.gy/u/Simpan)\ **Post date:** [May 1, 2020, 10:08am UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/11 "2020-05-01T10:08:37Z")
Really cool, will experiment with, thanks a lot for the explanation! 🙂 ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [May 1, 2020, 12:09pm UTC](https://support.prodi.gy/t/using-transformer-models-inside-prodigy-and-finetuning/2604/12 "2020-05-01T12:09:35Z")
@Simpan Cool, if you end up trying this, let me know how you go! I had a look and here's an old gist I found from some early experiments – maybe you can use that as inspiration or a starting point. (It still refers to `spacy_pytorch_transformers` instead of `spacy_transformers` and there may be a few other things that need to be adjusted. But it's more about the proposed workflow. The `textcat.pytt.create-data` is the interesting part – you can kinda ignore the rest.) > The trickier parts were finding the right thresholds and the right model for the task – and also the right task that's difficult enough that it'd benefit enough from a large transformer. For some of the datasets I created, the transformer model didn't actually beat a simpler CNN + bag of words [architecture](https://spacy.io/api/textcategorizer#architectures), or at least not by much.