# Some idea to optimize llm annotations in unbalanced datasets?
**URL:** https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893
**Category:** Uncategorized
**Tags:** usage, textcat
**Created:** [November 11, 2023, 3:06pm UTC](https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893 "2023-11-11T15:06:07Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@info2000](https://support.prodi.gy/u/info2000)
#### Post date: [November 11, 2023, 3:06pm UTC](https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893/1 "2023-11-11T15:06:08Z")
The questions:
1)on llm annotate or textcat.llm.fetch is possible to filter what examples send to the llm model to don't expend money on well trained labels?
1. some idea to annotate with model using Setfit or other local few shot models
Context
I have a textcat dataset with 100000 examples, 34 labels
4 of these labels need to have more examples
I pre-annotated these 4 cats with patterns, to speed up, and filtered with a custom recipe to display only these labels to human review
And use some model as annotator could speed up my model fine tuning
Thanks
---
### Author:  [@koaning](https://support.prodi.gy/u/koaning)
#### Post date: [November 15, 2023, 9:21am UTC](https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893/2 "2023-11-15T09:21:45Z")
I'll respond to your questions seperately below.
1. The `textcat.llm.fetch` recipe needs an `examples.jsonl` file to send to an LLM provider. Nothing is preventing you from doing some analysis yourself such that the `examples.jsonl` file is a subset of interest that only contains the examples that you'd want to send. In your case, you may remove examples that the model is already confident about.
2. Funny you mention `SetFit`. It's one of the models that I may add to the [Prodigy-Huggingface](https://prodi.gy/docs/plugins/#hf) plugin. I have no timeline for this, but theoretically there's no reason why a setfit model can't be used as a model in the loop. That said, you can also train a huggingface model/spaCy model to help you with that too.
Let me know if this helps. If you appreciate more advice, could you share some more context of your task? What four labels need more data? Are the labels mututally exclusive?
---
### Author:  [@info2000](https://support.prodi.gy/u/info2000)
#### Post date: [November 15, 2023, 2:15pm UTC](https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893/3 "2023-11-15T14:15:23Z")
good @koaning Vincent
I will play with the HuggingFace plugin and will update with the process
about the first question: there's some example of custom recipe for annotations? my idea is to extend the openai.fetch recipe
thanks
---
### Author:  [@koaning](https://support.prodi.gy/u/koaning)
#### Post date: [November 16, 2023, 10:05am UTC](https://support.prodi.gy/t/some-idea-to-optimize-llm-annotations-in-unbalanced-datasets/6893/4 "2023-11-16T10:05:36Z")
The `llm` recipes are in our internal repo for now, but this does serve as a nice reminder that we should port them to our [recipes repository](https://github.com/explosion/prodigy-recipes/). I've added an internal ticket and will let you know once it's been taken care of!