# textcat.batch-train

**URL:** <https://support.prodi.gy/t/textcat-batch-train/211>\
**Category:** Uncategorized\
**Tags:** usage, textcat\
**Created:** [January 11, 2018, 11:32pm UTC](https://support.prodi.gy/t/textcat-batch-train/211 "2018-01-11T23:32:17Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cbrew](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/cbrew/32/64_2.png) [@cbrew](https://support.prodi.gy/u/cbrew)\
**Post date:** [January 11, 2018, 11:32pm UTC](https://support.prodi.gy/t/textcat-batch-train/211/1 "2018-01-11T23:32:18Z")

</div>

I found the need for a version of the textcat.batch-train recipe that chooses the model based on F-score rather than accuracy. So I changed the code in textcat.py in the obvious way, introducing  
another argument to the recipe and picking it up wherever the current code says “accuracy”.  
It works. Happy to offer it back if desired.

Optimal accuracy does not align with optimal F score. I THINK this is happening because the eval dataset  
is much more unbalanced than the training set. This is probably something better fixed by changing the evaluation dataset to be more sensible, but its  
a shared task so I didn’t do that.

---

<div class="post-metadata">

**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\
**Post date:** [January 12, 2018, 12:47pm UTC](https://support.prodi.gy/t/textcat-batch-train/211/2 "2018-01-12T12:47:35Z")

</div>

Thanks! We’re in the process of publishing the recipes on Github. We’re just figuring out the best process to build them into the Prodigy wheels once they’re in a separate repo. Once they’re published, pull requests will be very welcome!

About the accuracy vs F-score: This is a topic that makes me feel dumb every now and again, because it seems like it should be quite obvious, but then I find myself scratching my head.

If the model is constrained to output one class prediction per instance, I think accuracy should be the same as micro-averaged F1, right? However, this is obviously not true if we let the model predict multiple classes per instance, which the default spaCy text classification model is allowed to do.

---

<div class="post-metadata">

**Author:** ![andy](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/andy/32/966_2.png) [@andy](https://support.prodi.gy/u/andy)\
**Post date:** [January 16, 2018, 9:09pm UTC](https://support.prodi.gy/t/textcat-batch-train/211/3 "2018-01-16T21:09:02Z")

</div>

I’d be interested in having that recipe if you’re sharing! One of my textcat models is for rare categories and the f-score just keeps dropping as the accuracy improves…

---

<div class="post-metadata">

**Author:** ![jphcoi](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/jphcoi/32/425_2.png) [@jphcoi](https://support.prodi.gy/u/jphcoi)\
**Post date:** [August 29, 2018, 11:39am UTC](https://support.prodi.gy/t/textcat-batch-train/211/4 "2018-08-29T11:39:15Z")

</div>

Dear Chris,  
I’m also experiencing some difficulties training a binary classification model with a very small number of examples assigned to “accept” (~5%). Searching for the optimal accuracy tends to produce a classifier which systematic outcome is “reject” with a very high accuracy score. I assume optimizing the learning using F-score would be more fruitful ? Could you please share changes you made in textcat.py ?  
Or is there a more generic version to do so ?  
Thanks a lot!