I found the need for a version of the textcat.batch-train recipe that chooses the model based on F-score rather than accuracy. So I changed the code in textcat.py in the obvious way, introducing
another argument to the recipe and picking it up wherever the current code says “accuracy”.
It works. Happy to offer it back if desired.
Optimal accuracy does not align with optimal F score. I THINK this is happening because the eval dataset
is much more unbalanced than the training set. This is probably something better fixed by changing the evaluation dataset to be more sensible, but its
a shared task so I didn’t do that.