training data format for multiclass textcat

[ryanwesslen]

Hi @n8te!

Thanks for your questions!

@n8te

The spacy data format you gave - is that what is fed into the "convert" CLI function to create the .spacy format?

Yes - for training in spaCy. You're not expected to do this in Prodigy (it's done for you).

@n8te

see a different format on the site which has the text field labeled as "text".

Did you see this on Prodigy data format? That's because Prodigy's formatting is slightly different.

@n8te

I have my text and label data ready after doing onehotencoding - i just need to get it in the right format to run through spacy to train. If prodigy is easier then i can use that too.

Yes - Prodigy is easier. There are a few different ways to convert, but can you get your data into a format like this:

{"text": "How can I get chewy chocolate chip cookies?", "label": "baking"}
{"text": "I want to make cake.", "label": "baking"}
{"text": "Change the order to pancakes.", "label": "substitutions"}
{"text": "Please substitute in bananas.", "label": "substitutions"}
{"text": "Where is the bathroom?", "label": "OTHER"}
{"text": "What's the price of the flowers?", "label": "OTHER"}

This is more similar to what Prodigy produces, hence what prodigy train will accept.

If that data above was in a file named data.jsonl, you can load into Prodigy database with:

python -m prodigy db-in mydata data.jsonl
✔ Created dataset 'mydata' in database SQLite
✔ Imported 6 annotations to 'mydata' (session 2022-08-26_16-02-11) in
database SQLite
Found and keeping existing "answer" in 0 examples

(FYI Ignore that "Found and keeping existing...", see this thread. The previous output confirms you loaded 6 annotations.)

@n8te

I ran a test with some labeled data i created using prodigy and manually labeling my data, but this thre an error using the "--exclusive" command when sing "textcat" and said i needed to use "textcat-multilabel" instead. I ran that and tested the model which output all the labels with the corresponding scores. Is that the correct use for exclusive labelling for more than 2 labels?

You're almost there!

You should use textcat, not textcat-multilabel, because you want mutually exclusive labeled. Here's prodigy train docs to explain the difference:

Argument	Type	Description	Default
--textcat, -tc	str	One or more (comma-separated) datasets for the text classifier (exclusive categories). Use the eval: prefix for evaluation sets.	None
--textcat-multilabel, -tcm	str	One or more (comma-separated) datasets for the text classifier (non-exclusive categories). Use the eval: prefix for evaluation sets.	None

You can then run prodigy train

python -m prodigy train output_dir --textcat mydata

textcat doesn't use --exclusive, that was the problem. This was changed in v1.11.0. When looking at Prodigy Support posts, definitely check the date. We try to update but can't update everything.

After you've trained your model, you can check by running your model after training. If the labels sum up to 1, you have trained for mutually exclusive. If they sum above 1, you have non-mutually exclusive.

import spacy
nlp = spacy.load("output_dir/model-best")
doc = nlp("I want cookies.")
doc.cats
# {'baking': 0.8704647421836853, 'substitutions': 0.09000309556722641, 'OTHER': 0.03953210636973381}

Hope this helps!