textcat.teach stream of data is linear ?

[OtterMe]

Hello !
I'm currently working on a sentence classifier. My data pool is a dataset composed of sentences (text) from different document/pages (meta).
After using the manual recipe and then training my model, I go to the teach recipe.
I would expect the sentences to be ambiguous one with around 0.5 score and from different documents/pages. However the stream of data is very linear (document A page 1 -> 2 -> 3...document B page 1 -> 2....) with heterogenous probabilities.
Is there something that I don't understand stopping prodigy to stream the data inside a teach job as Document D page 100 -> Document Z page 1 ->Document A page 10 with all probabilities around 0.5 or so ?
Thank you very much for your help !

[ryanwesslen]

hi @OtterMe!

Thanks for your question and welcome to the Prodigy community

@OtterMe

I would expect the sentences to be ambiguous one with around 0.5 score and from different documents/pages. However the stream of data is very linear (document A page 1 -> 2 -> 3...document B page 1 -> 2....) with heterogenous probabilities.
Is there something that I don't understand stopping prodigy to stream the data inside a teach job as Document D page 100 -> Document Z page 1 ->Document A page 10 with all probabilities around 0.5 or so ?

Hmm... yes, that doesn't seem right. Here's a good post that describes how Prodigy's default active learning strategies work:

prefer_uncertain: how does it use the stream to pick examples to score?

You can choose between two algorithms for prefer_uncertain: prefer_uncertain(stream, algorithm='probability'): This randomly drops examples with probability proportional to distance from 0.5. So if an example is given a score 0.5 by the model, it will be dropped with probability close to 0.0 (so you’ll probably see it for annotation). If an example is given a score 0.99 or 0.01 by the model, it will probably be dropped. Probabilities are clipped so that even if all of the scores are 0.0 or 1.…

Can you provide the code and/or a reproducible example? Obviously, what I'd need is the model which isn't possible, but we can still debug.

Just curious, how are you structuring your data?

If it were me, for very long documents, I'd start with something like this:

{"text": "This is my first sentence.", meta: {"document": "A", "page": 0, "sentence": 0}}
{"text": "This is my second sentence.", meta: {"document": "A", "page": 0, "sentence": 1}}
...
{"text": "This is a sentence later on.", meta: {"document": "D", "page": 100, "sentence": 14}}
...
{"text": "This is my last sentence.", meta: {"document": "Z", "page": 999, "sentence": 99}}

All of the "text" keys are by sentence (e.g., perhaps you ran split_sentences before starting textcat.manual). Also by putting the document/page/sentence, hopefully you could always identify where the sentence is.

If you're still having issues, you can look "under the hood" at what textcat.teach is doing to try to debug what's going wrong (or customize it too). You can find a simple version of textcat.teach on our projects repo. This may be a good one to try as it's easier to play around with it (e.g., trying different sorters).

Alternatively, you can look at the actual recipe by find the textcat.py recipe that's inside of your installed Prodigy library. You can find the location of that library by finding Location: in python -m prodigy stats and then looking within the /recipes folder.

[OtterMe]

Thank you @ryanwesslen for such a quick reply !

My input data looks like this with about 10^5 entries :

{"text":"Sentence extracted using split_sentences indeed.","meta":{"document":A,"page":"1"}}

First I ran:

 prodigy textcat.manual annotated_db ./assets/input_data.jsonl --label MY_LABEL

Then:

python -m prodigy train ./output_dir --eval-split 0.2 --textcat-multilabel annotated_db --base-model en_core_web_sm

Finally:

prodigy textcat.teach ./output_dir/model-best annotated_db ./assets/input_data.jsonl --label MY_LABEL

The model is actually quite good after the manual step (about 200 annotations and 0.8 score) but as mentioned the behaviour during the teaching step is the same as manual. I do have a score provided for each sentences, but it reads my jsonl from entry 1 to entry n.

I'll take a look under the hood, because I might I've missed something since I haven't seen until now the prefer_uncertain thing.