textcat.teach stream of data is linear ?

[ryanwesslen]

hi @OtterMe!

Thanks for your question and welcome to the Prodigy community

@OtterMe

I would expect the sentences to be ambiguous one with around 0.5 score and from different documents/pages. However the stream of data is very linear (document A page 1 -> 2 -> 3...document B page 1 -> 2....) with heterogenous probabilities.
Is there something that I don't understand stopping prodigy to stream the data inside a teach job as Document D page 100 -> Document Z page 1 ->Document A page 10 with all probabilities around 0.5 or so ?

Hmm... yes, that doesn't seem right. Here's a good post that describes how Prodigy's default active learning strategies work:

prefer_uncertain: how does it use the stream to pick examples to score?

You can choose between two algorithms for prefer_uncertain: prefer_uncertain(stream, algorithm='probability'): This randomly drops examples with probability proportional to distance from 0.5. So if an example is given a score 0.5 by the model, it will be dropped with probability close to 0.0 (so you’ll probably see it for annotation). If an example is given a score 0.99 or 0.01 by the model, it will probably be dropped. Probabilities are clipped so that even if all of the scores are 0.0 or 1.…

Can you provide the code and/or a reproducible example? Obviously, what I'd need is the model which isn't possible, but we can still debug.

Just curious, how are you structuring your data?

If it were me, for very long documents, I'd start with something like this:

{"text": "This is my first sentence.", meta: {"document": "A", "page": 0, "sentence": 0}}
{"text": "This is my second sentence.", meta: {"document": "A", "page": 0, "sentence": 1}}
...
{"text": "This is a sentence later on.", meta: {"document": "D", "page": 100, "sentence": 14}}
...
{"text": "This is my last sentence.", meta: {"document": "Z", "page": 999, "sentence": 99}}

All of the "text" keys are by sentence (e.g., perhaps you ran split_sentences before starting textcat.manual). Also by putting the document/page/sentence, hopefully you could always identify where the sentence is.

If you're still having issues, you can look "under the hood" at what textcat.teach is doing to try to debug what's going wrong (or customize it too). You can find a simple version of textcat.teach on our projects repo. This may be a good one to try as it's easier to play around with it (e.g., trying different sorters).

Alternatively, you can look at the actual recipe by find the textcat.py recipe that's inside of your installed Prodigy library. You can find the location of that library by finding Location: in python -m prodigy stats and then looking within the /recipes folder.