# textcat.teach stream of data is linear ?

**URL:** <https://support.prodi.gy/t/textcat-teach-stream-of-data-is-linear/6432>\
**Category:** Uncategorized\
**Tags:** textcat\
**Created:** [March 14, 2023, 3:53pm UTC](https://support.prodi.gy/t/textcat-teach-stream-of-data-is-linear/6432 "2023-03-14T15:53:44Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [March 14, 2023, 7:12pm UTC](https://support.prodi.gy/t/textcat-teach-stream-of-data-is-linear/6432/2 "2023-03-14T19:12:46Z")

</div>

hi @OtterMe!

Thanks for your question and welcome to the Prodigy community 👋

> [@OtterMe](#):
>
> I would expect the sentences to be ambiguous one with around 0.5 score and from different documents/pages. However the stream of data is very linear (document A page 1 -\> 2 -\> 3...document B page 1 -\> 2....) with heterogenous probabilities.  
> Is there something that I don't understand stopping prodigy to stream the data inside a teach job as Document D page 100 -\> Document Z page 1 -\>Document A page 10 with all probabilities around 0.5 or so ?

Hmm... yes, that doesn't seem right. Here's a good post that describes how Prodigy's default active learning strategies work:

> [@prefer\_uncertain: how does it use the stream to pick examples to score?](https://support.prodi.gy/t/prefer-uncertain-how-does-it-use-the-stream-to-pick-examples-to-score/131/2):
>
> You can choose between two algorithms for prefer\_uncertain: prefer\_uncertain(stream, algorithm='probability'): This randomly drops examples with probability proportional to distance from 0.5. So if an example is given a score 0.5 by the model, it will be dropped with probability close to 0.0 (so you’ll probably see it for annotation). If an example is given a score 0.99 or 0.01 by the model, it will probably be dropped. Probabilities are clipped so that even if all of the scores are 0.0 or 1.…

Can you provide the code and/or a reproducible example? Obviously, what I'd need is the model which isn't possible, but we can still debug.

Just curious, how are you structuring your data?

If it were me, for very long documents, I'd start with something like this:

```python
{"text": "This is my first sentence.", meta: {"document": "A", "page": 0, "sentence": 0}}
{"text": "This is my second sentence.", meta: {"document": "A", "page": 0, "sentence": 1}}
...
{"text": "This is a sentence later on.", meta: {"document": "D", "page": 100, "sentence": 14}}
...
{"text": "This is my last sentence.", meta: {"document": "Z", "page": 999, "sentence": 99}}

```

All of the "text" keys are by sentence (e.g., perhaps you ran [`split_sentences`](https://prodi.gy/docs/api-components#split_sentences) before starting `textcat.manual`). Also by putting the document/page/sentence, hopefully you could always identify where the sentence is.

If you're still having issues, you can look "under the hood" at what `textcat.teach` is doing to try to debug what's going wrong (or customize it too). You can find a simple version of [`textcat.teach`](https://github.com/explosion/prodigy-recipes/blob/master/textcat/textcat_teach.py) on our `projects` repo. This may be a good one to try as it's easier to play around with it (e.g., trying different [sorters](https://prodi.gy/docs/api-components#sorters)).

Alternatively, you can look at the actual recipe by find the `textcat.py` recipe that's inside of your installed Prodigy library. You can find the location of that library by finding `Location:` in `python -m prodigy stats` and then looking within the `/recipes` folder.

---

_[View the full topic](https://support.prodi.gy/t/textcat-teach-stream-of-data-is-linear/6432)._