# textcat.teach not using active learning **URL:** \ **Category:** Uncategorized\ **Tags:** textcat, solved\ **Created:** [April 3, 2018, 5:36am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464 "2018-04-03T05:36:53Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![WeinstockShahar](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/weinstockshahar/32/115_2.png) [@WeinstockShahar](https://support.prodi.gy/u/WeinstockShahar)\ **Post date:** [April 3, 2018, 5:36am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/1 "2018-04-03T05:36:53Z")
I’m using `textcat.teach` and it seems that the active learning is not working. I went over a couple of hundred examples and they are still served in the original file order. I do see a different score for each example so I assume that there is some kind of learning, maybe it is just the serving? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [April 3, 2018, 11:29am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/2 "2018-04-03T11:29:03Z")
That’s very strange. The same `prefer_uncertain` sort function is used in both the `textcat.teach` and `ner.teach` recipes, so It’s hard to see what step things could be going wrong here. The sequence of operations is: 1. Stream data through spaCy. The `spacy.pipeline.TextCategorizer` pipeline component attaches scores in `doc.cats` 2. Pass through `prodigy.models.textcat.TextClassifier` wrapper. This simply translates the `doc` object back into a stream of `(score, example)` tuples. 3. The scored stream is then reranked using the `prefer_uncertain` function. `prefer_uncertain` accepts two strategies for finding “good” examples in a (potentially infinite) stream. The default strategy tracks the exponential moving average of the scores, and emits examples that are better than the average by some amount of variance. In the initial examples the estimates of the average and variance aren’t confident, so it doesn’t do much manipulation of the earliest examples. The second strategy is simpler: it uses the score as an emission probability, and uses a random variable to find whether to emit the example or whether to filter it. It would be helpful if you could try the second strategy by changing `stream = prefer_uncertain(stream)` to `stream = prefer_uncertain(stream, algorithm='probability')`. This will tell us whether the problem seems to be in the sorter, or somewhere upstream. Another useful thing to check is whether the scores being displayed to you (which are read from `eg['meta']['score']` actually match the scores the sorter sees, which come in as a tuple on the stream. That is, the model produces sequence of `(score, example)` tuples — that score is what the sorter responds to. I guess a possible bug would be that the actual scores being used to sort are all 0.5, and the ones being displayed are a lie. I’ve read the code again and that doesn’t seem possible, but hey — it could be the thing. If the sorter isn’t the problem and the scores really do vary, then I’m truly flummoxed: it doesn’t seem like there’s a third place for the bug to be. Could you also tell me whether you’re using patterns, and whether you’re using the long text mode? ---
**Author:** ![WeinstockShahar](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/weinstockshahar/32/115_2.png) [@WeinstockShahar](https://support.prodi.gy/u/WeinstockShahar)\ **Post date:** [April 3, 2018, 1:53pm UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/3 "2018-04-03T13:53:10Z")
> [@honnibal](#): > > It would be helpful if you could try the second strategy by changing stream = prefer\_uncertain(stream) to stream = prefer\_uncertain(stream, algorithm='probability'). This will tell us whether the problem seems to be in the sorter, or somewhere upstream. I changed it and the problem still exist. > [@honnibal](#): > > Could you also tell me whether you’re using patterns, and whether you’re using the long text mode? I'm using patterns and I'm not in long text mode. I will check the scores and get back with more results. ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [April 3, 2018, 1:54pm UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/4 "2018-04-03T13:54:39Z")
It must be the patterns interaction — that’s the newest thing we added to the recipe too. Are all the early matches coming off the pattern matcher? They should tell you a pattern number along with the score. ---
**Author:** ![WeinstockShahar](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/weinstockshahar/32/115_2.png) [@WeinstockShahar](https://support.prodi.gy/u/WeinstockShahar)\ **Post date:** [April 10, 2018, 5:06am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/5 "2018-04-10T05:06:19Z")
> [@honnibal](#): > > Are all the early matches coming off the pattern matcher? No. That's why I suspected the active learning. ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [April 10, 2018, 10:09am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/6 "2018-04-10T10:09:30Z")
Hmm. This doesn’t add up. So, all of the following are true? 1. The examples receive different scores from the model 2. No examples are skipped 3. You’ve tried both the default sorter, and the `algorithm="probability"` sorter The probability sorter is super simple — if the scores are different I don’t understand how it could it be failing to drop examples. Let’s make this even simpler. Instead of the `prefer_uncertain` function, wrap the stream in this: ```python import random def drop_examples(scored_stream): for score, example in scored_stream: # Always give examples some chance of being emited score = min(0.999, max(0.0001, score)) certainty = abs(score - 0.5) * 2 if random.random() >= certainty: yield example ``` If the scores differ, this can’t fail to drop examples, right? So we should mostly get examples with scores close to 0.5. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [April 10, 2018, 10:34am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/7 "2018-04-10T10:34:00Z")
Another suggestion: if you're not seeing any pattern matches, you might also want to test your patterns against spaCy's tokenization to ensure that they match as expected. If your patterns don't actually match and you're doing a `textcat.teach` cold start with a new category, your model likely won't learn anything meaningful, and Prodigy will keep suggesting you very random examples. This still doesn't explain the behaviour to describe – but it can easily make the whole thing more confusing. We've actually just released an experimental tool that should help with testing patterns: > [@sparkles Tip: Test your patterns with our new Matcher Explorer demo](https://support.prodi.gy/t/tip-test-your-patterns-with-our-new-matcher-explorer-demo/475): > > We just published a (still experimental) new demo that should hopefully make it easier to create and test complex token match patterns. I've wanted a debugging tool like this for a while, so I finally decided to build it smile Each box on the left represents one token – you can add as many token attributes and tokens as you like, including operators (to repeat or negate tokens). You can then enter your text and view the matches, check spaCy's tokenization, open a displaCy visualization to ve… ---
**Author:** ![WeinstockShahar](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/weinstockshahar/32/115_2.png) [@WeinstockShahar](https://support.prodi.gy/u/WeinstockShahar)\ **Post date:** [April 10, 2018, 10:59am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/8 "2018-04-10T10:59:58Z")
I might just found my problem, sorry if I wasted your time for nothing. I created a different input data file with ordered `example_id` to test it again and I see that some examples are skipped (1-2 examples each time). The old `example_id` wouldn’t let me see those small skips. Is this behavior sounds correct? ---
**Author:** ![honnibal](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/honnibal/32/35_2.png) [@honnibal](https://support.prodi.gy/u/honnibal)\ **Post date:** [April 10, 2018, 11:04am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/9 "2018-04-10T11:04:52Z")
@WeinstockShahar No problem! Glad it seems to be working — now I just hope it does useful things as well 🙂 The default sorter (based on the exponential moving average) takes a while to get confident in its estimates of the first and second moments. So, at first it might not skip many. The interaction with the pattern matcher also complicates things. Once the model warms up, it should be fairly selective in what it asks you. ---
**Author:** ![WeinstockShahar](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/weinstockshahar/32/115_2.png) [@WeinstockShahar](https://support.prodi.gy/u/WeinstockShahar)\ **Post date:** [April 10, 2018, 11:10am UTC](https://support.prodi.gy/t/textcat-teach-not-using-active-learning/464/10 "2018-04-10T11:10:25Z")
Thanks a lot 🙂