textcat.teach uncertain sorter show options with score 0

[ryanwesslen]

Hi @info2000!

For your three new questions, I wasn't sure if you're asking for only textcat.teach or Prodigy in general. The answers below should be the same for both.

@info2000

-how are filtered the repeated samples into a dataset?

I'm sorry, I don't understand the question. Can you rephrase?

If you're interested in the default behavior for deduplication, you may find the Docs for the filter_duplicates function to be helpful or in the answers below.

@info2000

-if 2 samples are the "same text" but the sample b has one space different than sample A, are threat like same sample and deduplicated?

For two texts where only difference is one space, would have different values for their input_hash.

Since they have different input_hash values (and also different task_hash), they would be treated as two different samples.

@info2000

-if I start prodigy with 100 samples file, label them, and restart with other file with 50 samples, but 30 samples was also included on the initial file, are these 30 samples removed/avoid ?

Let's assume for both sessions, you'll set your dataset to the same Prodigy dataset.

For annotation, by default records are deduped by task_hash, see "exclude_by" field in Configuration. Therefore, if you're doing the same task (e.g., textcat recipe) you did in the first 100 samples, then only 20 samples would be shown (30 repeated tasks would be excluded).

But if the 2nd annotation session used a different task (e.g., different recipe like ner), then all 50 samples would be shown.

@info2000

Why could this be happening?

Is low effective classify the same text +24 times (I had 24 labels, but should review the same tuple example-label multiple times)

Sorry we haven't responded yet to this. Are you still interested in an explanation for this? As Vincent had recommended, if we have a reproducible example, it makes it a lot easier for us to debug.

I've found some past posts that may explain shed some light. While uncommon, it's not impossible to see low scores when using the default ema algorithm:

if you get a long sequence of low-scoring examples, the probability sorter will ask you fewer questions from them, while the exponential moving average sorter will get impatient, and start asking you questions even if the scores are low.

Taking advantage of TONS of unlabeled data

You might want to implement your own sorter method for this. The function just needs to take a sequence of (score, example) tuples and yield out a sequence of examples. Another simple thing you could try is to use prefer_uncertain(stream, algorithm='probability'). The probability algorithm randomly drops examples proportional to their score. In contrast, the default sorter doesn’t trust that the scores produced by the model are going to stay well calibrated. It asks you about examples which are …

As an alternative, you may want to consider using algorithm = 'probability' as the post you originally cited mentioned:

On the other hand, if you know the target class is rare, you want the sorter to “believe” the scores much more. In this case the probability sorter is better.

I also found the discussion here to be helpful:

Prodigy Active Learning prefer_uncertain mechanism

The sorting mechanism is designed to account for large input data sets, so it works in a streaming fashion, sorting chunks of data. It's a tricky balancing act because we don't want to block the annotation feed while we search for the best examples. The standard deviation we refer to is a running estimate of the standard deviation. This is an approximation, the calculation is similar to making an exponential moving average estimation. It's the standard deviation of the specified "figure of meri…