# Best Practices for text classifier annotations

**URL:** <https://support.prodi.gy/t/best-practices-for-text-classifier-annotations/135>\
**Category:** Uncategorized\
**Tags:** usage, textcat, best-practices\
**Created:** [December 14, 2017, 5:55pm UTC](https://support.prodi.gy/t/best-practices-for-text-classifier-annotations/135 "2017-12-14T17:55:16Z")\
**Posts on this page:** 1\
**Showing post:** 4

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [December 17, 2017, 8:57pm UTC](https://support.prodi.gy/t/best-practices-for-text-classifier-annotations/135/4 "2017-12-17T20:57:55Z")

</div>

> [@justindujardin](#):
>
> Just so I’m clear, when you say the last `25%` you mean the `100%` reading, correct?

Yes, exactly, the last step between 75% and 100%. In your case, this actually looks very good – 13% improvement is pretty significant. So there's a high chance that you will keep seeing improvements if you collect more annotations similar to the ones you already have. It also confirms that your approach is very feasible, and that it makes sense to invest more and explore it further 😊

(Really glad to see this is working well so far btw – this type of analysis and being able to verify an idea quickly is one of the central use cases we've had in mind for Prodigy!)

> [@justindujardin](#):
>
> when Twitter updated the tweet length to 280, their API started returning compatibility mode tweets that link to the full-text message. The search API calls need to have `tweet_mode=extended` set to receive complete messages.

Thanks for the info, this is really helpful. I wrote the Twitter loader before Twitter introduced the 280 limit, and we actually launched Prodigy shortly after they announced the [new API offerings](https://blog.twitter.com/developer/en_us/topics/tools/2017/introducing-twitter-premium-apis.html). I've been meaning to look into this and see if it'll finally make the Twitter API less frustrating to work with. I'd love to drop the external dependency and use a more straightforward implementation.

Loaders are pretty simple, though, so you can also just write your own. All it needs to do is query the API and reformat the response to match Prodigy's JSON format:

```python
def custom_loader():
    page = 0 # if API is paged, keep a counter
    while True:
        r = requests.get('http://some-api', params={'page': page})
        response = r.json()
        for item in response['results']: # or however it's structured
            yield {'text': item['text']} # etc.
        page += 1 # after page is exhausted, increment

```

Once you move past the experimental stage, you might also want to consider scraping the tweets manually instead of streaming them in via the live API. This way, you're also less dependent on arbitrary limitations and what Twitter decides to show you or not show you – assuming you're using the free API. (After all, the docs explicitly state that the API is focused on ["relevance and not completeness"](https://developer.twitter.com/en/docs/tweets/search/overview/standard).)

> [@justindujardin](#):
>
> Can you tell me about how the `split_evals` function works?

Sorry – this should probably be documented better. `split_evals` takes an already shuffled list of annotated examples and an optional `eval_split` (set via the command line). If no `eval_split` is set, it defaults to `0.5` for datasets of under 1000 examples, and `0.2` for larger sets. The function then splits the examples accordingly, and returns a `(train_examples, eval_examples, eval_split)` triple.

> [@justindujardin](#):
>
> Removing the ignore examples speeds up my training sessions. What role do the ignore examples play in the training process?

The `ignore` examples have no further purpose except for being ignored. They're still stored in the database in case you want to use or analyse them later. For example, if you're working with external annotators, you might want to look at the examples they ended up ignoring to find out what was most unclear, and whether there were any problems with your data. In some cases, you might also want to go back and re-annotate them later on.

Now that I think about it, Prodigy should probably filter out the `ignore` examples before doing the shuffling and splitting. Will fix this in the next release 👍 Maybe we could also add an option to the `prodigy drop` command that lets you delete example with a certain answer, if that's helpful? For example, you could do `prodigy drop my_dataset --answer ignore` and it'd remove all ignores from the set.

---

_[View the full topic](https://support.prodi.gy/t/best-practices-for-text-classifier-annotations/135)._