# How to evaluate the model accuracy with test data (not part of training)

**URL:** <https://support.prodi.gy/t/how-to-evaluate-the-model-accuracy-with-test-data-not-part-of-training/7114>\
**Category:** Uncategorized\
**Tags:** usage, ner, spacy\
**Created:** [February 22, 2024, 2:16pm UTC](https://support.prodi.gy/t/how-to-evaluate-the-model-accuracy-with-test-data-not-part-of-training/7114 "2024-02-22T14:16:13Z")\
**Posts on this page:** 1\
**Showing post:** 8

<div class="post-metadata">

**Author:** ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)\
**Post date:** [March 5, 2024, 2:57pm UTC](https://support.prodi.gy/t/how-to-evaluate-the-model-accuracy-with-test-data-not-part-of-training/7114/8 "2024-03-05T14:57:19Z")

</div>

hi @e101sg,

> [@e101sg](#):
>
> I am sure among 86 entries no duplicates. Also, i have examined the annotations using the `db-out` and export into `.jsonl` file . I can see same number (86) of `_input_hash'` and `_task_hash`. Is it normal ?

Where are you getting the `86` number from?

Your output had these numbers:

```python
============================== Generating data ==============================
Components: ner
Merging training and evaluation data for 1 components
  - [ner] Training: 590 | Evaluation: 84 (from datasets)
Training: 327 | Evaluation: 44
Labels: ner (7)
✔ Saved 327 training examples
spacy_model/train.spacy
✔ Saved 44 evaluation examples
spacy_model/dev.spacy

```

This shows 84 an input evaluation number of 84. Where is the 86 coming from?

I suspect they are in the dataset `evaluation_dataset_gold`. I'm wondering if you had rejected 2 of the examples, hence why it's 84 not 86.

Can you run?

```python
python -m prodigy stats -l evaluation_dataset_gold

```

What do you get? It should show the number of annotations by ACCEPT, REJECT and IGNORE.

Unfortunately, I still think the 84 -\> 44 are still due to duplicates. Are you using [overlapping annotations](https://prodi.gy/docs/api-web-app#multi-user-sessions)? Maybe you could try `"exclude_by": "input"` (see below), but I don't think that would do anything.

It may also be related to [this](https://support.prodi.gy/t/difference-between-input-hash-and-task-hash/3220/2):

> For example, `data-to-spacy` will group all annotations with the same input hash together, so you'll get one example annotated with all categories, entities, POS tags or whatever else you labelled.

> [@e101sg](#):
>
> The text chunks usually shorter but long sentence as well. Is it okay? any suggestion.

I don't see any big problems or at least relating to your dedupes.

> [@e101sg](#):
>
> Also the documents says about the **exclude\_by** , Did not understand what it means or not sure it is useful for my **ner.manual** task.

This is how deduplication is done. If set in default (`"exclude_by": "task"`), this means deduplication occurs for a task (individual input + unique task run). Alternatively, `"exclude": "input"` would mean deduplication strictly based on `input`. This essentially means is deduplication done by the `task_hash` (i.e., `"exclude_by": "task"`) or by `input_hash` (i.e., `"exclude_by": "input"`).

Can you read through this:

> [@Help! I have duplicates or missing data: Best practices on accounting for annotations](https://support.prodi.gy/t/help-i-have-duplicates-or-missing-data-best-practices-on-accounting-for-annotations/6704#h-2-if-youre-missing-data-do-you-have-duplicates-in-your-input-source-):
>
> Scaling annotations can be really hard. More scale means more complexity, which can make accounting for all of those annotations challenging, time-consuming, overwhelming, error-prone, or even impractical. Unfortunately, many ML engineers and developers don’t realize this until they scale their annotations, risking wasted time and effort after labeling hundreds or thousands of annotations. As you dig deeper, you'll also find trade-offs from memory to network/latency to human-centered issues wit…

Namely this post:

> [@Difference between Input hash and task hash](https://support.prodi.gy/t/difference-between-input-hash-and-task-hash/3220/2):
>
> Hi! You can read more about the hashing here: [https://prodi.gy/docs/api-loaders#hashing](https://prodi.gy/docs/api-loaders#hashing) A simple example: you might be annotating data for text classification with two categories, so you make two passes over the data, one with LABEL\_A and one with LABEL\_B. Each time, you accept/reject whether the label applies. This means you end up with two examples for each text: one with LABEL\_A and one with LABEL\_B. Both examples will have the same input hash, because they're questions about the same input …

---

_[View the full topic](https://support.prodi.gy/t/how-to-evaluate-the-model-accuracy-with-test-data-not-part-of-training/7114)._