# Is there a way to reduce the number of labels in the annotated dataset of prodi.gy?

**URL:** <https://support.prodi.gy/t/is-there-a-way-to-reduce-the-number-of-labels-in-the-annotated-dataset-of-prodi-gy/6113>\
**Category:** Uncategorized\
**Tags:** ner\
**Created:** [November 13, 2022, 7:35pm UTC](https://support.prodi.gy/t/is-there-a-way-to-reduce-the-number-of-labels-in-the-annotated-dataset-of-prodi-gy/6113 "2022-11-13T19:35:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rahul1](https://avatars.discourse-cdn.com/v4/letter/r/8e7dd6/32.png) [@rahul1](https://support.prodi.gy/u/rahul1)\
**Post date:** [November 13, 2022, 7:35pm UTC](https://support.prodi.gy/t/is-there-a-way-to-reduce-the-number-of-labels-in-the-annotated-dataset-of-prodi-gy/6113/1 "2022-11-13T19:35:27Z")

</div>

Hi,

My annotated data contains 24 labels, which I think are a lot. I would like to reduce the number labels, for example, change labels 'grant' and 'prize' to label 'award'. Another example would be to merge 'event', 'activity', 'output' 'product' to 'product'.

I used db-out command to export json file of the annotated data. I have looked on this forum to figure out next step of reducing the number of labels, but could not find it in recent posts.

regards  
Rahul

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [November 14, 2022, 1:57pm UTC](https://support.prodi.gy/t/is-there-a-way-to-reduce-the-number-of-labels-in-the-annotated-dataset-of-prodi-gy/6113/2 "2022-11-14T13:57:28Z")

</div>

Deciding on the labels of interest tends to be a manual process. That said, when you export your data nothing is stopping you from using a Python script to replace some of the labels.

The easiest way might be to use the Python API. Have you seen [this method](https://prodi.gy/docs/api-database#get-dataset-examples)? You could also write a script that takes examples from the "many labels" dataset, changes it, and then tells Prodigy to create a new "few labels" dataset so that you may proceed annotating from there.

## Advice

Before writing the script, I might advice having a meeting with some team members, annotators and other stakeholders to double-check if there's a clear consensus on what labels are of interest. It'd be a shame to rewrite a script, which is one reason to do this. But this might also be a great opportunity to understand the problem your team is trying to solve a bit better. It might also lead to an improved annotation [instructions](https://www.youtube.com/watch?v=E98okAEZn9U&ab_channel=Explosion).

---

<div class="post-metadata">

**Author:** ![rahul1](https://avatars.discourse-cdn.com/v4/letter/r/8e7dd6/32.png) [@rahul1](https://support.prodi.gy/u/rahul1)\
**Post date:** [December 5, 2022, 6:08am UTC](https://support.prodi.gy/t/is-there-a-way-to-reduce-the-number-of-labels-in-the-annotated-dataset-of-prodi-gy/6113/3 "2022-12-05T06:08:36Z")

</div>

**Renaming the labels in NER**

If you are not satisfied with the labels you gave during annotation, you can change via following process.

My annotated data contains 24 labels, which I think are a lot. I would like to reduce the number labels, for example, change labels ‘grant’ and ‘prize’ to label ‘award’. Another example would be to merge ‘event’, ‘activity’, ‘output’ ‘product’ to ‘product’.

_Step: export the annotated dataset as jsonl file by using following command in the terminal_

`prodigy db-out dataset > ./data.jsonl`

_Step: install jq via the terminal with code_

`sudo apt-get install jq`

_Step: use following code to change the labels_

`sed 's,"label":"OLD_LABEL","label":"NEW_LABEL",g' old_data.jsonl > new_data.jsonl`

You can reintegrate this new data jsonl file in the dataset of your prodigy database by command db-in:

`prodigy db-in new_dataset ./new_data.jsonl --rehash`