# Task routers - Problem in JSONL output/Annotators Problem

**URL:** <https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702>\
**Category:** Uncategorized\
**Tags:** ner\
**Created:** [July 25, 2023, 4:05am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702 "2023-07-25T04:05:16Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinoth](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vinoth/32/4375_2.png) [@Vinoth](https://support.prodi.gy/u/Vinoth)\
**Post date:** [July 25, 2023, 4:05am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/1 "2023-07-25T04:05:16Z")

</div>

I used the task routers for annotating a sample of 30 chunks using recipe _ner.manual_.

Followed the task routers documentation, first created the `data.jsonl` with 30 chunks. Then, updated `prodigy.json` with `annotation_per_task` as 3.

My annotator pool has 5 annotators where atleast 3 annotators should see the same chunks as per `prodigy.json` config file. This is a situation of `partial overlap`.

Out of 5 annotators, only 2 annotators were available at that time. So, `user1` got 30 chunks who started early for annotating the chunks and `user2` got 9 chunks who started after 10 minutes of 'user1. Following queries are:

1. Once annotation is done, using `prodigy db-out <dataset> <directory>`, I could see count of 30 chunks, that too annotated only by `user1` found with the help of `"_annotator_id":"group2_v3-user1"` and not by `'user2`. Here `user2` handles 9 chunks, those records are not there.  
Am I failing to do anything? Please comment on this.

2. I am assuming that, as `user1 and user2` at that moment and finished the annotation. So, now `user3, user4 and user5` can't see any chunks. Is this scenario will face if we use Task Routing? or Later `user3, user4 and user5` can see their dedicated chunks eventhough `user1, user2` finished their allotted chunks.

**prodigy.json**

```python
{
    "theme": "basic",
    "custom_theme": {},
    "batch_size": 10,
    "port": xxxx,
    "host": "xxx.yy.zzz.aaa",
    "cors": true,
    "db": "sqlite",
    "db_settings":{"sqlite": 
			{
			    "name": "prodigy.db",
			    "path": "path/to/my/prodigy"
		        }},
    "api_keys": {},
    "validate": false,
    "auto_create": true,
    "auto_exclude_current": true,
    "instant_submit": false,
    "annotation_per_task": 3,
    "show_stats": true,
    "hide_meta": false,
    "show_flag": false,
    "instructions": false,
    "swipe": false,
    "split_sents_threshold":5000,
    "diff_style": "words",
    "html_template": false,
    "global_css": null,
    "javascript": null,
    "writing_dir": "ltr",
    "hide_true_newline_tokens": false,
    "ner_manual_require_click": false,
    "ner_manual_label_style": "list",
    "choice_style": "single",
    "choice_auto_accept": false,
    "darken_image": 0,
    "show_bounding_box_center": false,
    "preview_bounding_boxes": false,
    "shade_bounding_boxes": false
}

```

I hope this helps to answer my queries.

Thanks in advance  
Vinoth Kumar S

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [July 25, 2023, 8:21am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/2 "2023-07-25T08:21:30Z")

</div>

Just to check, did you set the `PRODIGY_ALLOWED_SESSIONS` environment variable upfront so Prodigy is aware of all the annotators? If not, there's this caveat to be aware of.

[![](https://img.youtube.com/vi/vyOtq-UXP-E/maxresdefault.jpg "TASK ROUTERS in Prodigy v1.12 - Deep Dive") ](https://www.youtube.com/watch?v=vyOtq-UXP-E&t=754)

It's also explained in detail on the docs.

> **[Task Routing · Prodigy · An annotation tool for AI, Machine Learning & NLP](https://prodi.gy/docs/task-routing/)**
>
> A downloadable annotation tool for NLP and computer vision tasks such as named entity recognition, text classification, object detection, image segmentation, A/B evaluation and more.

That said, I may have spotted the issue in your configuration file. You seem to have configured `annotation_per_task`. The setting you want is `annotations_per_task`, there's an extra "s" in "annotations". I would've expected the validation system to catch this though, so I'll do a deep dive to figure out what may have gone wrong there.

Could you confirm if the `annotations_per_task` setting fixes things? If not I'll gladly dive deeper into this, but I'd like to rule out this issue is caused by a typo first.

---

<div class="post-metadata">

**Author:** ![Vinoth](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vinoth/32/4375_2.png) [@Vinoth](https://support.prodi.gy/u/Vinoth)\
**Post date:** [July 25, 2023, 11:24am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/3 "2023-07-25T11:24:24Z")

</div>

Hi Vincent,

1. I am always enabling `PRODIGY_ALLOWED_SESSIONS='user1,user2,user3,user4,user5'`. So that prodigy aware about the sessions going to handle.

2. changed to `annotations_per_task` (added 's')

Before solving the issues, is new updation of Task Routing works with v1.11??

Total number chunks: 30. Prodigy splits 30 as below:  
user1 --\> 18  
user2 --\> 10  
user3 --\> 7  
user4 --\> 0 (as he accessed the link after completion of other users)  
user5 --\> 15

Issues raised after your suggestions:

1. After giving the command prodigy db-out , I checked the JSONL file where only user1 and user5 tagging was there. w.r.t user1, only 10 counts and user5, 20 counts which are against the number what user1 and user5 tagged.

Where are the chunks tagged by user2 and user3? Also, chunks tagged by user1 ?

1. Here 4 annotators tagged the data, so output should contains more than actual data i.e 30.

Can you please help on this?

Thanks

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [July 25, 2023, 12:17pm UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/4 "2023-07-25T12:17:28Z")

</div>

Before diving into this further it might be good to highlight two mechanisms that might be at play here.

## Mechanism 1: Hashing

As the video explains, we're using a hashing trick to allocate the examples consistently. In the long run, these hashes will cause an even distribution, but even then, not a perfect one. That's because the hashes that we use, while evenly distributed, can still be seen as "random". So this might explain some of the deviations that you're seeing here.

If you're interested in checking this, you should be able to turn on [logging](https://prodi.gy/docs/install#debugging-logging) (set it to `verbose`) and see output appear whenever an allocation is made by one of our internal task routers.

## Mechanism 2: Work Stealing

Prodigy also comes with a work-stealing mechanic, which is described at [the bottom of the task routers docs](https://prodi.gy/docs/task-routing#work-stealing).

 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/9/9945b0ab6ffa251f297ba10185912d563b0e6c26.png)

To quote what is mentioned there:

> Besides task routing, Prodigy also offers a work stealing mechanic that might also influence who will annotate the examples. Work stealing occurs whenever an annotator has an empty queue and can use the queue of another user to keep annotating.
> 
> Work stealing is a preventive mechanism to avoid the loss of records in a stream. There might be a situation where an annotator requests a batch of examples for annotation, effectively locking those examples, but never actually annotates them. Work stealing enables annotators who reach the end of a shared stream to annotate these locked examples.

So it could also be that work stealing is causing the imbalance. You can manually force it off by setting `"allow_work_stealing"` to `false` in `prodigy.json`.

I'll now dive into some of your questions.

> [@Vinoth](#):
>
> Before solving the issues, is new updation of Task Routing works with v1.11??

Task routers were introduced in `v1.12` and they are not available in earlier versions. Is there a reason why you need v1.11?

> [@Vinoth](#):
>
> Where are the chunks tagged by user2 and user3? Also, chunks tagged by user1 ?

It's not entirely clear what you mean with "chunk" here, but I'm assuming that you're referring to an example that needs to get annotated.

Is it possible for you to run Prodigy one more time but with the verbose logging turned on? That might give us more of a window to see what is happening on your end. Theoretically it is possible that you have duplicates in your `examples.jsonl` file too, which would also ensure less examples being annotated than you might expect. Another reason why some of the annotations might not show up is that some of the user forgot to hit "save" before signing off. The logs should help give us a hint.

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [July 25, 2023, 12:46pm UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/5 "2023-07-25T12:46:03Z")

</div>

I just tried reproducing this locally, and while I wasn't able to find any issues, I figured that it couldn't hurt to share my findings and steps taken.

I started with this dataset.

```python
{"text":"1"}
{"text":"2"}
{"text":"3"}
{"text":"4"}
{"text":"5"}
{"text":"6"}
{"text":"7"}
{"text":"8"}
{"text":"9"}
{"text":"10"}
{"text":"11"}
{"text":"12"}
{"text":"13"}
{"text":"14"}
{"text":"15"}
{"text":"16"}
{"text":"17"}
{"text":"18"}
{"text":"19"}
{"text":"20"}
{"text":"21"}
{"text":"22"}
{"text":"23"}
{"text":"24"}
{"text":"25"}
{"text":"26"}
{"text":"27"}
{"text":"28"}
{"text":"29"}
{"text":"30"}

```

It's really just a dummy dataset that has exactly 30 rows. This dataset was used by this recipe call:

```python
PRODIGY_ALLOWED_SESSIONS="user1,user2,user3,user4,user5" PRODIGY_CONFIG_OVERRIDES='{"annotations_per_task": 3, "allow_work_stealing": false}' PRODIGY_LOGGING=verbose python -m prodigy ner.manual issue-6702 en_core_web_sm examples-30.jsonl --label number

```

Notice that I'm setting the sessions upfront, assigning 3 annotations per task and disallowing work stealing. I'm also turning on the verbose logs.

Next, I start a browser and open up five tabs, one for each user.

 ![CleanShot 2023-07-25 at 14.39.09](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/2/2e5ecca8c8792f1f979b81a83af53764f0f00f37.jpeg)

As I open up these tabs, I also see logs appear. These are the logs for the first time that the task router triggered, for `user1`.

```python
14:38:57: ROUTER: Routing item with _input_hash=-2045454197 -> ['issue-6702-2-user4', 'issue-6702-2-user5', 'issue-6702-2-user2']
14:38:57: ROUTER: Routing item with _input_hash=-784123405 -> ['issue-6702-2-user1', 'issue-6702-2-user5', 'issue-6702-2-user4']
14:38:57: ROUTER: Routing item with _input_hash=-805513229 -> ['issue-6702-2-user2', 'issue-6702-2-user5', 'issue-6702-2-user3']
14:38:57: ROUTER: Routing item with _input_hash=-1835389134 -> ['issue-6702-2-user2', 'issue-6702-2-user4', 'issue-6702-2-user1']
14:38:57: ROUTER: Routing item with _input_hash=-1991218384 -> ['issue-6702-2-user2', 'issue-6702-2-user1', 'issue-6702-2-user5']
14:38:57: ROUTER: Routing item with _input_hash=-1639312927 -> ['issue-6702-2-user4', 'issue-6702-2-user2', 'issue-6702-2-user5']
14:38:57: ROUTER: Routing item with _input_hash=372028165 -> ['issue-6702-2-user1', 'issue-6702-2-user3', 'issue-6702-2-user4']
14:38:58: ROUTER: Routing item with _input_hash=2018066853 -> ['issue-6702-2-user4', 'issue-6702-2-user2', 'issue-6702-2-user1']
14:38:58: ROUTER: Routing item with _input_hash=42709192 -> ['issue-6702-2-user3', 'issue-6702-2-user1', 'issue-6702-2-user4']
14:38:58: ROUTER: Routing item with _input_hash=1254603855 -> ['issue-6702-2-user1', 'issue-6702-2-user5', 'issue-6702-2-user2']
14:38:58: ROUTER: Routing item with _input_hash=-1789291377 -> ['issue-6702-2-user4', 'issue-6702-2-user5', 'issue-6702-2-user1']
14:38:58: ROUTER: Routing item with _input_hash=1626462944 -> ['issue-6702-2-user5', 'issue-6702-2-user1', 'issue-6702-2-user4']
14:38:58: ROUTER: Routing item with _input_hash=855192632 -> ['issue-6702-2-user3', 'issue-6702-2-user1', 'issue-6702-2-user5']

```

You'll notice that it keeps polling until it has 10 examples for user1. That's why there are more than 10 lines. The tasks are distributed somewhat evenly, but not perfectly because of the hashing.

I proceeded by annotating all of these examples by simply hitting accept everywhere. Then, I stop the recipe and call `db-out`. That results in the following output:

```python
{"text":"2","_input_hash":-784123405,"_task_hash":985633209,"_is_binary":false,"tokens":[{"text":"2","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288501,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"4","_input_hash":-1835389134,"_task_hash":678744456,"_is_binary":false,"tokens":[{"text":"4","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288502,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"5","_input_hash":-1991218384,"_task_hash":-1581206528,"_is_binary":false,"tokens":[{"text":"5","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288502,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"7","_input_hash":372028165,"_task_hash":1401556973,"_is_binary":false,"tokens":[{"text":"7","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288502,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"8","_input_hash":2018066853,"_task_hash":-1154883125,"_is_binary":false,"tokens":[{"text":"8","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288502,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"9","_input_hash":42709192,"_task_hash":811274923,"_is_binary":false,"tokens":[{"text":"9","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288503,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"10","_input_hash":1254603855,"_task_hash":16089945,"_is_binary":false,"tokens":[{"text":"10","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288503,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"11","_input_hash":-1789291377,"_task_hash":-474789062,"_is_binary":false,"tokens":[{"text":"11","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288503,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"12","_input_hash":1626462944,"_task_hash":2018616092,"_is_binary":false,"tokens":[{"text":"12","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288503,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"13","_input_hash":855192632,"_task_hash":335429472,"_is_binary":false,"tokens":[{"text":"13","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288504,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"14","_input_hash":-423927864,"_task_hash":-311834425,"_is_binary":false,"tokens":[{"text":"14","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288504,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"15","_input_hash":-902457195,"_task_hash":185203310,"_is_binary":false,"tokens":[{"text":"15","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288504,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"17","_input_hash":933542244,"_task_hash":-1066193932,"_is_binary":false,"tokens":[{"text":"17","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288505,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"19","_input_hash":2035079888,"_task_hash":30789149,"_is_binary":false,"tokens":[{"text":"19","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288505,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"21","_input_hash":-1139507680,"_task_hash":-1258641755,"_is_binary":false,"tokens":[{"text":"21","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288505,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"22","_input_hash":1765252975,"_task_hash":-1221112846,"_is_binary":false,"tokens":[{"text":"22","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288505,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"25","_input_hash":230261373,"_task_hash":329569934,"_is_binary":false,"tokens":[{"text":"25","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288506,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"27","_input_hash":-738557658,"_task_hash":613896265,"_is_binary":false,"tokens":[{"text":"27","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288506,"_annotator_id":"issue-6702-user1","_session_id":"issue-6702-user1"}
{"text":"1","_input_hash":-2045454197,"_task_hash":1182163795,"_is_binary":false,"tokens":[{"text":"1","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288508,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"3","_input_hash":-805513229,"_task_hash":1162353785,"_is_binary":false,"tokens":[{"text":"3","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288509,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"4","_input_hash":-1835389134,"_task_hash":678744456,"_is_binary":false,"tokens":[{"text":"4","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288509,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"5","_input_hash":-1991218384,"_task_hash":-1581206528,"_is_binary":false,"tokens":[{"text":"5","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288509,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"6","_input_hash":-1639312927,"_task_hash":-777789492,"_is_binary":false,"tokens":[{"text":"6","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288510,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"8","_input_hash":2018066853,"_task_hash":-1154883125,"_is_binary":false,"tokens":[{"text":"8","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288510,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"10","_input_hash":1254603855,"_task_hash":16089945,"_is_binary":false,"tokens":[{"text":"10","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288510,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"14","_input_hash":-423927864,"_task_hash":-311834425,"_is_binary":false,"tokens":[{"text":"14","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288510,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"15","_input_hash":-902457195,"_task_hash":185203310,"_is_binary":false,"tokens":[{"text":"15","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288511,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"16","_input_hash":-343056218,"_task_hash":96950103,"_is_binary":false,"tokens":[{"text":"16","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288511,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"17","_input_hash":933542244,"_task_hash":-1066193932,"_is_binary":false,"tokens":[{"text":"17","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288511,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"18","_input_hash":-1213183927,"_task_hash":-556031871,"_is_binary":false,"tokens":[{"text":"18","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288511,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"20","_input_hash":-883828987,"_task_hash":-1493531608,"_is_binary":false,"tokens":[{"text":"20","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288512,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"21","_input_hash":-1139507680,"_task_hash":-1258641755,"_is_binary":false,"tokens":[{"text":"21","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288512,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"23","_input_hash":348472867,"_task_hash":-1683186191,"_is_binary":false,"tokens":[{"text":"23","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288512,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"24","_input_hash":1124906533,"_task_hash":1709963953,"_is_binary":false,"tokens":[{"text":"24","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288512,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"25","_input_hash":230261373,"_task_hash":329569934,"_is_binary":false,"tokens":[{"text":"25","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288513,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"28","_input_hash":-165836021,"_task_hash":-1541557956,"_is_binary":false,"tokens":[{"text":"28","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288513,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"29","_input_hash":-539359274,"_task_hash":1560417868,"_is_binary":false,"tokens":[{"text":"29","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288513,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"30","_input_hash":1814743553,"_task_hash":-550346988,"_is_binary":false,"tokens":[{"text":"30","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288514,"_annotator_id":"issue-6702-user2","_session_id":"issue-6702-user2"}
{"text":"3","_input_hash":-805513229,"_task_hash":1162353785,"_is_binary":false,"tokens":[{"text":"3","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288519,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"7","_input_hash":372028165,"_task_hash":1401556973,"_is_binary":false,"tokens":[{"text":"7","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288519,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"9","_input_hash":42709192,"_task_hash":811274923,"_is_binary":false,"tokens":[{"text":"9","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288520,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"13","_input_hash":855192632,"_task_hash":335429472,"_is_binary":false,"tokens":[{"text":"13","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288520,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"14","_input_hash":-423927864,"_task_hash":-311834425,"_is_binary":false,"tokens":[{"text":"14","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288520,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"15","_input_hash":-902457195,"_task_hash":185203310,"_is_binary":false,"tokens":[{"text":"15","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288520,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"16","_input_hash":-343056218,"_task_hash":96950103,"_is_binary":false,"tokens":[{"text":"16","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288521,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"22","_input_hash":1765252975,"_task_hash":-1221112846,"_is_binary":false,"tokens":[{"text":"22","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288521,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"23","_input_hash":348472867,"_task_hash":-1683186191,"_is_binary":false,"tokens":[{"text":"23","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288521,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"24","_input_hash":1124906533,"_task_hash":1709963953,"_is_binary":false,"tokens":[{"text":"24","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288521,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"26","_input_hash":1310182142,"_task_hash":266669984,"_is_binary":false,"tokens":[{"text":"26","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288522,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"27","_input_hash":-738557658,"_task_hash":613896265,"_is_binary":false,"tokens":[{"text":"27","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288522,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"29","_input_hash":-539359274,"_task_hash":1560417868,"_is_binary":false,"tokens":[{"text":"29","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288522,"_annotator_id":"issue-6702-user3","_session_id":"issue-6702-user3"}
{"text":"1","_input_hash":-2045454197,"_task_hash":1182163795,"_is_binary":false,"tokens":[{"text":"1","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288527,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"2","_input_hash":-784123405,"_task_hash":985633209,"_is_binary":false,"tokens":[{"text":"2","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288528,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"4","_input_hash":-1835389134,"_task_hash":678744456,"_is_binary":false,"tokens":[{"text":"4","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288528,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"6","_input_hash":-1639312927,"_task_hash":-777789492,"_is_binary":false,"tokens":[{"text":"6","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288528,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"7","_input_hash":372028165,"_task_hash":1401556973,"_is_binary":false,"tokens":[{"text":"7","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288528,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"8","_input_hash":2018066853,"_task_hash":-1154883125,"_is_binary":false,"tokens":[{"text":"8","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288529,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"9","_input_hash":42709192,"_task_hash":811274923,"_is_binary":false,"tokens":[{"text":"9","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288529,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"11","_input_hash":-1789291377,"_task_hash":-474789062,"_is_binary":false,"tokens":[{"text":"11","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288529,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"12","_input_hash":1626462944,"_task_hash":2018616092,"_is_binary":false,"tokens":[{"text":"12","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288530,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"16","_input_hash":-343056218,"_task_hash":96950103,"_is_binary":false,"tokens":[{"text":"16","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288530,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"18","_input_hash":-1213183927,"_task_hash":-556031871,"_is_binary":false,"tokens":[{"text":"18","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288530,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"19","_input_hash":2035079888,"_task_hash":30789149,"_is_binary":false,"tokens":[{"text":"19","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288530,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"20","_input_hash":-883828987,"_task_hash":-1493531608,"_is_binary":false,"tokens":[{"text":"20","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288531,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"24","_input_hash":1124906533,"_task_hash":1709963953,"_is_binary":false,"tokens":[{"text":"24","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288531,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"25","_input_hash":230261373,"_task_hash":329569934,"_is_binary":false,"tokens":[{"text":"25","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288531,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"26","_input_hash":1310182142,"_task_hash":266669984,"_is_binary":false,"tokens":[{"text":"26","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288531,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"27","_input_hash":-738557658,"_task_hash":613896265,"_is_binary":false,"tokens":[{"text":"27","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288532,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"28","_input_hash":-165836021,"_task_hash":-1541557956,"_is_binary":false,"tokens":[{"text":"28","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288532,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"29","_input_hash":-539359274,"_task_hash":1560417868,"_is_binary":false,"tokens":[{"text":"29","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288532,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"30","_input_hash":1814743553,"_task_hash":-550346988,"_is_binary":false,"tokens":[{"text":"30","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288532,"_annotator_id":"issue-6702-user4","_session_id":"issue-6702-user4"}
{"text":"1","_input_hash":-2045454197,"_task_hash":1182163795,"_is_binary":false,"tokens":[{"text":"1","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288535,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"2","_input_hash":-784123405,"_task_hash":985633209,"_is_binary":false,"tokens":[{"text":"2","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288536,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"3","_input_hash":-805513229,"_task_hash":1162353785,"_is_binary":false,"tokens":[{"text":"3","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288536,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"5","_input_hash":-1991218384,"_task_hash":-1581206528,"_is_binary":false,"tokens":[{"text":"5","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288536,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"6","_input_hash":-1639312927,"_task_hash":-777789492,"_is_binary":false,"tokens":[{"text":"6","start":0,"end":1,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288536,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"10","_input_hash":1254603855,"_task_hash":16089945,"_is_binary":false,"tokens":[{"text":"10","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288537,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"11","_input_hash":-1789291377,"_task_hash":-474789062,"_is_binary":false,"tokens":[{"text":"11","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288537,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"12","_input_hash":1626462944,"_task_hash":2018616092,"_is_binary":false,"tokens":[{"text":"12","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288537,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"13","_input_hash":855192632,"_task_hash":335429472,"_is_binary":false,"tokens":[{"text":"13","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288537,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"17","_input_hash":933542244,"_task_hash":-1066193932,"_is_binary":false,"tokens":[{"text":"17","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288538,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"18","_input_hash":-1213183927,"_task_hash":-556031871,"_is_binary":false,"tokens":[{"text":"18","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288538,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"19","_input_hash":2035079888,"_task_hash":30789149,"_is_binary":false,"tokens":[{"text":"19","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288538,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"20","_input_hash":-883828987,"_task_hash":-1493531608,"_is_binary":false,"tokens":[{"text":"20","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288539,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"21","_input_hash":-1139507680,"_task_hash":-1258641755,"_is_binary":false,"tokens":[{"text":"21","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288539,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"22","_input_hash":1765252975,"_task_hash":-1221112846,"_is_binary":false,"tokens":[{"text":"22","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288539,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"23","_input_hash":348472867,"_task_hash":-1683186191,"_is_binary":false,"tokens":[{"text":"23","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288539,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"26","_input_hash":1310182142,"_task_hash":266669984,"_is_binary":false,"tokens":[{"text":"26","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288540,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"28","_input_hash":-165836021,"_task_hash":-1541557956,"_is_binary":false,"tokens":[{"text":"28","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288540,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}
{"text":"30","_input_hash":1814743553,"_task_hash":-550346988,"_is_binary":false,"tokens":[{"text":"30","start":0,"end":2,"id":0,"ws":false}],"_view_id":"ner_manual","answer":"accept","_timestamp":1690288540,"_annotator_id":"issue-6702-user5","_session_id":"issue-6702-user5"}

```

It's 90 lines, which is exactly what I'd expect given that there are three annotations per example (3 x 30 = 90). Next, I'll perform a count on that dataset.

```python
import polars as pl 

pl.read_ndjson("issue-6702.jsonl").groupby("_annotator_id").agg(pl.col("_input_hash").count())

```

This yields the following table.

```python
┌──────────────────┬─────────────┐
│ _annotator_id ┆ _input_hash │
│ --- ┆ --- │
│ str ┆ u32 │
╞══════════════════╪═════════════╡
│ issue-6702-user4 ┆ 20 │
│ issue-6702-user3 ┆ 13 │
│ issue-6702-user1 ┆ 18 │
│ issue-6702-user2 ┆ 20 │
│ issue-6702-user5 ┆ 19 │
└──────────────────┴─────────────┘

```

The distribution isn't perfect, again because of the hashing, but it doesn't feel out of bounds. The reason why we use hashes here, instead of round-robin, has to do with the consistent mapping. The hashing trick _really_ guarantees that a specific hash is mapped to a specific user, even if the order of the stream were to change. With a round-robin approach the allocation might change after the server restarts and new data is added.

Out of curiosity, if you were to follow these steps, do you see something different?

---

<div class="post-metadata">

**Author:** ![Vinoth](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vinoth/32/4375_2.png) [@Vinoth](https://support.prodi.gy/u/Vinoth)\
**Post date:** [July 25, 2023, 6:49pm UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/6 "2023-07-25T18:49:43Z")

</div>

Hi Vincent,

Thanks for the replies. I am working on the same. Initially I tried Task Routers with v1.11, thats the reason faced the problem.

Today we purchased for current version(1.12). Installed successfully in my machine.

By using same config file prodigy.json with parameter "annotations\_per\_task": 3 and used the command **PRODIGY\_ALLOWED\_SESSIONS="user1,user2,user3,user4,user5" python -m prodigy ner.manual group4 blank:en data2.jsonl --label LABEL1,LABEL2**

_Am getting the below error:_  
_'PRODIGY\_ALLOWED\_SESSIONS' is not recognized as an internal or external command, operable program or batch file._

So added the path `C:\Users\xxx\xxx\xxx\Python\Python311\site-packages\prodigy` in environment variable. Still facing the same error.

If eliminates the "annotations\_per\_task" from the config file and ran the simple command `python -m prodigy ner.manual sample1 blank:en data2.jsonl --label LABEL1,LABEL2`, it works and hosted.

Only facing problem with `PRODIGY_ALLOWED_SESSIONS` and other environment variables as well such as `export PRODIGY HOME`,` export PRODIGY_CONFIG` and so on.

Also tried to set the environment variable manually in CMD prompt:

conda env config vars set my\_var=PRODIGY\_ALLOWED\_SESSIONS  
reactivated the environment: conda activate test-env  
conda env config vars list  
output \>\> my\_var = PRODIGY\_ALLOWED\_SESSIONS

Still facing the error, _'PRODIGY\_ALLOWED\_SESSIONS' is not recognized as an internal or external command, operable program or batch file._

Thanks

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [July 26, 2023, 8:28am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/7 "2023-07-26T08:28:55Z")

</div>

> [@Vinoth](#):
>
> _Am getting the below error:_  
> _'PRODIGY\_ALLOWED\_SESSIONS' is not recognized as an internal or external command, operable program or batch file._

It seems that you're running windows, so it might be that your shell prefers to set environment variables differently. I found [this document](https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/set_1), but I can't give too much advise here since I am not a Windows user.

Alternatively, you might also configure a `.env` file and use [python-dotenv](https://pypi.org/project/python-dotenv/) to force the variables that way. If you go down this route, the command may look something like:

```python
# Assuming there's a `.env` file in the folder
dotenv run -- python -m prodigy ner.manual <arguments ...>

```

---

<div class="post-metadata">

**Author:** ![Vinoth](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/vinoth/32/4375_2.png) [@Vinoth](https://support.prodi.gy/u/Vinoth)\
**Post date:** [August 3, 2023, 5:20am UTC](https://support.prodi.gy/t/task-routers-problem-in-jsonl-output-annotators-problem/6702/8 "2023-08-03T05:20:40Z")

</div>

Hi,

Successfully, hosted in Linux terminal. But still a big question on Windows OS. I tried all possibilities for setting the environment variables w.r.t Windows OS, no improvement. Why we need such long process for setting env\_variables where none of the solution works better. Am still exploring myself on this.

If any solution, please post here.

Thanks