# Adding new data to be annotated without re-starting the server

**URL:** <https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876>\
**Category:** Uncategorized\
**Tags:** usage, database\
**Created:** [October 31, 2023, 3:24pm UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876 "2023-10-31T15:24:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [October 31, 2023, 3:24pm UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/1 "2023-10-31T15:24:48Z")

</div>

Please let me know if my understanding is completely flawed. At the moment, you can add a jsonl files to be annotated for (say) NER by running:

`prodigy ner.manual my_dataset en_core_web_sm my_file.jsonl --label my_labels`

which will then run the server.

Whereas you can add `annotated` data do the dataset `my_dataset` used the db-in command. My question is the following:

Suppose that I now want to add new data to my dataset. Is the _only way_ to do it to "bring the server down" and then reload it with

`prodigy ner.manual my_dataset en_core_web_sm my_new_file.jsonl --label my_labels`?

Is there no way to add text with no labels to be annotated through `db-in` or any other recipe?

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [October 31, 2023, 4:11pm UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/2 "2023-10-31T16:11:13Z")

</div>

> [@aCampello](#):
>
> Suppose that I now want to add new data to my dataset. Is the _only way_ to do it to "bring the server down" and then reload it with

There are some options available to Prodigy in this situation.

1. You can choose to write a custom recipe which uses a custom mechanism to fetch examples. Since custom recipes are just Python code, nothing is stopping you from writing a generator that fetches data from a database that has a table that receives updates. This way, you won't need to restart the server.
2. Alternatively, you may also point to a dataset that's been loaded in the Prodigy database instead of pointing to an `examples.jsonl` file. Theoretically that means you could also do something like this:

```shell
python -m prodigy db-in dataset-to-annotate examples.jsonl
python -m prodigy ner.manual dataset-with-annotations en_core_web_sm dataset:dataset-to-annotate --label mylabels

```

Notice that `dataset:`-prefix there? That points to a dataset in the database. That also means that while somebody is annotating you can do a `db-in` to append more examples to the `dataset-to-annotate`.

```shell
python -m prodigy db-in dataset-to-annotate moar-examples.jsonl

```

Technically, this would also prevent a restart but I imagine there is an edge case where an annotator may get to the end of their queue before new data is added.

### Main Advice

Both of these techniques should work, but I've never tried it extensively so there may be some edge cases that I can't come up with right now.

If I had to advise ... when you're dealing with a data source and you'd want to update the examples in a custom way then a custom recipe is probably the best path forward. It will require a small investment, but writing a bespoke Python solution usually pays off.

Just to check: is there a reason why a restart via CRON during the evening won't work for you?

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [October 31, 2023, 5:00pm UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/3 "2023-10-31T17:00:02Z")

</div>

Thanks for the very prompt response, @koaning, as usual! 🙂 The prodigy support forum is one of the greatest things of this tool so far !

I was trying to make the `dataset:` idea work, but what I found through some tests is that something like:

> [@koaning](#):
>
> ```python
> python -m prodigy db-in dataset-to-annotate moar-examples.jsonl
> 
> ```

does not work, because it adds the examples with "answer": "ACCEPTED" already, therefore it does not distribute the new texts to another, it rather records as already annotated. Can you advise?

> [@koaning](#):
>
> Just to check: is there a reason why a restart via CRON during the evening won't work for you?

I think it's a slightly different use case. I want to be able to add datapoints in an ad-hoc way to an annotation task living in a server. I am able to run `db-in ` from my local machine, but if this doesn't update the dataset, I'll have to "ssh" in, stop the server and re-run, or something similar, which doesn't sound very useful.

If none of the above works, yes, I'd invest in a custom recipe, but I wanted to exhaust all the "classical" pathways before going into that.

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [November 1, 2023, 9:19am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/4 "2023-11-01T09:19:56Z")

</div>

> [@aCampello](#):
>
> does not work, because it adds the examples with "answer": "ACCEPTED" already, therefore it does not distribute the new texts to another, it rather records as already annotated. Can you advise?

Ah, good point. I hadn't forseen that. This sounds to me like an issue that we should fix. It makes sense to me that a user might want to upload a dataset to the database for annotation and then there should be no answer flag. Will add a ticket internally to discuss this.

As an alternative for now, I think you should still be able to use the Python API instead. Something like this might work?

```python
from prodigy.components.db import connect
import srsly 

examples = srsly.read_jsonl("path/to/file.jsonl")
db.add_dataset('uploaded-dataset-name')
db.add_examples(examples, ('uploaded-dataset-name',))

```

This should be used in a script that generates candidates that are interesting to annotate. Would that work?

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 1, 2023, 9:33am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/5 "2023-11-01T09:33:41Z")

</div>

I will have a try a report back!

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 1, 2023, 10:39am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/6 "2023-11-01T10:39:05Z")

</div>

That was promising but STILL doesn't work as intended. Full Script her:

```python
from prodigy.components.db import connect
from prodigy.util import set_hashes
import srsly

examples = srsly.read_jsonl("test.jsonl")

db = connect()
db.add_dataset("test")

hashed_examples = [set_hashes(eg) for eg in list(examples)]

db.add_examples(hashed_examples, ["test"])

print(f"Added {len(hashed_examples)} examples to 'test' dataset")

```

And then when running `prodigy rel.manual test en_core_web_sm dataset:test --label TEST`  
I seem to get "No tasks available". The examples seem to have added as "already annotated"

![Screenshot 2023-11-01 at 10.42.01](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/7/718def867687dcc96fb66831ac23403f38f47bc5.png)

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 1, 2023, 10:49am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/7 "2023-11-01T10:49:48Z")

</div>

I think essentially what I am getting is that there seems to be no obvious way to make the `dataset:` idea work, because we can't import examples and then make prodigy get streams from the dataset. Is that intended or should that behaviour change?

---

<div class="post-metadata">

**Author:** ![koaning](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/koaning/32/230_2.png) [@koaning](https://support.prodi.gy/u/koaning)\
**Post date:** [November 1, 2023, 12:31pm UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/8 "2023-11-01T12:31:18Z")

</div>

I worry that something else might be up. I'll show what I've done to try and replicate this issue.

I start out with this `examples.jsonl` file.

```python
{"text": "hello my name is james"}
{"text": "hello my name is john"}
{"text": "hello my name is robert"}
{"text": "hello my name is michael"}
{"text": "hello my name is william"}
{"text": "hello my name is mary"}
{"text": "hello my name is david"}
{"text": "hello my name is richard"}
{"text": "hello my name is joseph"}

```

I use `db-in` to load it.

```python
python -m prodigy db-in issue-6876 examples.jsonl

```

This dataset is now in the Prodigy database and I can confirm via db-out that it also has hashes and an `accept` key.

```python
python -m prodigy db-out issue-6876

```

This yields:

```python
{"text":"hello my name is james","_input_hash":-1294982232,"_task_hash":465224705,"answer":"accept"}
{"text":"hello my name is john","_input_hash":-1282353592,"_task_hash":980664449,"answer":"accept"}
{"text":"hello my name is robert","_input_hash":803732484,"_task_hash":-658149367,"answer":"accept"}
{"text":"hello my name is michael","_input_hash":-817151764,"_task_hash":-831281946,"answer":"accept"}
{"text":"hello my name is william","_input_hash":-1774976813,"_task_hash":-349743654,"answer":"accept"}
{"text":"hello my name is mary","_input_hash":-1806564888,"_task_hash":910542438,"answer":"accept"}
{"text":"hello my name is david","_input_hash":-630231151,"_task_hash":-1900462764,"answer":"accept"}
{"text":"hello my name is richard","_input_hash":-277245364,"_task_hash":-1834276944,"answer":"accept"}
{"text":"hello my name is joseph","_input_hash":467632156,"_task_hash":-1782316920,"answer":"accept"}

```

But here's the thing, I seem to be able to annotate this data just fine.

```python
python -m prodigy textcat.manual xxx dataset:issue-6876 --label foo,bar,baz

```

This starts a server with this interface.

 ![CleanShot 2023-11-01 at 13.30.17](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/c/c95b5938d8bac0ec52060623f135440dec22fd5d.png)

So, on my side at least, the `accept` key doesn't block the annotations. Can you confirm this on your end as well on this demo dataset? I'm curious if perhaps there's something else happening on your machine. Do you have extra settings in a `config.json` file?

> [@aCampello](#):
>
> `db.add_dataset("test")`

Is it possible that a dataset called "test" already existed before you ran your script?

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 3, 2023, 9:38am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/9 "2023-11-03T09:38:27Z")

</div>

I'll have a look and report back!

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 3, 2023, 9:50am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/10 "2023-11-03T09:50:39Z")

</div>

Interesting. Yes, with your example, I can still annotate it again (which is somewhat disconcerting, probably not the intended behaviour for the `accepted` key?). I can also confirm that `ner.manual` and `rel.manual` also work! No idea what's going on.

---

<div class="post-metadata">

**Author:** ![aCampello](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/acampello/32/4180_2.png) [@aCampello](https://support.prodi.gy/u/aCampello)\
**Post date:** [November 3, 2023, 9:56am UTC](https://support.prodi.gy/t/adding-new-data-to-be-annotated-without-re-starting-the-server/6876/11 "2023-11-03T09:56:35Z")

</div>

Ok! I discovered what was happening in my example! When I was running the server I was doing:

```python
prodigy db-in examples.json dataset-to-be-annotated
prodigy rel.manual *dataset-to-be-annotated* en_core_web_sm dataset:*dataset-to-be-annotated* --label LABELS--span-label SPAN LABELS

```

Note how the dataset is repeated. If I replace the first one to `current-dataset` than it all works.

The quirk is that `db-in` STILL imports as accepted inside `dataset-to-be-annotated`, however it is available to annotate in `current-dataset`, which is quite interesting. I wonder if the `--answer` parameter in `db-in` should allow me to set no answer (`--answer null` or sonething) to avoid confusion.