# Only part of the files loaded

**URL:** <https://support.prodi.gy/t/only-part-of-the-files-loaded/2613>\
**Category:** Uncategorized\
**Tags:** image, solved, server\
**Created:** [March 2, 2020, 10:10pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613 "2020-03-02T22:10:36Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sissi](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sissi](https://support.prodi.gy/u/sissi)\
**Post date:** [March 2, 2020, 10:10pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/1 "2020-03-02T22:10:36Z")

</div>

I have a dataset of 100 images (saved locally) and I'm using a jsonl file to store the location of images and existed bounding boxes.  
A single line in the file may looks like this:

```python
{"image": "animals.png", "width": 2200, "height": 1700, "spans": [{"_id": "1", "points": [[55.0, 95.0], [187.0, 95.0], [187.0, 73.1], [55.0, 73.1]], "label": "cat", "color": "#8011ee"},...]}

```

The command that I used looks like this:

```python
prodigy image.manual animal_pictures.jsonl --loader jsonl --label cat dog

```

Everything works normally at the beginning, but after I finished labeling the first 80 images, it showed 'no tasks available' (I'm sure that this dataset is newly created and the other 20 images were not labeled before.)  
I tried to end this session and restart it using the same command (same output dataset and same input jsonl file), it worked, but I still only got to label image 81-96 this time before it shows 'no tasks available'. Then no matter if I restart the session or reload the web page, I just couldn't load the last 4 images.  
Could anyone tell me the potential reason for this error? Thank you!

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [March 3, 2020, 10:02am UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/2 "2020-03-03T10:02:59Z")

</div>

Thanks for the report! How large are your image files? And are there differences in file size between them?

I wonder if this is related to the stream generator somehow "timing out" and returning an empty list, which would then cause Prodigy to think that no more tasks are available. There could also be some weird interaction with the new async serving (introduced in v1.9) that causes empty batches to be sent out if something (e.g. image loading) is blocking 🤔

---

<div class="post-metadata">

**Author:** ![sissi](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sissi](https://support.prodi.gy/u/sissi)\
**Post date:** [March 3, 2020, 10:26pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/3 "2020-03-03T22:26:21Z")

</div>

The images are around 200kB each. Below is a screeshot of examples.  
 ![image](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/7/7660873cac5195b5ce5c35f8c87f579e05962a8a.png)  
Is there something I can try to avoid this kind of problems?

---

<div class="post-metadata">

**Author:** ![sissi](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sissi](https://support.prodi.gy/u/sissi)\
**Post date:** [March 3, 2020, 10:28pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/4 "2020-03-03T22:28:53Z")

</div>

The images are not large, but there might be around 10-100 existed bounding boxes on each page, could this be related? Too many bounding boxes pre-labeled?

---

<div class="post-metadata">

**Author:** ![tiangolo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/tiangolo/32/1143_2.png) [@tiangolo](https://support.prodi.gy/u/tiangolo)\
**Post date:** [March 4, 2020, 4:29pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/5 "2020-03-04T16:29:09Z")

</div>

Hi @sissi !

I tried to replicate your environment and all the process, but I wasn't able to replicate the issue. 😕

I used this script to copy (generate) 100 images, and a file `animals.jsonl` with a line per image, each image having 100 pre-defined labels.

```python
import shutil
from pathlib import Path
import srsly

if __name__ == " __main__":
    dir_path = Path("./img")
    source_file = Path("./animal000.jpg")
    lines = []
    for i in range(1, 101):
        file_name = f"animal{i:03}.jpg"
        file_path: Path = dir_path / file_name
        shutil.copyfile(source_file, file_path)
        spans = []
        for span in range(100):
            spans.append(
                {
                    "_id": f"{i}_{span}",
                    "points": [
                        [55.0 + span, 95.0 + span],
                        [187.0 + span, 95.0 + span],
                        [187.0 + span, 73.1 + span],
                        [55.0 + span, 73.1 + span],
                    ],
                    "label": "cat",
                    "color": "#8011ee",
                }
            )
        lines.append(
            {"image": f"img/{file_name}", "width": 2200, "height": 1467, "spans": spans}
        )
    srsly.write_jsonl("animals.jsonl", lines=lines)

```

Then I ran it with:

```python
$ prodigy image.manual animals8 animals.jsonl --loader jsonl --label cat,dog

```

Using the latest version of Prodigy.

I tried passing several times through the whole dataset (each time in a new Prodigy "dataset", like `animals8`). I tried with different combinations, adding annotations to each image, and also accepting or rejecting fast, to make sure that if there was something related to concurrency it would show up.

But I wasn't able to replicate your issue 😕 All the times I got all the images and finished at exactly 100.

Maybe you could try generating a fake/testing dataset like that with that script and see if it works correctly or not. If it works as expected with the fake/generated dataset, then it might be something related to your dataset or the configs used with it. If it has the same error, then your environment and mine would probably have something different... Try with the latest Prodigy version and a recent Python (e.g. 3.7). Also, check the logs, there could be an error somewhere that could give some hints about what's happening underneath.

---

<div class="post-metadata">

**Author:** ![sissi](https://avatars.discourse-cdn.com/v4/letter/s/82dd89/32.png) [@sissi](https://support.prodi.gy/u/sissi)\
**Post date:** [March 5, 2020, 2:34pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/6 "2020-03-05T14:34:36Z")

</div>

Thank you for testing out! I'm using Python 3.7 and Prodigy 1.9.6. I tried with testing dataset before, and it seems that this issue is pretty random. Here's what I tried:

1. Same input data, same commands (only difference was name of dataset), I was able to finish the 100 pages in the testing dataset.
2. A new .jsonl with only those lines that didn't load in the first annotation, start a new annotation in a new dataset, I also finished the annotation for these lines.

I did change some of the configurations, all that I changed are as below:

```python
 "custom_theme": {
   "cardMaxWidth": "1000px"
 },
"db_settings": {
    "sqlite": {
      "name": "prodigy.db",
      "path": "my_path"
    }
  },
"global_css": ".prodigy-title {font-size: 14px} .prodigy-content svg text {font-size:20px !important; opacity:0.6} .prodigy-content path {stroke-width:2 !important}",
"labels": ["cat","dog","pig","bird"...]

```

I wasn't able to replicate the issue too 😅 annotations went well after that time.

---

<div class="post-metadata">

**Author:** ![tiangolo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/tiangolo/32/1143_2.png) [@tiangolo](https://support.prodi.gy/u/tiangolo)\
**Post date:** [March 5, 2020, 3:01pm UTC](https://support.prodi.gy/t/only-part-of-the-files-loaded/2613/7 "2020-03-05T15:01:28Z")

</div>

Argh, I don't like it when bugs get shy and disappear without giving a chance to understand them... 😅 🐛

But I'm glad it seems you solved the issue for now! 🎉