# Asynchronous stream

**URL:** <https://support.prodi.gy/t/asynchronous-stream/773>\
**Category:** Uncategorized\
**Tags:** enhancement, api\
**Created:** [August 23, 2018, 10:46pm UTC](https://support.prodi.gy/t/asynchronous-stream/773 "2018-08-23T22:46:41Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![alejandro.mesa](https://avatars.discourse-cdn.com/v4/letter/a/8dc957/32.png) [@alejandro.mesa](https://support.prodi.gy/u/alejandro.mesa)\
**Post date:** [August 23, 2018, 10:46pm UTC](https://support.prodi.gy/t/asynchronous-stream/773/1 "2018-08-23T22:46:41Z")

</div>

Hello,

The stream for our recipe is composed of text elements that are loaded asynchronously every hour and for several days. For this reason, when we first run prodigy with our custom recipe, we’re able to label the tasks that are first loaded into the stream. Once the stream is fully consumed, the page displays No tasks available. An hour later, once new tasks have been loaded, we have to manually restart prodigy to create a new stream with the new items.

In order to get around this issue, we could potentially patch get get\_questions method in the Controller as follows:

```
# python queue that contains new tasks loaded asynchronously
q = queue.Queue()

def patch_get_questions():
    items = []
    if not q.empty():      
        for _ in range(controller.batch_size):
            try:
                item = q.get(block=False)
                items.append(set_hashes(item))
            except queue.Empty as e:
                break

    return items

controller.get_questions = patch_get_questions

```

This seems to work, but we’re not sure what other processing may be done in the original get\_questions method. We know this is highly dangerous, so we were wondering if you had any other ideas on how to accomplish this.

Thanks

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [August 24, 2018, 10:53am UTC](https://support.prodi.gy/t/asynchronous-stream/773/2 "2018-08-24T10:53:31Z")

</div>

Thanks for sharing your code and your use case!

I do think the solution you’ve come up with is pretty reasonable, especially considering the specific use case and constraints. The default `controller.get_questions` method is very basic and only really yields batches from the stream anyways.

Going forward, it might actually make sense for Prodigy to adopt a `Queue` approach similar to yours, at least internally. This would make the behaviour of the stream more predictable, and would also make it easier to handle multiple consumers.

---

<div class="post-metadata">

**Author:** ![akshitasood63](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/akshitasood63/32/455_2.png) [@akshitasood63](https://support.prodi.gy/u/akshitasood63)\
**Post date:** [October 15, 2018, 9:13am UTC](https://support.prodi.gy/t/asynchronous-stream/773/3 "2018-10-15T09:13:49Z")

</div>

Has this feature been updated in the newer versions of prodigy , for handling multiple consumers ?

---

<div class="post-metadata">

**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\
**Post date:** [October 15, 2018, 11:50am UTC](https://support.prodi.gy/t/asynchronous-stream/773/4 "2018-10-15T11:50:03Z")

</div>

The upcoming version of Prodigy will include more internals that help with managing streams, but you still have to decide how you want to implement the consumers and also how the data should flow though.

Our own solution will be implemented in Prodigy Scale – see here for details:

> [@sparkles Prodigy Annotation Manager Update // Prodigy Scale // Prodigy Teams](https://support.prodi.gy/t/prodigy-annotation-manager-update-prodigy-scale/805/1):
>
> Many of you have been asking about how to scale your Prodigy projects – including how to manage more annotators, how to keep multiple feeds running, and how to make sure your annotators stay in agreement. These are challenging problems that are outside of the scope of the [Prodigy](https://prodi.gy) software itself, so we've always planned an extension product that helps you use Prodigy in larger products. We've been quietly working away on this for some time, so we're excited to finally announce our progress! The …