# Custom loader and Stream not compatible?

**URL:** <https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340>\
**Category:** Uncategorized\
**Tags:** usage\
**Created:** [August 22, 2024, 9:51am UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340 "2024-08-22T09:51:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![MarieCo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marieco/32/4465_2.png) [@MarieCo](https://support.prodi.gy/u/MarieCo)\
**Post date:** [August 22, 2024, 9:51am UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/1 "2024-08-22T09:51:50Z")

</div>

Hi there,

I am trying to set up a custom interface, together with a custom csv loader. I am following quite closely the [example recipe for the custom UI](https://prodi.gy/docs/custom-recipes#example-blocks), as it is quite close to what I am trying to achieve: I want my reviewers to tell me if a piece of text is relevant for them, and have some possibly interesting entities highlighted to help them review (for which I think I'll need to use ner.correct instead of ner.manual, but not there yet).

I am pasting the code I am using below

```python
import prodigy
import csv
from prodigy.components.preprocess import add_tokens
import spacy

@prodigy.recipe("retrieval-validation")
def retrieval(dataset, source):
    # We can use the blocks to override certain config and content, and set
    # "text": None for the choice interface so it doesn't also render the text
    blocks = [
        {"view_id": "ner_manual"},
        {"view_id": "choice", "text":None},
        {"view_id": "text_input", "field_rows": 3, "field_label": "If ambiguous, why?"}
    ]
    options = [
        {"id": 2, "text": "😺 Relevant"},
        {"id": 1, "text": "🙀 Ambiguous"},
        {"id": 0, "text": "😾 Not relevant"}
    ]

    
    def custom_csv_loader(source): 
        with open(source) as csvfile: 
            reader = csv.DictReader(csvfile) 
            for row in reader: 
                r_score = row.get('r_score') 
                text = row.get('query') + '\n\n' + row.get('text')
                s_score = row.get('s_score') 
                pdf = row.get('pdf') 
                yield {'text': text, "options": options,'r_score':r_score,'s_score':s_score, 'meta': {'pdf':pdf}}

    stream = custom_csv_loader(source) #load stream with custom loader
    nlp = spacy.load("en_ner_jnlpba_md")#load model                          
    stream.apply(add_tokens, nlp=nlp, stream=stream) # tokenize the stream for ner_manual

    return {
        "dataset": dataset, # the dataset to save annotations to
        "view_id": "blocks", # set the view_id to "blocks"
        "stream": stream, # the stream of incoming examples
        "config": {
            "labels": ["DNA", "PROTEIN"],
            "blocks": blocks # add the blocks to the config
        }
    }

```

When running this with my csv file I get:

```python
AttributeError: 'generator' object has no attribute 'apply'

```

When I remove the ner.manual view\_id and just load the stream with my custom loader, I have no issue, but as soon as I need to modify the tasks I have a problem.  
The prodigy version I am using is 1.15.

Is there something I am missing to make this work?

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [August 26, 2024, 8:35am UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/2 "2024-08-26T08:35:50Z")

</div>

Welcome to the forum @MarieCo 👋

Apologies for a slightly delayed response but for some reason I've missed your post until now!

The reason why the `apply` method from the example doesn't work is that it is meant to be used with the [`Stream`](https://prodi.gy/docs/api-components#stream) data structure, while your custom loader is returning a generator object.  
I admit it might be a bit confusing because Prodigy built-in loaders used to return the `generator` until version 1.12. Then we made a complete refactor and introduced the `Stream` data structure for better tractability of the stream of tasks (some more [context](https://support.prodi.gy/t/prodigy-1-12-alpha-release-llm-assisted-workflows-prompt-engineering-fully-custom-task-routing-for-multi-annotator-scenarios/6552)). We've updated most of the examples in the docs to use the updated `Stream` API, without stating that explicitly in the custom loaders section (note taken!)

You can still use "the old" generator way (and not use the `apply` method) but I'll show you how to edit your code to make it compatible with the new `Stream` data structure.  
The easiest and the smallest edit would be to convert your generator into `Stream` using `Stream`'s `.from_iterable` method:

```python
from prodigy.components.stream import Stream
stream_as_generator = custom_csv_loader(source) #load stream with custom loader
stream = Stream.from_iterable(stream_as_generator) # convert it into Stream

```

That should make the rest of your code run as expected - let me know if it doesn't 🙂

---

<div class="post-metadata">

**Author:** ![MarieCo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marieco/32/4465_2.png) [@MarieCo](https://support.prodi.gy/u/MarieCo)\
**Post date:** [August 27, 2024, 8:35am UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/3 "2024-08-27T08:35:51Z")

</div>

Hi @magdaaniol , thanks a lot for your answer (and for the welcome)!

Yes, I had seen some elements from your answer in the documentation but wasn't quite sure how to fix it, so thanks for that!  
Maybe a stupid question but I can't figure it out: do I import Stream from somewhere? I get a

```python
NameError: name 'Stream' is not defined

```

with the code you gave me.

Thanks!

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [August 27, 2024, 2:15pm UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/4 "2024-08-27T14:15:30Z")

</div>

Oh my bad! Should have added it. Yes, it needs to be imported form `prodigy.components.stream` module just like the `get_stream` function i.e.

```python
from prodigy.components.stream import Stream

```

---

<div class="post-metadata">

**Author:** ![MarieCo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marieco/32/4465_2.png) [@MarieCo](https://support.prodi.gy/u/MarieCo)\
**Post date:** [August 28, 2024, 2:34pm UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/5 "2024-08-28T14:34:38Z")

</div>

Thanks! It is what I would have thought, but...

```python
AttributeError: type object 'Stream' has no attribute 'from_iterator'

```

Maybe I am missing something, but not sure where to look.

---

<div class="post-metadata">

**Author:** ![magdaaniol](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/magdaaniol/32/2787_2.png) [@magdaaniol](https://support.prodi.gy/u/magdaaniol)\
**Post date:** [August 28, 2024, 3:12pm UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/6 "2024-08-28T15:12:24Z")

</div>

Hi @MarieCo ,

Apologies, I made a mistake when typing the snippet. The `Stream` method name is `from_iterable` not `from_iterator`. I've updated my original response. above.

---

<div class="post-metadata">

**Author:** ![MarieCo](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/marieco/32/4465_2.png) [@MarieCo](https://support.prodi.gy/u/MarieCo)\
**Post date:** [August 29, 2024, 7:20am UTC](https://support.prodi.gy/t/custom-loader-and-stream-not-compatible/7340/7 "2024-08-29T07:20:05Z")

</div>

Woop woop, that works!  
Thanks again for all your help @magdaaniol , have a great day!