Streaming data from a dataframe directly

ks1996 · August 24, 2020, 6:28pm

@ines @honnibal
Hey Ines and Honnibal, I have a question regarding the loaders.

I know that we can stream data from a csv file. How do we stream data from a dataframe?
e.g :
"python -m prodigy ner.correct dataset spacy_model df --loader csv --label yes,no --unsegmented"
I know this will not work. Can you help me with how I can read from a dataframe with the column name set to "text" and stream it to prodigy.

Thank you in advance

ines · August 25, 2020, 8:22am

Hi! You should be able to just write a custom loader for that – either in a custom recipe, or by writing a script that can take any arguments and then pipe its output forward to Prodigy. So your loader loads your dataframe however you want, writes the JSON tasks to standard output, and Prodigy reads them from standard input. This will work with all built-in recipes. You can see examples here: https://prodi.gy/docs/api-loaders#loaders-custom

ks1996 · August 26, 2020, 11:15pm

@ines i have written custom recipes before to stream data. How to write custom recipes that would stream data like ner.correct recipe would?
for example, this recipe below would stream like ner.manual recipe would

How do I write it as a ner.correct recipe?

ines · August 27, 2020, 9:00am

You can find examples and recipe templates to get started in our prodigy-recipes repo. For example, this one uses a pretrained model to highlight entities and lets you edit them manually, just like ner.correct:

github.com

explosion/prodigy-recipes/blob/master/ner/ner_make_gold.py

import prodigy
from prodigy.components.loaders import JSONL
from prodigy.components.preprocess import add_tokens
from prodigy.util import split_string, set_hashes
import spacy
import copy
from typing import List, Optional


def make_tasks(nlp, stream, labels):
    """Add a 'spans' key to each example, with predicted entities."""
    # Process the stream using spaCy's nlp.pipe, which yields doc objects.
    # If as_tuples=True is set, you can pass in (text, context) tuples.
    texts = ((eg["text"], eg) for eg in stream)
    for doc, eg in nlp.pipe(texts, as_tuples=True):
        task = copy.deepcopy(eg)
        spans = []
        for ent in doc.ents:
            # Continue if predicted entity is not selected in labels
            if labels and ent.label_ not in labels:

This file has been truncated. show original

Prodigy also includes the source for its built-in recipes, so you can run prodigy stats to find the location of your Prodigy installation, and look at the files if you want to see how they work under the hood.

Topic		Replies	Views
Custom loader and Stream not compatible? usage	6	24	August 29, 2024
Adding column from CSV into meta with custom loader usage , solved	3	888	September 21, 2018
Custom recipe - stream loops through all examples. usage , custom	6	392	May 4, 2022
How to replace text in stream data? usage , ner , solved	2	310	November 19, 2021
Prodigy loads all the stream before annotation starts usage	6	817	March 26, 2019

Streaming data from a dataframe directly

Related topics