Using a spaCy-stanza model for tokenization with ner.manual

ines · June 30, 2020, 9:11am

Hi! The spacy-stanza wrapper currently doesn't serialize the stanza model data, so loading it back in still requires the loaded Stanza model. (I'm kinda confused by the error, though – this is normally what spaCy raises if the directory isn't valid or doesn't have a meta, so there might be something else going on here on top of it?)

Anyway, the good news is, once you have your spacy-stanza model loaded, you can use the nlp object like any other nlp object to tokenize text. So you should be able to adapt this example recipe here and just replace the nlp object with your spacy-stanza nlp object:

github.com

explosion/prodigy-recipes/blob/master/ner/ner_manual.py

import prodigy
from prodigy.components.loaders import JSONL
from prodigy.components.preprocess import add_tokens
from prodigy.util import split_string
import spacy
from typing import List, Optional


# Recipe decorator with argument annotations: (description, argument type,
# shortcut, type / converter function called on value before it's passed to
# the function). Descriptions are also shown when typing --help.
@prodigy.recipe(
    "ner.manual",
    dataset=("The dataset to use", "positional", None, str),
    spacy_model=("The base model", "positional", None, str),
    source=("The source data as a JSONL file", "positional", None, str),
    label=("One or more comma-separated labels", "option", "l", split_string),
    exclude=("Names of datasets to exclude", "option", "e", split_string),
)
def ner_manual(

This file has been truncated. show original

Topic		Replies	Views
How to use customized spaCy model in Prodigy? ner , spacy	6	470	July 3, 2023
Prodigy is losing my tokeniser usage , spacy	2	419	February 18, 2022
Annotate using ner.manual for a new language usage , ner , spacy , solved	2	670	October 27, 2019
Roadmap of having a unified model for tokenizing, NER and dependency parsing using Prodigy ner , spacy , custom , training	1	414	July 7, 2023
Migration from spaCy 2.3 to 3.x + Annotating data in prodigy usage , spacy	1	455	August 29, 2021

Using a spaCy-stanza model for tokenization with ner.manual

Related topics