Prodigy Output Visualization, Dependencies Structure Training Help

[hmousa961 (Hmousa96)]

Hello,
I would like about 2 things. First, I trained a model and would like to test it on plain text. How can I do that and is there a recipe to visualize the output text?

Secondly, I have a text file with dependencies for ontonotes dataset for arabic language. Is there a specific structure to follow in order to create the json file for dependencies to be imported to prodigy?

Thank you so much for the help in advance.

[ines (Ines Montani)]

@hmousa961

I would like about 2 things. First, I trained a model and would like to test it on plain text. How can I do that and is there a recipe to visualize the output text?

Hi! If you've trained a spaCy model, I think the easiest way would be to just load it with spaCy directly and look at the annotations you're interested in. spaCy also comes with built-in visualizers that you can run from your terminal or in a Jupyter notebook: Visualizers · spaCy Usage Documentation

import spacy
from spacy import displacy

nlp = spacy.load("/path/to/model/you/just/trained")
doc = nlp("Some plain text goes here")
# Print the output (choose the attributes you're interested in)
print([(token.text, token.pos_, token.dep_) for token in doc])
print([(ent.text, ent.label_) for ent in doc.ents])
# Visualize
displacy.serve(doc, style="dep")  # or displacy.render if you're in a notebook
displacy.serve(doc, style="ent")

If you want a more interactive demo you can play with or share with others, check out the spacy-streamlit package:

GitHub - explosion/spacy-streamlit: 👑 spaCy building blocks and visualizers...

@hmousa961

Secondly, I have a text file with dependencies for ontonotes dataset for arabic language. Is there a specific structure to follow in order to create the json file for dependencies to be imported to prodigy?

Yes, see my comment on this thread:

Ontonotes 5 Prodigy Training JSON Format

Hi! You can see an example of the JSON format that Prodigy uses here: https://prodi.gy/docs/api-interfaces#ner_manual It uses character offsets and lets you provide a list of "tokens" as well that the spans can reference. If you need to convert token-based tags to offsets, you could use the helper functions spaCy provides: https://prodi.gy/docs/named-entity-recognition#tip-biluo-offsets I'd say that importing your data into Prodigy really only makes sense if you're planning on annotating it, e…

[hmousa961 (Hmousa96)]

Thank you so much. Can I ask you also about another thing? When I save each trained model from prodigy, the size of it is big like in gb. Is there a way to compress the size of the model?

Thank you in advance.

[ines (Ines Montani)]

@hmousa961

When I save each trained model from prodigy, the size of it is big like in gb. Is there a way to compress the size of the model?

This is a trade-off you have to make – if you're using word vectors to initialise your model, those will be required at runtime, so they need to be saved out with the model. Vectors can get quite large and they're mostly what makes up the difference in size (without them, your model directory would be more like 10-20mb).

Depending on the vectors you use, you could experiment with pruning them: https://v2.spacy.io/usage/vectors-similarity#custom-vectors-coverage This can give you similar coverage, which a much smaller vectors table overall. You could also try using smaller vectors, or train your own on a smaller but more representative sample of raw text.