# Model explanation
**URL:** https://support.prodi.gy/t/model-explanation/2656
**Category:** Uncategorized
**Tags:** enhancement, usage, custom
**Created:** [March 12, 2020, 3:18pm UTC](https://support.prodi.gy/t/model-explanation/2656 "2020-03-12T15:18:20Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@curious](https://support.prodi.gy/u/curious)
#### Post date: [March 12, 2020, 3:18pm UTC](https://support.prodi.gy/t/model-explanation/2656/1 "2020-03-12T15:18:20Z")
Model explanation is a hot topic now and business users want it. If Prodigy can highlight which words/phrases carry more weight, human should be able to annotate even faster and more accurately.
I've seen some discussion on having model returns the weight for each token. I wonder whether Prodigy is thinking of introducing a model explanation out of the box. Or is there way I can implement this function by myself? Thanks.
---
### Author:  [@simon.gurcke](https://support.prodi.gy/u/simon.gurcke)
#### Post date: [March 13, 2020, 5:54am UTC](https://support.prodi.gy/t/model-explanation/2656/2 "2020-03-13T05:54:26Z")
I think this would be a great addition to Prodigy. I have implemented this outside of Prodigy and it really helps to understand what the model has actually learned.
My implementation is for a RoBERTa text classification model and build on top of [PyTorch's Captum](https://github.com/pytorch/captum) using `GradientShap`.
---
### Author:  [@ines](https://support.prodi.gy/u/ines)
#### Post date: [March 13, 2020, 11:13am UTC](https://support.prodi.gy/t/model-explanation/2656/3 "2020-03-13T11:13:13Z")
> [@curious](#):
>
> Model explanation is a hot topic now and business users want it. If Prodigy can highlight which words/phrases carry more weight, human should be able to annotate even faster and more accurately.
Yes, that's a cool idea! Prodigy should already have all the building blocks for that – you just need to implement the process that weights for the tokens/subtokens or whatever else you want to interpret.
This thread is slightly older, but it has some custom recipes and ideas for visualizing attention during text classification annotation:
> [@Highlighting the matching words for text classfication](https://support.prodi.gy/t/highlighting-the-matching-words-for-text-classfication/254/8):
>
> I created variants on textcat.teach and texcat.eval that render examples using a custom template and attention weight data from @honnibal’s example. The [code is on Github](https://github.com/justindujardin/prodigy-scratch/blob/master/recipes/textcat_attention_weights.py), and it renders text where items with more attention have larger fonts, and those that have a bunch of attention get a special color. [attention\_weights] I transform the attention weights into metadata for use in the HTML template: def attach\_attention\_data(input\_stream, nlp, attn\_weights): """Attach attention weights t…
> [@curious](#):
>
> I've seen some discussion on having model returns the weight for each token. I wonder whether Prodigy is thinking of introducing a model explanation out of the box.
We'd love to have this more easily accessible for spaCy models! But otherwise, it really depens on your model, the framework you're using (both for ML and interpretability) and what you're trying to do. There's no out-of-the-box answer for that. But Prodigy should provide the building blocks you need to incorporate model interpretability into your annotation workflow.
> [@simon.gurcke](#):
>
> My implementation is for a RoBERTa text classification model and build on top of [PyTorch's Captum](https://github.com/pytorch/captum) using `GradientShap` .
This looks great! 💯 And this seems to already return formatted HTML, right? So I guess you could stream that in using the `html` interface, or add it as a separate block?
(Also, a small detail that I want to add to the regular `ner`/ `spans` interface: If individual spans can take a `"color"` value, you could easily implement the same visualization just with character offsets and different colour shades depending on the score, without having to assign distinct labels and label colours.)
---
### Author:  [@curious](https://support.prodi.gy/u/curious)
#### Post date: [March 13, 2020, 1:30pm UTC](https://support.prodi.gy/t/model-explanation/2656/4 "2020-03-13T13:30:20Z")
Great. I'll try both solutions and update the result here. Thanks!
---
### Author:  [@sdspieg](https://support.prodi.gy/u/sdspieg)
#### Post date: [May 17, 2023, 11:11am UTC](https://support.prodi.gy/t/model-explanation/2656/5 "2023-05-17T11:11:47Z")
This thread is already quite old, but was any progress made on this front?
---
### Author:  [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)
#### Post date: [May 17, 2023, 6:56pm UTC](https://support.prodi.gy/t/model-explanation/2656/6 "2023-05-17T18:56:54Z")
hi @sdspieg!
Thanks for the suggestion. I dabbled about a year ago with a SHAP recipe:
>
Haven't tried it lately but I suspect it'd still work (not sure if `shap` package has changed any).
Just curious - do you have any techniques or specifics you're looking for? Especially if you know of any existing packages we could port in like `shap`?
We can make an internal note about the interest in an explanation recipe. We're working hard on several significant updates to Prodigy (coming out soon!) but will look back into custom recipes when we can.