How can I compare my ner model with a huggingface transformer model?

,
1

I created a model to recognize drugs using this flow of recipes:

  • terms.to-patterns
  • ner.manual
  • terms.teach

And then, I execute:
prodigy train ./tmp_model --ner drug_data -l pt --eval-split 0.2

How can I compare using seqeval (GitHub - chakki-works/seqeval: A Python framework for sequence labeling evaluation(named-entity recognition, pos tagging, etc...))? There is these model, who does a similar task: pucpr/clinicalnerpt-pharmacologic · Hugging Face.

How can I import my annotated databases to huggingface data?

2

Hi @giovaninb , welcome to Prodigy! :slight_smile:

How can I compare using seqeval (GitHub - chakki-works/seqeval: A Python framework for sequence labeling evaluation(named-entity recognition, pos tagging, etc...))?

How about running spacy evaluate on BioBERT's test set, obtain the Precision/Recall/F-score, and comparing it with the reported results in the paper? It might also be done the other way around: run the trained model to a test set, get the predictions, and use those predictions with the seqeval repo. :thinking:

How can I import my annotated databases to huggingface data?

This thread should be useful: Exporting NER annotations for HF datasets - #4 by ChrisM
It talks about a user project that uses Prodigy annotations in the datasets library and training a transformer with transformers.

1 Like