Probs with ner.teach in bertbasedcase

if i user this call
!prodigy ner.teach jg02 de_trf_bertbasecased_lg corp2mb.txt --label PERSON,LOCATION

i get an error
_pickle.PicklingError: it's not the same object as typing.Union

using net.manual it works quiet well. The pickle points to a serializer problem. Any idea how to fix this?

It looks like something is trying to deepcopy an object that includes a Python type Union, which is a known Python issue. Not sure where that happens and where the type would come from, though :thinking:

In any case, there's not really a point in using a transformer starter model for ner.teach. The entity recognizer you train won't use the transformer weights anyways (you're just training a regular spaCy NER model) – and even if it did, updating/finetuning the model in the loop would be way too slow and not really effective, as the transformer models typically need larger batch sizes and need more than just a simple, single update with a small batch. So the model wouldn't really change as you update.