Segmentation fault when using ner.batch-train

,
1

Hey,

I’m getting a segmentation fault error when using ner.batch-train, any idea on how I can trace this issue?

prodigy ner.batch-train domain_entities_annotations en_core_web_sm --output ./models/domain_entites --eval-split 0.2 --n-iter 25 --batch-size 32

Loaded model en_core_web_sm
Using 20% of accept/reject examples (357) for evaluation
Using 100% of remaining examples (10909) for training
Dropout: 0.2  Batch size: 32  Iterations: 25


BEFORE     0.014
Correct    4
Incorrect  276
Entities   4296
Unknown    4292


#          LOSS       RIGHT      WRONG      ENTS       SKIP       ACCURACY
01         58.564     107        174        36333      0          0.381
02         40.013     144        143        37429      0          0.502
03         37.137     161        126        37274      0          0.561
04         35.002     193        92         36999      0          0.677
05         33.377     210        75         36718      0          0.737
 89%|██████████████████████████████████████████████████████▏      | 9696/10909 [02:42<00:20, 59.80it/s]
/Users/*/*/*/env/bin/prodigy: line 1:  7146 Segmentation fault: 11  python -m prodigy "$@"

Thanks in advance! :smiley:

Update 2: Adding some context

  ✨  Prodigy stats

  Version            1.5.0
  Location           /Users/*/*/*/env/lib/python3.6/site-packages/prodigy
  Prodigy Home       /Users/ocselvig/.prodigy
  Platform           Darwin-17.6.0-x86_64-i386-64bit
  Python Version     3.6.5
  Database Name      SQLite
  Database Id        sqlite
  Total Datasets     15
  Total Sessions     77

Update 1: Setting up experiments on ubuntu sever to see what happens there.

1 Like
2

Fair question! Debugging segfaults is quite painful. Just to rule something out: could you check that you’re not running out of memory during training?

Failing that, you could set the batch size to 1, and hack the ner.batch_train recipe to use this subclass:

from prodigy.models import EntityRecognizer

class DebugEntityRecognizer(EntityRecognizer):
    def _update(self, texts, *args, **kwargs):
        print(list(texts))
        return EntityRecognizer._update(self, texts, *args, **kwargs)

This will help figure out which sentence is failing. Sorry we don’t have a logging statement in the right place here!