Terms Trains Crashing

[madhujahagirdar]

We had corpus (word2vec 3GB ) from PubMed-and-PMC focused on Biomedical information. I wanted to use this corpus, which was generic biomedical and build model focused on radiology reports. The radiology reports amounts to 4GB of data with 3 million unique entries. I was using the following command to build a word2vec model. I have 48 core machine with 200GB ram. This runs for 4 hours and consumes most of the cores and run out of memory after 4 hrs. Not sure if there is a memory leak or does it need more resources. Is there way we can debug this or dump any logs to identify issue.

nohup python -m prodigy terms.train-vectors /home/ubuntu/cnn-annotation/model/radiologymodel /home/ubuntu/cnn-annotation/InstallPackages/source/allReporttext.txt --spacy-model /home/ubuntu/cnn-annotation/InstallPackages/model/pmcmodel/PubMed-and-PMC-w2v-spacy.bin --size 300 --merge-nps --merge-ents &

[honnibal (Matthew Honnibal)]

It’s definitely a memory leak — your resources should be fine for that workload.

You can enable logging with the export PRODIGY_LOGGING=basic or export PRODIGY_LOGGING=verbose

The source for the recipes is available in `prodgy/recipes/’. You can either unzip the .whl file, or you can get it from your installation — so you can also add print statements.

The question is whether it runs out of memory during the spaCy pre-processing, or whether it runs out of memory when it passes the text to Gensim to train the vectors. I would expect Gensim to easily be able to handle 4gb of text. At the same time, what we’re accumulating from spaCy is just the text objects, so the working set there shouldn’t be so large either.

Try running with a sample of your text and watching the memory usage at each stage. The problem’s likely to be non-linear in the amount of text, so small samples might not reflect the problem — but maybe they do.

By the way, does your spaCy model have an entity recogniser and parser loaded? If not, the merge-nps and merge-ents options won’t work.

[madhujahagirdar]

@honnibal

By the way, does your spaCy model have an entity recogniser and parser loaded? If not, the merge-nps and merge-ents options won’t work

Yest night, I ran without merge-nps and merge-ents still it runs out of memory.

[madhujahagirdar]

@honnibal

The source for the recipes is available in `prodgy/recipes/’. You can either unzip the .whl file, or you can get it from your installation — so you can also add print statements.

I will try to do this and report back.

[honnibal (Matthew Honnibal)]

Great, thanks. For ease of reference, you can always get the path to the file being run with python -c "import prodigy.recipes.terms; print(prodigy.recipes.terms.__file__)"

[madhujahagirdar]

In terms.py, The log output does not log print(“Extracted {} sentences”.format(len(sentences))). My thoughts are that for loop where the sentences are appended is where the memory leak is occurring and causing the crash.

print(“Generating strea now …”,loader)
stream = get_stream(source, loader=loader, input_key=‘text’)
print(“Finished Generating strea now …”)
sentences = []
for doc in nlp.pipe((eg[‘text’] for eg in stream)):
for sent in doc.sents:
sentences.append([w.text for w in doc])
print(“Extracted {} sentences”.format(len(sentences)))

[honnibal (Matthew Honnibal)]

What if you write out the sentences to a file, instead of accumulating the list? So something like

with open('/tmp/output.txt', 'w') as file_:
    for doc in nlp.pipe((eg['text'] for eg in steam)):
        for sent in doc.sents:
            file_.write(sent.text.replace('\n', ' ') + '\n')
with open('/tmp/output.txt') as file_:
    sentences = file_.read().split('\n')

If we don’t get to read the sentences back in, there’s definitely a memory leak in spaCy.

[madhujahagirdar]

Let me try this and report back in next 4 hrs

[madhujahagirdar]

@honnibal

If we don’t get to read the sentences back in, there’s definitely a memory leak in spaCy.

If there a memory leak, any workaround ? Can I build word2vec using Gensim and then convert to spacy model would that work ?

[honnibal (Matthew Honnibal)]

You can definitely build the word2vec in Gensim, FastText, GloVe etc, and then load into spaCy afterwards.

The thing is, that’s actually what this recipe is doing: spaCy doesn’t learn the vectors, we’re just pre-processing the text here. And we do want to use spaCy’s text pre-processing, because we want to make sure we’re loading back terms that have been segmented the same way.

For intance, if spaCy segments the string don't as ["do", "n't"], we want to learn vectors for those two tokens. If the word vector package has preprocessed the text to produce ["don't"], that will be the key in the vector table — so we won’t get the right vectors back out when we try to associate text to vectors.

This is how the merging works, by the way — we’re just pre-processing the text, to produce different tokens. Then those tokens are passed forward to Gensim’s word2vec. Gensim tries to learn a word vector for each word type. If we tell it “anterior cruciate ligament” is a token, it’ll learn a vector for that. Then we can load the table back in, and so long as we can apply the same pre-processing on the text to get “anterior cruciate ligament” to be a token, we can access the vector for it.

[madhujahagirdar]

Got it ! So, if it is memory leak issue then I would wait for the fix in Spacy.

[madhujahagirdar]

@honnibal

What if you write out the sentences to a file, instead of accumulating the list? So something like

with open('/tmp/output.txt', 'w') as file_:
for doc in nlp.pipe((eg['text'] for eg in steam)):
for sent in doc.sents:
file_.write(sent.text.replace('\n', ' ') + '\n')
with open('/tmp/output.txt') as file_:
sentences = file_.read().split('\n')

If we don’t get to read the sentences back in, there’s definitely a memory leak in spaCy.

I tried this out and found that /tmp/output.txt is never read. Appears to be memory leak in Spacy. Let me know how to go about ! Additionally, it ran out memory after 319k documents out of 3 million.

[honnibal (Matthew Honnibal)]

Could you paste me /home/ubuntu/cnn-annotation/InstallPackages/model/pmcmodel/PubMed-and-PMC-w2v-spacy.bin/meta.json?

Let’s also simplify the script we’re working with to make this easier to reason about. Does the memory leak still happen if you run this?

def main():
    source_loc = '/home/ubuntu/cnn-annotation/InstallPackages/source/allReporttext.txt'
    spacy_loc = '/home/ubuntu/cnn-annotation/InstallPackages/model/pmcmodel/PubMed-and-PMC-w2v-spacy.bin'
    output_loc = '/tmp/sentences.txt'

    nlp = spacy.load(spacy_loc)

    with open(output_loc, 'w') as output_file:
        i = 0
        with open(source_loc) as input_file:
            texts = (line.strip() for line in input_file)
            for doc in nlp.pipe(texts):
                for sent in doc.sents:
                    output_file.write(sent.text + '\n')
                i += 1
                if i and i % 1000 == 0:
                    print('%d lines processed' % i)

if __name__ == '__main__':
    main()

If the memory leak still occurs, does it still happen if you add nlp.disable_pipes(*nlp.pipe_names) after loading the NLP model?

The next step will be to inspect the object graph to figure out just what is sticking around. That’s a bit annoying to do, so I thought we’d start with these basic checks first.

[madhujahagirdar]

@honnibal

Could you paste me /home/ubuntu/cnn-annotation/InstallPackages/model/pmcmodel/PubMed-and-PMC-w2v-spacy.bin/meta.json?

{
"lang":"en",
"pipeline":[
"tagger",
"parser",
"ner"
],
"accuracy":{
"token_acc":99.8698372794,
"ents_p":84.9664503965,
"ents_r":85.6312524451,
"uas":91.7237657538,
"tags_acc":97.0403350292,
"ents_f":85.2975560875,
"las":89.800872413
},
"name":"core_web_sm",
"license":"CC BY-SA 3.0",
"author":"Explosion AI",
"url":"https://explosion.ai",
"vectors":{
"width":200,
"vectors":4644099,
"keys":4087446
},
"sources":[
"OntoNotes 5",
"Common Crawl"
],
"version":"2.0.0",
"spacy_version":">=2.0.0a18",
"parent_package":"spacy",
"speed":{
"gpu":null,
"nwords":291344,
"cpu":5122.3040471407
},
"email":"contact@explosion.ai",
"description":"English multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. Assigns word vectors, context-specific token vectors, POS tags, dependency parse and named entities."
}

[honnibal (Matthew Honnibal)]

Thanks.

Surely something to do with the custom vectors. If you run without your vectors by just using spacy.load('en_core_web_sm'), can you confirm there’s no leak?

I’ll open an issue for this on the spaCy tracker, probably later today. Hopefully I’ll have a runtime workaround, but if not I can upload a development build for you to pip install.

[madhujahagirdar]

@honnibal

Let’s also simplify the script we’re working with to make this easier to reason about. Does the memory leak still happen if you run this?

This does not cause any memory leak. I ran it for an hour and it did not lead to memory spike.

[honnibal (Matthew Honnibal)]

Thanks! Closing in on this…

Now add:

from prodigy.components import preprocess
nlp.add_pipe(preprocess.merge_entities, name='merge_entities')
nlp.add_pipe(preprocess.merge_noun_chunks, name='merge_noun_chunks')

This should be added after the call to spacy.load(). If this leaks, the problem is in spaCy’s doc.merge(). If it doesn’t leak, the problem is in Prodigy, somewhere innocuous – possibly the text loader or something.

[madhujahagirdar]

Even without merge_entities and merge_noun_chunks we had memory leak. Should i still try this ?

[honnibal (Matthew Honnibal)]

As a sanity check I think it would be useful.

[madhujahagirdar]

I wrote simple code with blank model and it gives me memory leak.

from future import unicode_literals
import plac
import numpy
from gensim.models.word2vec import Word2Vec
from gensim.models import KeyedVectors
import spacy
from prodigy.components import preprocess
from prodigy.components.loaders import get_stream

nlp = spacy.blank(“en”)
nlp.add_pipe(nlp.create_pipe(‘sentencizer’))

#doc = nlp(“testing is good habit. I am doing great”)

stream = get_stream("/Users/philips/Development/BigData/RS/word2vecreport/report/allReporttext.txt", loader=None, input_key=‘text’)
sentences = []
count = 0

for doc in nlp.pipe((eg[‘text’] for eg in stream)):
count = count + 1
for sent in doc.sents:
sentences.append([w.text for w in doc])
print(count)

w2v = Word2Vec(sentences, size=300, window=5, min_count=10,
sample=1e-5, iter=1, workers=6,
negative=5)

nlp.vocab.reset_vectors(width=300)
output_model = “/Users/philips/Development/BigData/RS/word2vecreport/report/pmcmodel”
for word in w2v.wv.vocab:
nlp.vocab.set_vector(word, w2v.wv.word_vec(word))

nlp.to_disk("/Users/philips/Development/BigData/RS/word2vecreport/report/pmcmodel")
print(‘Trained Word2Vec model’, output_model.resolve())