I’d like to understand more about the error message:
“Cannot close object, library is destroyed. This may cause a memory leak”
I am unsure of the behaviour that triggers it and what fixes it, so that makes it difficult to manage. I’ve tried getting out of venv and reactivating. I think that has worked sometimes, but not always.
It seems not to stop the creation of data when I have checked the jsonl output.
However, sometimes downstream code doesn’t work afterwards. And then it is a case of was it the code? or has the memory leak created a problem with the data?
Leading to a need to reannote – which is obviously timeconsuming bit (and doesn’t always solve the problem)
I could do with knowing whether the error message is deriving from prodigy (and what might trigger it) or something else.
It looks like the code is trying to use the object that has already been removed memory. It's hard to say what could that exactly without debugging.
Could you share more details about the code you're running (which built-in recipe? or perhaps is it a custom recipe that you can share?).
Is there any traceback that points to the actual code line that errors out? Have you done any memory profiling of the code already?
Also, could you share the size and perhaps an example of the data that the recipe is consuming?
I've had a bit more experience with Prodigy now and my experience is that a memory leak usually denotes something wrong with custom code. Fixing the custom code fixes the memory leak
