Wrap breaks for long documents

[ryanwesslen]

hi @Martin,

Thanks for your post.

Unfortunately, it seems like you're getting this error due to the long size of the document, especially for relations:

relation is responding very slowly

Hi and sorry about this – it's currently expected that the interface becomes less performant for very long documents with many tokens, and we're working on a rewrite that doesn't have this problem. I think what makes it additionally tricky in your case is that you just end up with more tokens overall, due to the way the characters are segmented. As a workaround, one thing you can do is use patterns to disable all tokens that you know won't ever be part of a relation – of course, only if that's …

One option could be to disable tokens (use --disable-patterns) you're not interested in, for example see the docs.

However, I suspect your document is still way too long.

I would love at least a work around to have line breaks in our interface, as we have line breaks in our documents, and they are significant for our purposes. I have seen newlines in relations annotation and it doesn't specifically help given that wrap breaks on us ! Is there any way to break lines on newline tokens ?

Why don't you write a pre-processing script for this?

For example, this Stack Overflow shows how to break up by new line characters this before using spaCy. You could combine this with this example:

how to annotate a longer text in rel.manual?

Is there a reason why you can't split the data into smaller segments? I can imagine that long pieces of text require a lot of scrolling and can make it much harder to annotate. I sometimes like to use spaCy to preprocess my data, via something like: import spacy nlp = spacy.load("en_core_web_md") # I'm assuming a list/generator of texts loaded in Python orig_examples = ... # This function can then turn them into seperate json blobs with sentences def to_sentences(orig_examples): for d…

So now your source (input) file would be text broken up by the new line character, so you can still use the rel.manual recipe.

@Martin

As an aside, I noticed that on wider screens, prodigy uses very little of the available space, which is a bummer for us given the quantity of text needed to grasp our documents (we work on legal decisions in France).

You can modify the card size by modifying the CSS in your configuration like:

"global_css": ".prodigy-container { max-width: 950px; }"

There are also other options (e.g., change font size) in the docs to modify the UI space.

Hope this helps!