Token indices sequence length is longer than the specified maximum sequence length for this model

[ryanwesslen]

hi @foscraft!

Does this help? See the linked spaCy discussions post too for more details.

Token indices sequence length is too long

This is a warning coming internally from transformers or tokenizers and you don't see actual errors because long sequences are truncated internally before they're passed to the model. If it happens rarely, you can probably ignore it. If it's frequent, you may want to adjust the window and stride for the transformer span getter in your config. See: Receiving the warning messgae 'Token indices are too long' even after validating doc length is under max sequence length · Discussion #9277 · explosi…

Also you may want to search more in spaCy discussions or spacy-llm discussions pages for tips and similar examples.

For example, for your out-of-memory issue:

How to train with a dataset that doesn't fit into memory · explosion/spaCy ·...

Since your problem is really a spaCy issue, you'll likely find a lot more helpful posts there (let alone the spaCy core dev team answers questions there).

Hope this helps!