# Iterative meaning of data format after using bert model correct

**URL:** https://support.prodi.gy/t/iterative-meaning-of-data-format-after-using-bert-model-correct/6746
**Category:** Uncategorized
**Tags:** usage, ner
**Created:** [August 21, 2023, 4:12pm UTC](https://support.prodi.gy/t/iterative-meaning-of-data-format-after-using-bert-model-correct/6746 "2023-08-21T16:12:06Z")
**Posts on this page:** 1
**Showing post:** 2

<div class="post-metadata">

### Author: ![ryanwesslen](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ryanwesslen/32/2969_2.png) [@ryanwesslen](https://support.prodi.gy/u/ryanwesslen)
#### Post date: [August 21, 2023, 6:07pm UTC](https://support.prodi.gy/t/iterative-meaning-of-data-format-after-using-bert-model-correct/6746/2 "2023-08-21T18:07:38Z")

</div>

hi @luoshengmen98,

Thanks for your question and welcome to the Prodigy community 👋

> [@luoshengmen98](#):
>
> Therefore, only a small part of my existing annotated data conforms to the form of ([CLS] sentence, and the rest are ordinary data. Is there any impact on such processing？

It sounds like you may have misaligned tokenization, i.e., inconsistent tokenization. Likely your model had a different tokenization than your annotations, which used Prodigy's `bert.ner.manual`.

If that's the case, you may want to add tokenization to your input (source) file using the `"tokens"` key, and then Prodigy will use use that tokenization.

The [docs](https://prodi.gy/docs/named-entity-recognition#custom-model-token-mismatch) explain this and the impact of misaligned tokenization:

> Pre-tokenizing the text for the manual interfaces allows more efficient annotation, because the selection can “snap” to the token boundaries and doesn’t require pixel-perfect highlighting. You can try it out in the [live demo](https://demo.prodi.gy/?view_id=image_manual) – even if you only select parts of a word, the word is still locked in as an entity. (Pro tip: For single-token entities, you can even double-click on the word!)
> 
> Surfacing the tokenization like this also lets you spot potential problems early: if your text isn’t tokenized correctly and you’re updating your model with token-based annotations, it may never actually learn anything meaningful because it’ll never actually produce tokens consistent with the annotations.
> 
> If you’re using your own model and tokenization, you can pass in data with a `"tokens"` property in [Prodigy’s format](https://prodi.gy/docs/api-interfaces#ner_manual) instead of using spaCy to tokenize. Prodigy will respect those tokens and split up the text accordingly. If you do want to use spaCy to train your final model, you can [modify the tokenization rules](https://spacy.io/usage/linguistic-features#native-tokenizer-additions) to match your annotations or set `skip=True` in the [`add_tokens` preprocessor](https://prodi.gy/docs/api-components#add_tokens) to just ignore the mismatches.

> [@luoshengmen98](#):
>
> a preliminary bert model was formed through training

How did you do your training? Can you provide code and the setup steps?

Per the [docs](https://prodi.gy/docs/named-entity-recognition#transformers-tokenizers), this shouldn't be an issue if you train with spaCy (e.g., `spacy train`).

> spaCy v3 lets you train a transformer-based pipeline and will take care of all tokenization alignment under the hood, to ensure that the subword tokens match to the linguistic tokenization. You can use [`data-to-spacy`](https://prodi.gy/docs/recipes#data-to-spacy) to export your annotations and train with spaCy v3 and a [transformer-based config](https://spacy.io/usage/training) directly, or run [`train`](https://prodi.gy/docs/recipes#train) and provide the config via the `--config` argument.

However, if you trained with non-spaCy, you may have misaligned tokenization, which would explain the differences you're seeing.

If you want to use spaCy for training, here's a great post:

> [@Transform annotations to match tokenization required for SpanBERT/BERT](https://support.prodi.gy/t/transform-annotations-to-match-tokenization-required-for-spanbert-bert/5923/4):
>
> Gotya. Then I think the following steps are what you need. Step 1: Annotation Label some data. In my case I've made this example dataset: {"text": "hi my name is Vincent"} {"text": "people call me Noa"} And I've annotated these two examples in Prodigy via: python -m prodigy ner.manual issue-5923 blank:en examples.jsonl --label name This saves the NER annotations in a dataset named issue-5923. Step 2: Convert Next, let's prepare the data so that spaCy can use it. python -m prodigy data-t…

Also - if you have future examples, can you post examples using Markdown instead of posting images? Images can't be searched/indexed, and all we'd need is one example. It was a bit hard to compare the two examples you had by image. But thanks for the details!

---

_[View the full topic](https://support.prodi.gy/t/iterative-meaning-of-data-format-after-using-bert-model-correct/6746)._