# Create training and testing data for triples **URL:** \ **Category:** Uncategorized\ **Tags:** usage, ner, done, custom, relations\ **Created:** [March 31, 2020, 1:22pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734 "2020-03-31T13:22:07Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![aila1095](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@aila1095](https://support.prodi.gy/u/aila1095)\ **Post date:** [March 31, 2020, 1:22pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/1 "2020-03-31T13:22:07Z")
Hi, I am new to NLP, prodigy, and spaCy so please pardon my basic questions. Our team has been given a NER model which can identify 3 custom entities. Now, we need to find the relationship between these entities and generate triples (Subject, Object, Predicate) using spaCy and prodigy. We are using the dependency parse tree to find the relationships. I have 2 questions regarding this: 1. We have a huge dataset of research papers. Is there any way we can train our model to improve the dependency parse tree? Can we achieve this using prodigy? 2. For our complete pipeline, we would like to create testing data of triples. Any suggestions on how we can do this using prodigy? ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [March 31, 2020, 6:14pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/2 "2020-03-31T18:14:17Z")
Hi! Your workflow sounds good πŸ™‚ > [@aila1095](#): > > We have a huge dataset of research papers. Is there any way we can train our model to improve the dependency parse tree? Can we achieve this using prodigy? Prodigy comes with a workflow called [`dep.teach`](https://prodi.gy/docs/recipes#dep-teach) that lets you collect training data to improve an existing dependency parser in the loop. The model will suggest possible analyses of the given parse with the most uncertain scores and you can accept and reject them. This lets you work around the otherwise pretty tedious manual annotation process for dependencies, and can work well if what you mostly care about are the most common dependency labels like `nsubj` or `dobj` (which seems to be the case for your pipeline). After you've collected the annotations, you can run the `train` recipe with `--binary` to update from binary annotations. I'd also recommend evaluating the models you're training on your full information extraction pipeline so you can see the impact of the parser updates on the full downstream task. > [@aila1095](#): > > For our complete pipeline, we would like to create testing data of triples. Any suggestions on how we can do this using prodigy? I'm not 100% sure I understand the question correctly. Do you mean creating evaluation data for your model and annotating everything (entities and relationships) together? ---
**Author:** ![aila1095](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@aila1095](https://support.prodi.gy/u/aila1095)\ **Post date:** [April 1, 2020, 6:32am UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/3 "2020-04-01T06:32:38Z")
Thank you so much for the prompt reply. This is very helpful and is exactly what we were looking for. > [@ines](#): > > I'm not 100% sure I understand the question correctly. Do you mean creating evaluation data for your model and annotating everything (entities and relationships) together? Yes, we would like to create some evaluation data. I was thinking of annotating some triples manually and use that to evaluate the output of our pipeline. I would love to hear your views about it and if I could make this process of creating evaluation data (triples) faster. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [April 1, 2020, 9:08am UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/4 "2020-04-01T09:08:24Z")
How many of these triples do you typically expect per sentence? Only one? And do you also want to label the original entities manually, or can you just use what's predicted by the model? If it's typically one triple and you want to label everything manually, you could model the annotation task as a manual span labelling task (like `ner.manual`) with 3 labels, `SUBJECT`, `OBJECT`, `PREDICATE` (or similar)? Alternatively, if your NER model is supposed to identify the spans and you're only annotating the relationships, you could stream in one task per span, plus multiple choice options for subject / predicate / object (and maybe other / none). So you're always annotating the current span. You could even use a special label for the span you're focusing on and then set a [custom color](https://prodi.gy/docs/api-web-app#theme-highlight-colors) – so at any given point, you can see all entities predicted in the text and the one you're currently labelling highlighted in a special colour. ---
**Author:** ![aila1095](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@aila1095](https://support.prodi.gy/u/aila1095)\ **Post date:** [April 2, 2020, 10:19pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/5 "2020-04-02T22:19:46Z")
Thank you for the recommendations. ner.manual seems to fit our use case better. We typically have only one triple per line and are conveniently able to mark that using Prodigy. But about 25% of the times, we have multiple triples in a sentence. Eg Obama was born in Hawaii and studied in Chicago. So the triples we have are: Obama =\> was born in =\> Hawaii Obama =\> studied in =\> Chicago Is there an easy way to handle such a case? One way is to duplicate such input lines so that we still end up marking one triple per sentence but I was wondering if you have any other suggestions. I am not sure how to duplicate lines easily after we have already loaded the source file and are annotating using the web tool. ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [April 6, 2020, 9:41am UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/6 "2020-04-06T09:41:12Z")
Yes, I definitely see the problem here, hmm πŸ€” One option would be to start by doing the 75% first and then deal with the remaining 25% with multiple relations separately. In general, Prodigy avoids anything related to having the annotator modify the actual examples to annotate (e.g. duplicate examples) because that's something that should typically be decided on the development level. But this case is kind of an exception here. One option could be to set `"instant_submit": true`, which will send annotated examples back immediately. And in your recipe, you could then keep sending the same example (e.g. with a `while True` loop) and only break and move on to the next one if you've received the same example with a `"reject"` answer. So once there are no more relations in the example, you reject it empty and the server moves on to the next. (Just make sure to use distinct `"_task_hash"` values for the duplicates to prevent Prodigy from filtering them.) Also, on a related note: This thread actually inspired me to do some experimentation for use cases like this to better support this type of relation annotation, while also keeping the workflow efficient. Still super experimental, but here's the one screenshot I remembered to take, complete with messy test data πŸ˜… ![Screenshot 2020-04-05 at 19.14.07](https://us1.discourse-cdn.com/flex020/uploads/prodigy/original/2X/f/f177cc98d82a68501d16341f96daefca0b3b16f6.png) The idea would be to allow streaming in pre-tokenized data with merged entities and phrases (and optionally disable all other tokens to only make the relevant spans selectable). Don't have an ETA yet for the beta, but I'll definitely share it on the forum once it's ready for testing πŸ™‚ ---
**Author:** ![aila1095](https://avatars.discourse-cdn.com/v4/letter/a/8c91f0/32.png) [@aila1095](https://support.prodi.gy/u/aila1095)\ **Post date:** [April 20, 2020, 5:09pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/7 "2020-04-20T17:09:06Z")
Sorry for the late reply, Ines! I really appreciate your suggestions and it seems to fit our use case. Thanks a lot πŸ˜ƒ ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [May 11, 2020, 10:10am UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/8 "2020-05-11T10:10:39Z")
@aila1095 Update πŸŽ‰ > [@sparkleslink Beta testers wanted: new manual dependencies & relations UI (v1.10)](https://support.prodi.gy/t/beta-testers-wanted-new-manual-dependencies-relations-ui-v1-10/2893): > > I already mentioned this briefly in the call for [image UI beta testers](https://support.prodi.gy/t/beta-testers-wanted-new-manual-image-ui-v1-10/2840): the upcoming Prodigy v1.10 will feature a completely new interface and a suite of recipes for efficient relation and dependency annotation tada We're especially excited about this one, since it's been one of the missing features in Prodigy and it took a while to develop efficient solutions for the various challenges involved. Features include manually annotate labelled dependencies and relationships between words and phr… ---
**Author:** ![ines](https://sea2.discourse-cdn.com/flex020/user_avatar/support.prodi.gy/ines/32/3_2.png) [@ines](https://support.prodi.gy/u/ines)\ **Post date:** [June 17, 2020, 5:39pm UTC](https://support.prodi.gy/t/create-training-and-testing-data-for-triples/2734/9 "2020-06-17T17:39:31Z")
Just released Prodigy v1.10, which includes the new `relations` UI and recipes for implementing different types of relation annotation workflows! > [@rocket Out now: Prodigy v1.10 (plus new video!)](https://support.prodi.gy/t/out-now-prodigy-v1-10-plus-new-video/3030): > > Prodigy v1.10 is out now tada And it includes a bunch of cool new features, including dependency and relation annotation, audio and video annotation, an enhanced manual image UI, more settings for NER annotation, including character-based highlighting and a cool example workflow for creating training data for fine-tuning transformer models, new recipe callbacks for validating answers at runtime and for modifying examples before the database, new settings for UI customisation, and lots more! Th…