Feedback - bug/typo + questions :)

1

Hi,

Thank you for the beta invite ! I spent some time today running through prodigy and here are some feebacks:

  • First congratulation ! The frontend is beautiful with a nice UX & the backend appears to be very flexible thanks to the recipe / component orchestration.

  • I couldn’t remove dataset, prodigy drop <dataset> seems not to be working.

  • When trying the textcat.teach recipe, I modified the input stream to use my data (and its metadata) but using 'stream': prefer_uncertain(model(stream)) remove the meta field of the input. As a result I don’t see the meta information on the UI, only the score. Is it intended ?

  • In compare / diff mode, the symbols (v,x) button in addition to the red / green color make it quite confusing. One may think that the red with symbol (x) is the wrong answer.

  • Some typo in the README:

    • “Choiche” -> “Choice” (#annotation-interfaces)
    • “taks” -> “task” ( in Annotation task formats/Choice)

  • I see that for now the classification mode only support one class. Is there a multiclass mode coming ? If not what would be your solution ? Because we don’t want to restart prodigy with a new label for each class. Plus not sure if the “active learning” part would be the most efficient that way…

  • I really like the “customizability” of input fields with spans, which combined with the choice mode allows to make a simple annotator for entity-linking for instance :slight_smile: (given a span in a context and some candidates entity, let the user choose the right entity)

Hope that helps a bit… Looking forward for your advice on multiclass annotation :slight_smile:

Thomas

2

Thanks so much for testing Prodigy and posting your feedback! :smiley: Nice to hear that it's useful and working so far!

Sorry – this was a bug in the database. It's already fixed and will be included in the next release.

Ah, I actually just came across this bug myself a few minutes ago while training a text classifier with Prodigy! Already fixed it and will be included in the next release as well. The error happened because the annotation tasks are just plain Python dictionaries and the model was accidentally overwriting the meta value instead of updating it. To prevent these issues in the future, we're thinking about introducing a Task object that Prodigy uses internally to manage and update annotation tasks, and assign the input and task hashes (used to distinguish between the original inputs and the annotated task).

I'm not sure I understand this correctly. In the compare / diff mode, the idea is that you see two answers: one suggested as the correct one, and one suggested as the wrong one. If you accept the task, you agree with this analysis – if you reject the task, you disagree. Or, phrased differently: if you prefer the green analysis, you press green. If you prefer the red analysis, you press red. For annotation projects where this seems weird or not fitting, the choice mode is probably a better option.

Thanks, fixing!

Yes, multi-labels for text classifcation are definitely in the works! For NER, you can already supply more than one label (separated by commas, e.g. --label ORG,GPE), or leave out the labels entirely to annotate all labels recognised by the model.

3

I’m following this thread with some feedback on the UI-UX part.

Some context: I’ve been using the choice interface to label news article (= title + body) with one or several labels.

  • One thing which is a bit annoying is that when cards are a bit long or at least longer than your screen, you have to scroll to see the content of the card, then make an answer and then the next card appears but it stays at the bottom of the card. It would be great if there was an autoscroll to the top when a new card appears !

  • On mobile, it would be nice if

    • one can make the menu disappear by clicking on the card (on the right…)
    • one could access the save button without opening the menu (for instance in the top bar ?)

  • Also on mobile, the answer button are “cropped” before you scroll to the bottom

  • Nice to have for the choice interface: Allowing to customize the option button with a background color depending on the content of the option :slight_smile:

Another thing, in the history on the left, there is juste “…” but I guess this is because my text input starts with a <h2> balise…

4

Edit: Both suggestions are implemented! Will be included in the next update.

Edit: This might need some refactoring of how the sidebar is currently shown/hidden on small screens. Will look into this!

Are you using Safari? I noticed this myself the other day in Safari and I think it might have something to do with how the browser places the navigation buttons at the bottom (i.e. as an overlay that only goes away once you scroll). I'll look into this and see if there's a way to fix this.

Sure – the easiest way would probably to allow a style property on the option (or tasks in general?) that lets you insert any custom CSS that's rendered as the style attribute of the specific container. This means you'd have to write and double a little more markup yourself, but it'd also keep it more flexible (if users want to add a text colour instead, or bold the text etc.)

Edit: Implemented and will be included in the next version – only for choice options for now, but may extend this to other elements as well. You can now add an additional style key with an object of CSS properties in the camel-cased JavaScript format, mapping to their values. For example:

{
    options: [
        { id: 'a', text: 'Option A', style: { background: '#ff6666' }}, 
        { id: 'b', text: 'Option B', style: { fontWeight: 'bold' }}, 
    ]
}

The little snippet shown in the history is usually taken from the text key of the task (or input.text if you're using the compare or choice mode). We originally decided against using the html key in the history, because it'd often look messy and like a glitch or leaking markup, rather than a feature (especially if the developer and annotator are not the same person). But maybe we should revisit that decision.

For now, the easiest workaround would be to add a text key to your task dictionary, containing a plain-text summary, an ID or anything else you want Prodigy to display in the history. This should also work when using the choice mode with no input (i.e. content displayed above the options). The ... is shown if no task property is found that can be used to display a snippet.

5

Nope chrome on android. I didn't tried on iOS with safari but on the two android phone I tried it was cropped.

Happy to hear that many features are already implemented. I'm waiting for the next release :slight_smile:

Concerning the history "..."

I think you're right, html fields would certainly break the design... Actually I found a very simple solution, even in choice mode. I am using an html input for the card and I added a text field with just the article title, and It nicely choose to use the html for the card content and the text for history. I don't know if it's expected but it's handy :slight_smile:

{
    "text": title, # is used to show in history
    "html": text, # is used for the card content
    "meta": {
        "source": item["source"],
        "url": item["link"],
        "date": date},
    "options": options
}