db-out utf-8 character problem

[ines (Ines Montani)]

Hi! This is just the default behaviour of json.dumps, which is called under the hood to export your data. It's the safest way to represent utf8 and prevent encoding issues. When you load the text back in Python etc., the characters will look as expected again. You can re-export the data without ASCII-only characters – you just need to be careful you don't end up with encoding issues. Also see here for details:

Language with macrons (āēīōūĀĒĪŌŪ) in output

Yes, under the hood, that justs saves the JSON. If you load the exported data back into Python, you'll see the original unicode characters. For example: import srsly data = srsly.read_jsonl("/path/to/your_file.jsonl") print(data[0]) Saving data as ASCII (e.g. with \u) is how JSON data is saved by default in Python – this has nothing to do with Prodigy. It's a useful default, because it prevents encoding issues. You can decide to not store your text as ASCII by saving out the data again like t…