I saved the json file in python like this
with io.open('tokenizer.json', 'w', encoding='UTF-8') as f:
f.write(json.dumps(tokenizer_json, ensure_ascii=False))
Then when I try to open the file in javascript, the json is read as a raw file
const js = JSON.parse(mytokenizer);
//this does not work properly
//result = {"\uc544\ub2c8\ub2e4": 1, "\uc788\ub2e4": 2, "\uac00\ub2a5\ud558\ub2e4": 3, "\uc5c6\ub2e4": 4, "\ub9c8\uc2a4\ud06c": 5, "\ud55c\ub2e4": 6, "\uc815\ubd80\uac00": 7, "\ub3c5\uac10": 8, "\ubc31\uc2e0\uc758": 9, "\uc54a\ub2e4": 10, "\uad00\uacc4\uac00": 11,
However it works fine when I open this file in python
with open('tokenizer.json') as f:
data = json.load(f)
loaded_tokenizer = tokenizer_from_json(data)
//works fine
//result = {'아니다': 1, '있다': 2, '가능하다': 3, '없다': 4, '마스크': 5, '한다': 6, '정부가': 7, '독감': 8, '백신의': 9, '않다': 10, '관계가': 11, '마스크를': 12,
what is the matter?
you need to set encoding when you open the file:
reader.readAsText(file, 'CP936');