0
142
ViewsJavascript: compare two strings with actually different encoding
I am comparing two presumably differently encoded file names, in Javascript, with the hope to find matches:
- One file name is an actual file name, from within a unarchived zip (using https://stuk.github.io/jszip/)
- One file name is a name extracted from a bplist (iOS archive format unarchived with https://github.com/joeferner/node-bplist-parser)
Analysis
When comparing the log output in the javascript console, these file names look exactly identical:
15 - Beschänkt und gsägnet - PLAYBACKVERSION.mp3
15 - Beschänkt und gsägnet - PLAYBACKVERSION.mp3
Note the german umlauts.
Now, when I just copy and paste these strings into Notepad++ and enable the hex editor, it looks like this:
- In the first case the A-Umlaut is encoded with 3 (three) bytes
- In the second case the A-Umlaut is encoded with only 2 (two) bytes.
Question
How can I safely compare those two strings. Is there a general "unencode" method in Javascript that can handle these instances? Or should I / must I guess each encoding and then compare explicitly?
Note
- I am specifically asking for a solution in javascript
- This question, Compare strings with different encodings althoug similar is not actually about encoding
1 answers
Answer question0
What's happening here?
If you have a String in JavaScript, it's a sequence of Unicode codepoints. Some component has already decoded the bytes representing those strings from the ZIP or the plist into a sequence of codepoints.
That is, this question is not quite about encodings, but about Unicode decomposition and normalization forms.
It's possible to encode an ä in (at least) two different ways in Unicode (examples below in Python due to the useful outputs).
>>> "ä".encode("UTF-8")
b'\xc3\xa4' # two bytes
>>> [ord(c) for c in "ä"]
[228]
>>> [unicodedata.name(c) for c in "ä"]
['LATIN SMALL LETTER A WITH DIAERESIS']
or in the NFKD normalization form, taking two codepoints and three bytes in UTF-8.
>>> unicodedata.normalize("NFKD", "ä").encode("UTF-8")
b'a\xcc\x88' # three bytes
>>> [ord(c) for c in unicodedata.normalize("NFKD", "ä")]
[97, 776] # two codepoints
>>> [unicodedata.name(c) for c in unicodedata.normalize("NFKD", "ä")]
['LATIN SMALL LETTER A', 'COMBINING DIAERESIS']
Answer
Long story short, in JavaScript, you'll need to call String#normalize() to make sure the strings are in the same normalization form before attempting regular comparison.
$ node
Welcome to Node.js v16.6.1.
Type ".help" for more information.
> var a = '15 - Beschänkt und gsägnet - PLAYBACKVERSION.mp3';
undefined
> var b = '15 - Beschänkt und gsägnet - PLAYBACKVERSION.mp3';
undefined
> a.length
50
> b.length
48
> a === b
false
> a.normalize() === b.normalize()
true
>