I've written the code below to illustrate this behaviour that happens when concatting these two string.
const getBytes = x => {
let buf = Buffer.from(x);
const n = [];
for (const value of buf.values()) {
n.push(value);
}
return [n, parseInt(buf.toString('hex'), 16)];
};
let x = unescape('%uDB40');
let y = unescape('%uDD31');
console.log(typeof(x), typeof(y));
console.log(Buffer.from(x), getBytes(x), );
console.log(Buffer.from(y), getBytes(y));
console.log(Buffer.from(x+y), getBytes(x+y));
The result is:
string string
<Buffer ef bf bd> [ [ 239, 191, 189 ], 15712189 ]
<Buffer ef bf bd> [ [ 239, 191, 189 ], 15712189 ]
<Buffer f3 a0 84 b1> [ [ 243, 160, 132, 177 ], 4087383217 ]
I'm unable to understand how it ends up as a completely different result which is preventing me from successfully porting this behaviour.
As the Buffer.from documentation states: "When converting between Buffers and strings, a character encoding may be specified. If no character encoding is specified, UTF-8 will be used as the default."
To convert from a JavaScript string (UTF-16) to a Buffer (UTF-8), you wrote:
let buf = Buffer.from(x);
If a UTF-16 to UTF-8 character conversion fails, it writes the Unicode replacement character to Buffer, as prescribed by the Unicode Standard.
I used these links to answer your question: Node.js: Buffer, U+DB40, U+DD31, U+FFFD, U+E0131.
For a JavaScript reference: JavaScript: The Definitive Guide, by David Flanagan.
For a Unicode reference: Unicode Standard.