I am working on a scraping script and found about Set object which should store unique datas and fast in performance. So I tried it like this
let scrapedMessages = new Set()
scrapedMessages.add({
text,
...(images.length > 0 && {
images,
}),
senderID,
timestamp,
})
But when looking at the data scraped I found out these type of duplicate datas
{
"text": "Acne Fighting Facial Wash With Jojoba Beads",
"senderID": "361571627329333",
"timestamp": "1613017270619"
},
{
"text": "Acne Fighting Facial Wash With Jojoba Beads",
"senderID": "361571627329333",
"timestamp": "1613017270619"
}
Does it mean Set of Objects might not be unique or I am doing some mistakes. I was doing it through simple array. But shifted to Set type for better performance. Can it be achieved? What is the best practice?
I am running the puppeteer script on Nodejs.
JS use === operator to compare the new element to all others which already exists in the set and based on that it decides to either add the new element or ignore it.
The problem is, objects are address of memory:
const a = {x: 1};
const b = {x: 1};
const c = a;
console.log(a===b); // false
console.log(a===c); // true
console.log(b===c); // falseSo here there are couple of things you can do:
Extend the Set class and override the add method to compare the object's properties instead of using ====.
NOT RECOMMENDED but you can simply use JSON.stringify(object) before adding it.