Using JavaScript: I have an array of objects that I'm trying to determine the duplicate entries of so that I can eventually pass those duplicate entries to a separate function to remove them from a database.
My sample array could be:
const myArray = [
{ 'id': 111, 'lorem': 'ipsum' },
{ 'id': 222, 'lorem': 'dorem' },
{ 'id': 111, 'lorem': 'polus' },
{ 'id': 111, 'lorem': 'waifu' },
]
I'd want to return an array of all items that would be duplicate by the key id. In this example, my returned array would be:
[
{ 'id': 111, 'lorem': 'ipsum' },
{ 'id': 111, 'lorem': 'polus' },
{ 'id': 111, 'lorem': 'waifu' },
]
Most of the online tutorials have me iterating over a short list of data, and is great for such small data examples. But my dataset is in the thousands, if not millions, as my data grows. So I'm trying to find a smarter way of handling this logic.
I understand that I can run a Set(), but that doesn't actually give me the duplicate entries - that gives me an array with non-duplicates. My need is to return such duplicates, not to have a new array of non-duplicate entries.
Without using a third party such as lodash or underscore, how would I ideally iterate over an array with unknown size, to eventually return the duplicate items for me to pass up the stream for processing?
Array#reduce, iterate over the array while updating a Map to group items by idMap#values, get the list of grouped arraysArray#filter, keep the arrays with more than one itemArray#flat, return all arrays in one listconst myArray = [ { 'id': 111, 'lorem': 'ipsum' }, { 'id': 222, 'lorem': 'dorem' }, { 'id': 111, 'lorem': 'polus' }, { 'id': 111, 'lorem': 'waifu' } ];
const duplicates =
[...myArray.reduce((map, item) => // group items by id
map.set(item.id, [...(map.get(item.id) || []), item])
, new Map)
.values()] // get grouped arrays
.filter(list => list.length > 1) // keep duplicates
.flat(); // return one array
console.log(duplicates);This may take a while
I recommend you do this on the server
// create an array with random IDs
const myArray = []
for (let i = 0; i < 10000; i++) {
myArray.push({
id: String(Math.floor(Math.random() * 10000)).padStart(3, "0"),
"lorem": "ipsum"
})
}
// examine them
const ids = []
const dupes = []
myArray.forEach(({id}) => {
if (ids.includes(id)) dupes.push(id);
else ids.push(id)
})
console.log(ids.length, dupes, dupes.length)Expanding on my comment:
const myArray = [
{ 'id': 111, 'lorem': 'ipsum' },
{ 'id': 222, 'lorem': 'dorem' },
{ 'id': 111, 'lorem': 'polus' },
{ 'id': 111, 'lorem': 'waifu' },
];
const uniqueItems = new Set();
const duplicates = [];
myArray.forEach(
function(a) {
if (uniqueItems.has(a.id)) {
duplicates.push(a);
}
uniqueItems.add(a.id);
}
);
console.log("duplicates",duplicates);
console.log("uniqueItems",uniqueItems);