Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

162
Views
Trying to prevent duplicate entries not working

I am fetching data from an API (array of objects) and I want to then check in the database if any of the records exist, using the url as the basis for duplicates. When it comes to inserting the new records, I get this monogoDB error:

MongoBulkWriteError: E11000 duplicate key error collection: news.news index: sourceUrl_1 dup key: { sourceUrl: "https://example.com/article-abc123" }

  try {
    const covidDbArticles = await News.find({ category: "Coronavirus" });
    const filterCovidDuplicates = covidData.data.news.filter(
      (covidApiSource) =>
        !covidDbArticles.some(
          (covidDatabaseSource) =>
            covidApiSource.link === covidDatabaseSource.sourceUrl
        )
    );
    if (filterCovidDuplicates.length) {
      try {
        await News.insertMany(covidNewsObj);
      } catch (err) {
        console.log("Error inserting covid data: " + err);
      }
    }
  } catch (err) {
    console.log("saving data failed: " + err);
  }
about 4 years ago · Juan Pablo Isaza
1 answers
Answer question

0

Filter does not change the original array. So when you use covidNewsObj, you are sending up all the duplicates.

So you would need to set the result of the filter back to the property.

covidData.data.news = covidData.data.news.filter(...);
about 4 years ago · Juan Pablo Isaza Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!