I'm using Meteor framework with Blaze. How can I fetch data from an API and only insert new data in my MongoDB collection and not duplicates?
Fetching data from the API.
if (Meteor.isServer) {
Meteor.methods({
fetchApiData: function () {
this.unblock();
return Meteor.http.call('GET','http://jsonplaceholder.typicode.com/posts');},
Insert data into database:
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
myCollection.insert({
//upsert: true,
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
},
When using "myCollection.update" with "upsert: true" it does not insert new entries obviously. What is best practice to go about checking the API for data and inserting ONLY new entries with no duplicates and updating existing entries?
Thank you.
here's how i handle what i call reference data at startup. it's driven off of JSON data. you have to pick a field that serves as your "reference" for each JSON object, so you can see if it's already in the db.
_.each(ItemData.items, function(q) {
check(q, ItemsSchema);
Items.upsert({
item: q.item
}, {
$set: {
item: q.item,
}
}, function(error, result) {
if (error) {
let errMsg = 'Error while writing item data';
console.error(errMsg, error);
throw new Meteor.Error('500', errMsg);
}
});
});
i use an upsert to handle insert vs update.
I'm not familiar with your specific framework, so I can't help with syntax, but you should be able to find all documents with the same properties as the document you're trying to insert (there should be only one). If there is one, then save it using upsert. If there isn't, then the object you're saving is unique, and you should save a new one.
Using only "vanilla" Meteor, assuming your api object have unique ids and that you have the proper data access (ie, if item exist findOne would find it), I'd use :
populateDatabaseApi: function () {
Meteor.call('fetchApiData', function(error, result) {
var item = myCollection.findOne({A : result.data.id})
if(item){
//do nothing, this item already is in the db
}else{
myCollection.insert({
A: result.data.title,
B: result.data.userId,
C: result.data.id });
});
}
},