Got this pseudo code for a scraping service that first explores the pagination urls then the subpage urls on the scraped "parent" website
const pagination = [["url-1", "url-2", "url-3", "url-4"], ["url-5"]];
const timeInterval = 2;
sendRequestsInIntervals(pagination, timeInterval )
.pipe()
.subscribe({
next(x) {
console.log(x);
},
error(err) {
console.error(err);
},
complete() {
console.log("done");
sendRequestsInIntervals(resDataArray, timeInterval ).subscribe({
next(x) {
console.log(x);
},
});
},
});
I wanna avoid nesting as its an incorrect way to use observables
Is there a way to convert this to something like this:
sendRequestsInIntervals(pagination, timeInterval )
.pipe(
waitUntilCompletes()
mergeMap((resDataArray) => {
return sendRequestsInIntervals(resDataArray, timeInterval );
})
)
.subscribe({
next(x) {
console.log(x);
},
error(err) {
console.error(err);
},
complete() {
console.log("done");
},
});
Added a pseudo function called waitUntilCompletes()
Is there such a thing in rxJS that it makes the observable in the mergeMap wait before runing until the previous observable is completed?
the last operator will ignore all emissions until the final one upon completion:
sendRequestsInIntervals(pagination, timeInterval )
.pipe(
last(),
mergeMap((resDataArray) => { // resDataArray is only the final emitted value from sendRequestsInIntervals()
return sendRequestsInIntervals(resDataArray, timeInterval );
})
)
.subscribe({
next(x) {
console.log(x);
},
error(err) {
console.error(err);
},
complete() {
console.log("done");
},
});
or you can use reduce to gather the emissions and emit once complete:
sendRequestsInIntervals(pagination, timeInterval )
.pipe(
// this will gather all outter emissions into an array and emit once outter completes
reduce((acc, val) => acc.concat(val), []),
mergeMap((resDataArray) => { // resDataArray is all values emitted from sendRequestsInIntervals() in an array
return sendRequestsInIntervals(resDataArray, timeInterval );
})
)
.subscribe({
next(x) {
console.log(x);
},
error(err) {
console.error(err);
},
complete() {
console.log("done");
},
});