I'm polling(using node-cron) an external service every N minutes for an array of data, each element in the array has a unique id. How can I setup my bull queue such that if I add an element of data to be processed I cannot add(and consequently reprocess) the same element to the queue if the element of data is already being processed or is already processed/completed, however if the element of data fails to be processed for whatever reason it can be re-added to the queue to attempt processing again.

The following is a rough setup of my task.

import BullQueue from 'bull';
import cron from 'node-cron';

const myQueue = new BullQueue('my-queue');

myQueue.process((job, done) => {
    // while job is processing do not allow a new job with same uniqueLogId to be added
    console.log('job.id:'); // job id auto-generated by bull
    console.log(job.id);

    console.log('uniqueLogId:');
    console.log(job.data.id);

    const result = myProcessor(job.data);
    if (result.ok) {
        // in this case a new job with uniqueLogId should not be allowed to be created
        done();
    } else {
        // in this case a new job with uniqueLogId should be allowed to be created
        done(new Error('job failed'));
    }
});

// run every 10 minutes
cron.schedule('*/10 * * * *', () => {
    pollExternalAPI() // can return the same log results
        .then((res) => {

            if (res.ok) {
                const logs = res.logs;
                for (const log of logs) {
                    // log.id uniquely identifies a job,
                    // hence jobs with the same id should not be allowed to be created provided the last job is either pending or complete
                    // however if job failed then create another job to attempt processing the given log data
                    myQueue.add(log);
                }
            }
    });
});