I've looked through all of the documentation, and I'm having some trouble finding examples of the correct implementation. First of all, I have the Heroku Redis add-on as you can see below:
https://i.stack.imgur.com/9kcXW.png
I installed Redis and Bull to my Node.js environment and set up Redis as follows:
const redis = require('redis');
const Queue = require('bull');
const redisClient = redis.createClient(process.env.REDIS_URL, {
tls: {
rejectUnauthorized: false
}
});
const bullQueue = new Queue('queue', process.env.REDIS_URL)
I'm trying to run the below function as a background task (this is what Heroku recommends for functions that take longer than 0.5 seconds to complete):
app.post('/', async function(request, response) {
const client = await pool.connect() //pool is just a node-postgres pool
try {
await client.query('BEGIN')
let data = await function1(argumentList);
await function2(data);
await client.query('COMMIT')
} catch(err) {
await client.query('ROLLBACK')
console.log(err)
}
try {
await client.query('BEGIN')
const setOfItems = await function3()
var array = []
for (item of setOfItems) {
if (conditional) {
array.push(item)
}
}
await client.query('COMMIT')
let job = await bullQueue.add()
response.send(JSON.stringify(array))
} catch(err) {
await client.query('ROLLBACK')
console.log(err)
}
});
This function does some web scraping, database calls, and other things, so like I said it takes a few seconds to complete. How should I change my function to add the request to the background queue and then return the JSON.stringified 'array' back to the user. Sorry if this is a noob Redis question, but I've looked at all the docs and I really am not sure how to proceed.
For starters, const redis = require('redis'); isn't necessary. Bull connects to Redis under the hood, so you need only provide process.env.REDIS_URL for Heroku as you're doing. (heroku config | grep REDIS returns the endpoint URLs if needed)
Currently, all of your database code isn't delegated to the task queue at all. Try registering a processing function associated with the queue using
bullQueue.process(job => {/* your database work goes here */})
The parameter job needs to be populated with whatever serializable data your worker needs to do its job. You populate this parameter using
bullQueue.add(/* job data goes here */)
A high-level approach might be something like:
const Queue = require('bull');
const bullQueue = new Queue('queue', process.env.REDIS_URL)
bullQueue.process(async job => {
console.log(job.data.argumentList);
/* do SQL stuff */
return [1, 2, 3, 4];
});
app.post('/', async (request, response) => {
const job = await bullQueue.add({argumentList: ['some', 'data']});
const array = await job.finished();
response.send(JSON.stringify(array))
});
Many other strategies exist for getting the results. See returning job completions for some ideas.
See also: