I am writing a node.js script which uses mqtt to receive message published to a single topic. I want to execute a function that takes a data from the message and creates a list of topics to which the client subscribes. I have the working code as given below.
The problem is that I'll be having thousands devices that will be sending message to the topic - "topic/info" and some hundred might be simultaneous messages. Although the messages can be received with QoS settings in MQTT but the serial execution is not very efficient as I have a for loop running as well as a database insert (not in code just a commented line) inside the function. All the device are sending a JSON message such as - { "ID" - "abc_001" } to the topic - "topic/info".
Can somebody suggest how can I overcome this issue and handle it? Can I go with the option of having multiple processes to execute the function? Any help regarding the same is welcome as I am stuck at this. Thanking you in advance.
EDIT - I'd like to clarify that a single instance of this script will be running on a remote server. There will be multiple devices sending message to this remote server running this script using MQTT. Hence the subscriber is this one single mqtt client and there are multiple publishers.
const mqtt = require("mqtt");
const host = "0.0.0.0";
const port = "1883";
const connectURL = "mqtt://"+host + ":" + port;
const subTopics = ["ABC", "XYZ","PQR"];
function subscribeTopics(client, ID){
topic = "tch/stb/"+ID.toString()+"/";
for (let i in subTopics) {
client.subscribe(topic+subTopics[i], () => {
console.log(`Subscribe to topic '${(topic+subTopics[i])}'`)
});
}
//Perform Database Insertion
}
const client = mqtt.connect(connectURL);
client.on('connect', () => {
console.log('Connected');
client.subscribe("topic/info", () => {
console.log('Subscribe to topic topic/info')
});
})
client.on('message', (topic, payload) => {
message = JSON.parse(payload.toString());
if(topic === "topic/info"){
subscribeTopics(client, message.ID);
}
});
If you intend to use multiple instances of the script to distribute the load then you will need to use a MQTT broker that supports Shared Subscriptions (this is an optional feature added to the spec in v5 but some brokers had propitiatory implementations at v3).
Without it all instance of the script will receive ALL the messages published to a topic so they would all process every messages. With Shared Subscriptions each message will only be delivered to one instance of the client.
This blog post should help explain Shared Subscriptions in more detail: https://www.hivemq.com/blog/mqtt5-essentials-part7-shared-subscriptions/