The task is:
Expected result: all lines from all workers are concatenated to resulting file
Actual result: on some machines the result is such as expected, on others the result is interlaced.
Tested on 0.10.47 and 0.12.7 nodes under Ubuntu OS. On two machines under Ubuntu 16.04.2 the result is good. On one under Ubuntu 14.04.5 is interlaced.
I need help to understand this situation and fix it on 14.04.
cluster.js:
var cluster = require("cluster");
cluster.setupMaster({
exec: "./worker.js"
});
cluster.fork();
cluster.fork();
worker.js:
var cluster = require("cluster");
var fs = require("fs");
var workerId = cluster.worker && cluster.worker.id || 1;
var stream = fs.createReadStream("./data" + workerId + ".log");
stream.on("end", function () {
process.exit();
});
stream.pipe(process.stdout);
You need two files in the same folder named data1.log and data2.log with data to read:
$ wc -l data*
9260 data1.log
111636 data2.log
120896 total
The piping always gives good result:
$ node cluster.js | wc -l
120896
The redirection to file gives good result only on Ubuntu 16.04 but never on Ubuntu 14.04. Even more on Ubuntu 14.04 it gives every time different results:
$ node cluster.js >result.log; wc -l result.log
114135 <= MUST BE 120896
$ node cluster.js >result.log; wc -l result.log
110136 <= MUST BE 120896
The redirection to file for addition always gives good result:
$ rm result.log
$ node cluster.js >>result.log; wc -l result.log
120896
Note that resulting file is lesser than original. It has less lines and even less bytes. So I suppose that lines from different workers are rewriting each other.
In general you should avoid calling process.exit() because of the problems it can cause when you have data buffered to be sent somewhere. process.exit() ends the process pretty much immediately.
In this particular case, it could very well be that process.stdout has not completely flushed its internal buffer (for when process.stdout writes are async) before the process exits.
My suggestion would be to allow the worker process to exit naturally by simply disconnecting the (unused) IPC channel in the worker via process.disconnect() and removing the process.exit() in the worker. This should allow process.stdout to drain/flush completely before the process exits.
To fix the problem one must explicitly set piping mode in master code:
var cluster = require("cluster");
cluster.setupMaster({
exec: "./worker.js",
=> silent: true
});
for (var i = 0; i < 2; i++) {
var worker = cluster.fork();
=> worker.process.stdout.pipe(process.stdout);
=> worker.process.stderr.pipe(process.stderr);
}
I think the reason for such differences in the behaviour of the same code on different versions of ubuntu is in the linux kernel file handle buffer implementations.