Given the following NodeJS Transform Stream:
class ObjectToCSVTransform extends Transform {
private hasSetHeaders: boolean;
constructor() {
super({ objectMode: true });
this.hasSetHeaders = false;
}
static create(): ObjectToCSVTransform {
return new this();
}
_write(
object: Record<string, unkwnown>,
_encoding: BufferEncoding,
callback: TransformCallback
): void {
this.push(this.generateCSV(telemetry));
callback();
}
private generateCSV(object: Record<string, unkwnown>): string {
let csv = '';
if (!this.hasSetHeaders) {
csv += this.createCSVHeaders(object);
this.hasSetHeaders = true;
}
csv += this.createRecord(object);
return csv;
}
private createCSVHeaders(object: Record<string, unkwnown>): string {
return `${Object.keys(object)}\n`;
}
private createCSVRecord(object: Record<string, unkwnown>): string {
return `${Object.values(object)}\n`;
}
}
I implemented the following test case (using jest), to test that given a stream of plain objects of the "same type" the expected output is a valid CSV representation of them:
describe('object to csv stream', () => {
const items: Record<string, unknown> = [
{ foo: 1, bar: 2, baz: 3 },
{ foo: 10, bar: 20, baz: 30 },
{ foo: 100, bar: 200, baz: 300 },
];
it('should transform a list of items to csv', (done) => {
const expectedCsv = 'foo,bar,baz\n1,2,3\n10,20,30\n100,200,300\n';
let csv = '';
Readable.from(items)
.pipe(ObjectToCSVTransform.create())
.on('data', (csvResult) => {
csv += csvResult;
console.log(csvResult); // just for debugging purposes
})
.on('end', () => {
console.log('streaming ended'); // just for debugging purposes
expect(csv).toEqual(expectedCsv);
done();
});
});
});
Apparently, during the test case implementation I wanted to see how my test case failed: the stream seemed to work as it logged every expected csv result line and the 'streaming ended' message also at the end once the stream ended, but the test execution did not finish at all then. It was jest that actually finished the execution after exceeding the default timeout:
thrown: "Exceeded timeout of 5000 ms for a test. Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."
What am I missing here?
Note that I am currently using the jest callback to notify when the async operation has finished.
The problem is that the returned value of csv is not matching the expected value. The actual returned value includes a \n at the end. If you add this to your expectedCsv string, your test will pass.
The timeout error is a red herring.
If you console.log(csv.replace(/\n/g, '\\n')) in the 'end' function you will see the extra linefeed at the end of the string.
import { Readable, Transform, TransformCallback } from 'stream';
class ObjectToCSVTransform extends Transform {
private hasSetHeaders: boolean;
constructor() {
super({ objectMode: true });
this.hasSetHeaders = false;
}
static create(): ObjectToCSVTransform {
return new this();
}
// eslint-disable-next-line no-underscore-dangle
_write(
object: Record<string, unknown>,
_encoding: 'utf8',
callback: TransformCallback,
): void {
this.push(this.generateCSV(object));
callback();
}
private generateCSV(object: Record<string, unknown>): string {
let csv = '';
if (!this.hasSetHeaders) {
csv += this.createCSVHeaders(object);
this.hasSetHeaders = true;
}
csv += this.createCSVRecord(object);
return csv;
}
private createCSVHeaders(object: Record<string, unknown>): string {
return `${Object.keys(object)}\n`;
}
private createCSVRecord(object: Record<string, unknown>): string {
return `${Object.values(object)}\n`;
}
}
describe('object to csv stream', () => {
const items: Record<string, unknown>[] = [
{ foo: 1, bar: 2, baz: 3 },
{ foo: 10, bar: 20, baz: 30 },
{ foo: 100, bar: 200, baz: 300 },
];
it('should transform a list of items to csv', (done) => {
const expectedCsv = 'foo,bar,baz\n1,2,3\n10,20,30\n100,200,300\n';
let csv = '';
Readable.from(items)
.pipe(ObjectToCSVTransform.create())
.on('data', (csvResult) => {
csv += csvResult;
console.log(csvResult); // just for debugging purposes
})
.on('end', () => {
console.log('streaming ended'); // just for debugging purposes
console.log(csv.replace(/\n/g, '\\n'));
expect(csv).toEqual(expectedCsv);
done();
});
});
});
There is something that I was missing in my test case that wasn't handling asynchrony correctly at all and therefore was making the Exceeded timeout exception to be thrown by jest:
done() callback called in case the assertion failed..on('end', () => {
try {
expect(csv).toEqual(expectedCsv);
done();
} catch (error) {
done(error);
}
});
For more info / reference, check the jest docs: https://jestjs.io/docs/asynchronous#callbacks