It's bit strange to me the way stream is behaving sometime. Simple task at work is to pull a image from S3 and transform it using sharp library; finally upload resized image back to S3. I can see resized image in S3 but code fails with error.
Here is a code snippet doing all the work -
// Read from S3
const readStream = s3Handler.readStream(params)
// resize Stream
let resizeStream = sharp()
resizeStream
.metadata()
.then((metadata) => {
// Resize the image to a width specified by the `percent` value and output as PNG
sharpOptions = { width: width*(metadata.percent), height: height*(metadata.percent), fit: 'contain' }
resizeStream
.resize(sharpOptions)
.toFormat('png')
.png({ quality: 100, compressionLevel: 5 })
.toBuffer()
//return streamResize.resize(sharpOptions).png().toBuffer()
})
// push to s3
const { writeStream, uploaded } = s3Handler.writeStream({
Bucket: bucket,
Key: key,
Ext: ext,
})
readStream.pipe(resizeStream).pipe(writeStream)
await uploaded
Here is the error I get from above code -
{
"errorType": "Error",
"errorMessage": "stream.push() after EOF",
"code": "ERR_STREAM_PUSH_AFTER_EOF",
"stack": [
"Error [ERR_STREAM_PUSH_AFTER_EOF]: stream.push() after EOF",
" at readableAddChunk (_stream_readable.js:260:30)",
" at Sharp.Readable.push (_stream_readable.js:213:10)",
" at Object.<anonymous> (/var/task/node_modules/sharp/lib/output.js:863:18)"
]
}
Any suggestions much appreciated.
You're trying to do something which is currently a bit hard with node-sharp: see https://github.com/lovell/sharp/issues/236
In short, the problem is that .metadata() requires reading the entire image into memory (as a Buffer) to compute values. In addition to that, relative resizing (percentage of original size) creates a data dependency - you need to know the image dimensions before you can compute the target width and height.
Given how you're using Sharp as a pass-through stream (S3 read → resize → S3 write), calling .metadata() on it will not work, because it happens too late: you need to set up the resize options before you pipe the data in (so that the resize process may know the target size). At the same time, you need to pipe all the data in first, because .metadata() needs the complete image. The two needs conflict and make it impossible to have a single Sharp pass-through Stream that does everything.
You're left with a few options:
If you're not dealing with very big images, 2. is decidedly the easiest way to implement this:
const original = (await s3.getObject(getParams).promise()).Body;
const metadata = await sharp(original).metadata();
const resized = await sharp(original).resize({
// NOTE: This computation was bugged in the original example - there's no metadata.percent, it's our own setting.
width: metadata.width * percent,
height: metadata.height * percent,
fit: 'contain'
}).toFormat('png').png({ quality: 100, compressionLevel: 5 }).toBuffer()
await s3.putObject({
...putParams,
Body: resized
});