I need to send a video file from google cloud storage to an api, this api normally accepts fs filestreams. However this means I have to save the video file by downloading it and saving it to a file locally before sending it. I'd really like to avoid that if possible.
This is how I'm currently sending my video files to the api, it expects a fs readstream.
const filestream = fs.createReadStream('C:/Users/[me]/Downloads/testvid.mp4');
data.append('video',filestream);
axios({
method: 'post',
url: postUrl,
headers: {
'Content-Type': 'multipart/form-data',
...data.getHeaders()
},
data: data
})
What I'd like to do is essentially grab the file from the storage bucket create a readstream out of it WITHOUT saving it locally and pass it onto my axios post request.
const file = await storage.bucket("[bucket]").file("filename.mp4");
fs.createReadStream(file);
How would I accomplish this?
So far I've tried passing the google read stream into it directly, and created a passthrough stream by importing 'stream' but neither have worked. I'd appreciate any input.
In the Download objects page, you can see at the bottom a link to Create transfers using different sources and destinations like Cloud Storage or file system.
If this is not what you want to do, you can use or combine it with Streaming transfers. There is a good example of download streams using Node.js as follows:
/**
* TODO(developer): Uncomment the following lines before running the sample.
*/
// The ID of your GCS bucket
// const bucketName = 'your-unique-bucket-name';
// The ID of your GCS file
// const fileName = 'your-file-name';
// The filename and file path where you want to download the file
// const destFileName = '/local/path/to/file.txt';
// Imports the Google Cloud client library
const {Storage} = require('@google-cloud/storage');
// Creates a client
const storage = new Storage();
async function streamFileDownload() {
// The example below demonstrates how we can reference a remote file, then
// pipe its contents to a local file.
// Once the stream is created, the data can be piped anywhere (process, sdout, etc)
await storage
.bucket(bucketName)
.file(fileName)
.createReadStream() //stream is created
.pipe(fs.createWriteStream(destFileName))
.on('finish', () => {
// The file download is complete
});
console.log(
`gs://${bucketName}/${fileName} downloaded to ${destFileName}.`
);
}
streamFileDownload().catch(console.error);
I followed a solution similar to what Alex suggested but instead of getting everything from Google directly I just used a get request on Google's URL to get a stream and passed that onto my formdata as a stream. This works perfectly with so I can send a file from a remote server without downloading it.
const downStream = await axios({ method: 'GET', responseType: 'stream', url: url })
var data: FormData = new FormData();
data.append('video', downStream.data);
const response = await axios({
method: 'post',
url: postUrl,
headers: {
'Content-Type': 'multipart/form-data',
...data.getHeaders()
},
data: data,
maxContentLength: Infinity,
maxBodyLength: Infinity,
})