Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

486
Views
NodejS Read/Write Binary Data to/from file

Alright, so I have a string with binary data ("111011 10001 etc"), and I'm trying to save it to a file to them read it on another file using streams, the issue is that, the stream is cutting the data off (final binary number in chunk is cut off)enter image description here

This is how I'm sending the data to the file (reading a file, encoding it with the golombRice encoder and storing it in a file, using chunks of data)

  const writer = fs.createWriteStream(
    `./encodedAndDecoded/encoded${filename}`,
    {
      encoding: "binary",
    }
  );
  const reader = fs.createReadStream(`./silesia/${filename}`, {
    encoding: "base64",
  });

  await new Promise((resolve, reject) => {
    reader.on("data", (chunk) => {
      writer.write(Buffer.from(golombRiceEncoding(chunk, 3)));
    });
    reader.on("end", () => {
      writer.end();
      resolve();
    });
  });

This is how I'm reading it (reading the encoded file, decoding it with the golombRice encoder and storing it in a file, using chunks of data, the issue with this is that the chunks don't have the full binary data because the stream cuts it)

  const writer = fs.createWriteStream(`./encodedAndDecoded/decoded${filename}`);

  const reader = fs.createReadStream(`./encodedAndDecoded/encoded${filename}`, {
    encoding: "binary",
  });

  await new Promise((resolve, reject) => {
    reader.on("data", (chunk) => {
      writer.write(Buffer.from(golombRiceDecoding(chunk, 3), "base64"));
    });
    reader.on("end", () => {
      writer.end();
      resolve();
    });
  });

Wonder if there's a way to make it read the data using streams, but without cutting a binary number? I don't mind if it reads x at a time, the issue is when it cuts a binary number, invalidating the data when decoding. Thank you

about 4 years ago · Juan Pablo Isaza
1 answers
Answer question

0

Your particular compression scheme that you're trying to decode looks like it's a variable byte scheme. So, if a chunk boundary doesn't perfectly line up with the size of a variable byte piece of compression, your decompression library will fail.

To decode something like this on the fly, you have to build logic right into the decoder that recognizes when it only has a partial piece of data and then buffers that to combine it with the next chunk of data that arrives. Because the compression is variable byte, you can't known where proper boundaries are without being part of the decompression logic.

Or, you can give up doing on-the-fly decoding and buffer all the compressed data into memory and then decompress it all at once when you have all the data (then you don't have any chunk boundaries to worry about).

about 4 years ago · Juan Pablo Isaza Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!