Question

0

160

Views

Is it possible to read/decode .avro uInt8Array Container File with Javascript in Browser?

I'm trying to decode an .avro file loaded from a web server.

Since the string version of the uInt8Array starts with
"buffer from S3 Objavro.schema�{"type":"record","name":"Destination",..."
I assume it's avro Container File

I found 'avro.js' and 'avsc' as tools for working with the .avro format and javascript but reading the documentation it sound's like the decoding of a Container File is only possible in Node.js, not in the browser. (The FileDecoder/Encoder methods are taking a path to a file as string, not an uInt8Array)

Do I get this wrong or is there an alternative way to decode an .avro Container File in the browser with javascript?

about 4 years ago · Juan Pablo Isaza

2 answers

Answer question

0

Thanks for posting!

Below is my integration of avro/binary with axios, in case it helps anyone else trying to implement browser side decoding:

[before browserifying]

const axios = require('axios')
const avro = require('avsc')

const config = {
  responseType: 'blob'
};

const url = 'https://some-url.com'

axios.get(url, config)
.then(res => {
  avro.createBlobDecoder(res.data)
    .on('metadata', (type) => console.log(type))
    .on('data', (record) => console.log(record))
})
.catch(e => console.error(e))

about 4 years ago · Juan Pablo Isaza Report

0

Answer question

Find remote jobs

Accepted Answer · 2022-07-12T03:34:36.306Z

Luckily I found a way using avsc with broserify

avro.createBlobDecoder(blob, [{options}])

[avro.js - before browserifying]

var avro = require('avsc');

const AVRO = {

decodeBlob(blob) {

        let schema, columnTitles, columnTypes, records = []

        return new Promise((resolve) => {
            avro.createBlobDecoder(blob, {
                // noDecode: true
            })
                .on('metadata', (s) => {
                    schema = s
                    columnTitles = schema.fields.map(f => f.name)
                    columnTypes = schema.fields.map(f => f.type)
                })
                .on('data', (data) => {
                    records.push(data)
                })
                .on('finish', () => {
                    resolve(
                        {
                            columnTitles: columnTitles,
                            columnTypes: columnTypes,
                            records: records
                        }
                    )
                })
        })
    }
}

module.exports = AVRO

[package.json]

"scripts": {
    "avro": "browserify public/avro.js --s AVRO > public/build/avro.js"
  }

[someOtherFile.js]

//s3.getObject => uInt8array

const blob = new Blob([uInt8array])  //arr in brackets !important

const avroDataObj = await AVRO.decodeBlob(blob)

0

160