• Jobs
  • About Us
  • professionals
    • Home
    • Jobs
    • Courses and challenges
    • Questions
    • Teachers
  • business
    • Home
    • Post vacancy
    • Our process
    • Pricing
    • Assessments
    • Payroll
    • Blog
    • Sales
    • Salary Calculator

0

110
Views
Parsing big JSON file using Typescript

I want to parse/process a 25 MB JSON file using Typescript and filter out/sort the objects .. The code I wrote is taking minutes (and sometimes timeouts) not sure why is this happening or if there's another way to make the code more efficient.

Note: the code worked on a small file

import fs from 'fs';
searchAccounts(): Promise<Account[]> {
       const accountSearchCriteria: AccountSearchCriteria = {
                country: 'NZ',
                mfa: 'SMS',
                name: 'TEST',
                sortField: 'dob'
        };
        const jsonPath = './src/file.json';
        const rawAccounts = fs.readFileSync(jsonPath, 'utf-8');
        let accounts: Account[] = JSON.parse(rawAccounts);
        if (accountSearchCriteria) {
            if (accountSearchCriteria.name) {
                accounts = accounts.filter(
                    account =>
                        account.firstName.toLowerCase() ===
                            accountSearchCriteria.name.toLowerCase() ||
                        account.lastName.toLowerCase() ===
                            accountSearchCriteria.name.toLowerCase()
                );
            }
            if (accountSearchCriteria.country) {
                accounts = accounts.filter(
                    account =>
                        account.country.toLowerCase() ===
                        accountSearchCriteria.country.toLowerCase()
                );
            }
            if (accountSearchCriteria.mfa) {
                accounts = accounts.filter(
                    account => account.mfa === accountSearchCriteria.mfa
                );
            }
            if (accountSearchCriteria.sortField) {
                accounts.sort((a, b) => {
                    return a[accountSearchCriteria.sortField] <
                        b[accountSearchCriteria.sortField]
                        ? -1
                        : 1;
                });
            }
            return accounts;
        }
        return accounts;
}
almost 3 years ago · Juan Pablo Isaza
2 answers
Answer question

0

Since your data size is 25 MB, you should use a more memory-efficient sorting algorithm.

You can try to use cycle sort.

cycle-sort you can find an implementation here and use it in your code to see if there is a difference.

almost 3 years ago · Juan Pablo Isaza Report

0

Node.js is single-threaded if your code blocking the thread for a long time it will give you a timeout error. There are two problems with your code.

  1. you are using fs.readFileSync(jsonPath, 'utf-8');, it is an asynchronous function and blocks the thread while reading the file. Use instead fs.readFile('./Index.html', callback):
const fs = require('fs');
fs.readFile('./Index.html', function read(err, data) {
   if (err) {
       throw err;
   }
   console.log(data);
});
  1. Sorting data is also a thread-blocking task try a different sorting technique, which doesn't occupy thread for a long time.

Note: Node.js is not good with CPU-centric tasks i.e sorting, image processing, etc. It's good with I/O tasks.

almost 3 years ago · Juan Pablo Isaza Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Our process Sales
Legal
Terms and conditions Privacy policy
© 2025 PeakU Inc. All Rights Reserved.

Andres GPT

Recommend me some offers
I have an error