Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

131
Vistas
How to speed up an endpoint that's passing a significant amount of data to db

Essentially I have a React web app that I'm creating to upload a large set of data to a db. As of now I'm using a test file that's only around 3000 lines long and the total time to upload all the data from the file is around 4 minutes. I need to speed this up significantly but I'm not totally sure how. For reference the actual size of files I plan to upload will be around 100k lines long. Also another issue that I'm pretty sure I can fix with a little tinkering is I start to run out of net resources around 80k request so if there are any suggestions to modifying my delayQue function to avoid that then it would be appreciated as well.

Test file size: 76kb

Real file size: 5 mb

Endpoint

  const stepData = async() =>{

    const body = {a,b,c,d,e,f}
    assignToBody(body)

    try{

        const response = await fetch('http://localhost:5000/stepdata',{
                method:"POST",
                headers: {"Content-Type": "application/json"},
                body: JSON.stringify(body)
    })}catch(err){
        console.log(err.message)
    }
    
}

Throttle (note: assignVals just assigns the values to the body in my endpoint)

    const delayQue = () =>{

        assignVals(row)
        row++
        
        if(row % 500 == 0){
            setTimeout(delayQue,2000)
        }else{
            setTimeout(delayQue,100)
        }
    
  }

also another thing incase its important each line contains anywhere from 6 to 20 points of data. I lowered my number of calls by assigning the values to an object and passing the object. Any help on speeding this up will be greatly appreciated.

Edit: To clarify the issue is that I am simply having to make 100k request or more and am trying to find a way to speed up my request or at least manage them better instead of using settimeout and embedded try and catches.

Update: I'm fairly certain the issue is a resouce leak in my endpoint

about 4 years ago · Juan Pablo Isaza
1 Respuestas
Responde la pregunta

0

Regarding the amount of data transferred over the network:

File size can be reduced very efficiently using a compression algorithm like gzip or brotli. Depending on the webserver you are using you will have to activate compression first for it to be used.

Here a example how to use with express: https://expressjs.com/en/resources/middleware/compression.html

Depending on the data you want to transfer, you can reduce the file size significantly. Repetitive text files compress best. Already compressed files like jpeg images don't compress at all.

about 4 years ago · Juan Pablo Isaza Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda