Empresas
Empregos
  • Sobre nós
  • Soluções
    • Publicação de vagas
      Publique sua vaga e receba candidatos qualificados em 48h.
    • Avaliações de candidatos
      Mais de 500 testes técnicos e psicológicos, mais anti-fraude.
    • Headhunting
      Busca executiva personalizada do início ao fim.
    • Folha de Pagamento + EOR
      Dispersão de folha e EOR em mais de 15 países da LATAM.
  • Preços
  • Empregos

0

136
Visualizações
How to speed up an endpoint that's passing a significant amount of data to db

Essentially I have a React web app that I'm creating to upload a large set of data to a db. As of now I'm using a test file that's only around 3000 lines long and the total time to upload all the data from the file is around 4 minutes. I need to speed this up significantly but I'm not totally sure how. For reference the actual size of files I plan to upload will be around 100k lines long. Also another issue that I'm pretty sure I can fix with a little tinkering is I start to run out of net resources around 80k request so if there are any suggestions to modifying my delayQue function to avoid that then it would be appreciated as well.

Test file size: 76kb

Real file size: 5 mb

Endpoint

  const stepData = async() =>{

    const body = {a,b,c,d,e,f}
    assignToBody(body)

    try{

        const response = await fetch('http://localhost:5000/stepdata',{
                method:"POST",
                headers: {"Content-Type": "application/json"},
                body: JSON.stringify(body)
    })}catch(err){
        console.log(err.message)
    }
    
}

Throttle (note: assignVals just assigns the values to the body in my endpoint)

    const delayQue = () =>{

        assignVals(row)
        row++
        
        if(row % 500 == 0){
            setTimeout(delayQue,2000)
        }else{
            setTimeout(delayQue,100)
        }
    
  }

also another thing incase its important each line contains anywhere from 6 to 20 points of data. I lowered my number of calls by assigning the values to an object and passing the object. Any help on speeding this up will be greatly appreciated.

Edit: To clarify the issue is that I am simply having to make 100k request or more and am trying to find a way to speed up my request or at least manage them better instead of using settimeout and embedded try and catches.

Update: I'm fairly certain the issue is a resouce leak in my endpoint

about 4 years ago · Juan Pablo Isaza
1 Respostas
Responde à pergunta

0

Regarding the amount of data transferred over the network:

File size can be reduced very efficiently using a compression algorithm like gzip or brotli. Depending on the webserver you are using you will have to activate compression first for it to be used.

Here a example how to use with express: https://expressjs.com/en/resources/middleware/compression.html

Depending on the data you want to transfer, you can reduce the file size significantly. Repetitive text files compress best. Already compressed files like jpeg images don't compress at all.

about 4 years ago · Juan Pablo Isaza Relatório
Responde à pergunta
Encontrar trabalhos remotos

Descubra a nova forma de encontrar um emprego!

melhores empregos
Principais categorias de trabalho
Empresas
Postar vaga Preços Comercial
Jurídico
Termos e Condições Política de privacidade
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomende algumas ofertas para mim
Preciso de ajuda