Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

453
Vistas
What is the difference between AWS Glue ETL Job and AWS EMR?

If I had to perform ETL on a huge dataset(say 1Tb) stored in S3 as csv files, Both AWS Glue ETL job and AWS EMR steps can be used. Then how is AWS Glue different from AWS EMR. And which is the better solution in this case.

over 4 years ago · Santiago Trujillo
3 Respuestas
Responde la pregunta

0

Glue allows you to submit ETL scripts directly in PySpark/Python/Scala, without the need for managing an EMR cluster. All setup/tear-down of infrastructure is managed.

There are also a few other managed components like Crawlers, Glue Data Catalog, etc which make it easier to work on your data.

You could use either for your use-case, Glue would be faster however you may not have the flexibility you get with EMR.

over 4 years ago · Santiago Trujillo Denunciar

0

Most of the differences are already listed so I'll focus more on the use case specific.

When to choose aws glue

  1. Data size is huge but structured i.e. it is in the table structure and is of known format (CSV, parquet, orc, json).
  2. Lineage is required, if you need the data lineage graph while developing your etl job prefer developing the etl using glue native libraries.
  3. The developers don't need to tweak the performance parameters like setting number of executors, per executor memory and so on.
  4. You don't want the overhead of managing large cluster and pay only for what you use.

When to use EMR

  1. Data is huge but semi-structured or unstructured where you can't take any benefit from Glue catalog.
  2. You believe only in the outputs and lineage is not required.
  3. You need to define more memory per executor depending upon the type of your job and requirement.
  4. You can manage the cluster easily or if you have so many jobs which can run concurrently on the cluster saving you money.
  5. In case of structured data, you should use EMR when you want more Hadoop capabilities like hive, presto for further analytics.

So it depends on what your use case is. Both are great service.

over 4 years ago · Santiago Trujillo Denunciar

0

Glue uses EMR under the hood. This is evident when you ssh into the driver of your Glue dev-endpoint.

Now since Glue is a managed spark environment or say managed EMR environment, it comes with reduced flexibility. The type of workers that you can chose is limited. The number of language libraries that you can use in your spark code is limited. Glue did not support packages like pandas, numpy until recently. Apps like presto cant be integrated with Glue although Athena is a good alternative to a separate presto installation.

The main issue however is that Glue jobs have a cold start time from anywhere between 1 minute to 15 minutes.

EMR is a good choice for exploratory data analysis but for a production environment with CI/CD, Glue seems to be the better choice.

EDIT - Glue jobs no longer have a cold start wait time

over 4 years ago · Santiago Trujillo Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda