Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

532
Vistas
write a spark dataframe or write a glue dynamic frame, which option is better in AWS Glue?

In AWS Glue, I read the data from data catalog in a glue dynamic frame. Then convert the dynamic frame to spark dataframe to apply schema transformations. To write the data back to s3 I have seen developers convert the dataframe back to dynamicframe. Is there any advantage over writing a glue dynamic frame to writing a spark dataframe?

over 4 years ago · Santiago Trujillo
1 Respuestas
Responde la pregunta

0

You will find that there is functionality that is available only to dynamic frame writer class that cannot be accessed when using data frames:

  1. Writing to a catalog table based on an s3 source as well when you want to utilize connection to JDBC sources. i.e using from_jdbc_conf
  2. Writing to parquet using format glueparquet as a format.
  3. Tracking processed files in the target location using bookmarks

These are some of the use-cases I can think of, but if you have a use case that requires using save modes, for example, mode('overwrite') you could use data frames. A similar approach however exists at dynamic frame but is implemented slightly different. You can take a look at [purge_s3_path][3] then write.

over 4 years ago · Santiago Trujillo Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda