Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

336
Views
Maximum number of concurrent tasks in 1 DPU in AWS Glue

A standard DPU in AWS Glue comes with 4 vCPU and 2 executors. I am confused about the maximum number of concurrent tasks that can be run in parallel with this configuration. Is it 4 or 8 on a single DPU with 4vcpu and 2 executors?

over 4 years ago · Santiago Trujillo
1 answers
Answer question

0

I had a similar discussion with the AWS Glue support team about this, I'll share with you what they told me about Glue Configuration. Take in example the Standard and the G1.X configuration.

Standard DPU Configuration:

  • 1 DPU reserved for MasterNode
  • 1 executor reserved for Driver/ApplicationMaster
  • Each DPU is configured with 2 executors
  • Each executor is configured with 5.5 GB memory
  • Each executor is configured with 4 cores

G.1X WorkerType Configuration:

  • 1 DPU added for MasterNode
  • 1 DPU reserved for Driver/ApplicationMaster
  • Each worker is configured with 1 executor
  • Each executor is configured with 10 GB memory
  • Each executor is configured with 8 cores

If we have for example a Job with Standard Configuration with 21 DPU means that we have:

  • 1 DPU reserved for Master
  • 20 DPU x 2 = 40 executors
  • 40 executors - 1 Driver/AM = 39 executors

Which we then end up with a total amount of 156 cores. Meaning, your job has 156 slots for execution. If for example you read files from S3 that means that you will be able to accept 156 input files in parallel.

Hope it helps.

over 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!