Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

315
Views
how does Spark limit the usage of cpu cores and memory?

How does Spark limit the usage of cpu cores and memory?Does it use cgroups? How about Yarn?

about 4 years ago · Santiago Trujillo
2 answers
Answer question

0

In standalone cluster Spark only manages application predefined resource configs with provided resource pool. Resource pool combined based on executors which added as salves to cluster. Yarn uses containers and resource limitation applies config of container which defines minimum and maximum core and memory allocation.

about 4 years ago · Santiago Trujillo Report

0

In YARN NodeManager is monitoring spark executors' memory usage and killing them if they use above spark.executor.memory

In case of CPU, spark.executor.cores is the amount of concurrent tasks executor can run. More information on Spark Configuration Documentation

You can enable cgroups in yarn and limit CPU usage or YARN containers (spark executors).

about 4 years ago · Santiago Trujillo Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!