Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

205
Views
How come multiprocessing.managers.BaseManager is this slow on linux?

I'm trying to diagnose a performance problem with the server created by multiprocessing.managers.BaseManager. The problem boils down to this:

server.py:

import multiprocessing
import multiprocessing.managers

auth_key = b'12345678901234567890'
port = 32001

shared_values = {}

def _shared_values():
    global shared_values
    return shared_values

class SharedStateManager(multiprocessing.managers.BaseManager): pass

SharedStateManager.register('SharedValues', callable=_shared_values)

multiprocessing.current_process().authkey = auth_key
server = SharedStateManager(address=('', port), authkey=auth_key)

server.get_server().serve_forever()

client.py:

import time
import multiprocessing
import multiprocessing.managers

auth_key = b'12345678901234567890'
port = 32001

class SharedStateManager(multiprocessing.managers.BaseManager): pass

SharedStateManager.register('SharedValues')

multiprocessing.current_process().authkey = auth_key
ssm = SharedStateManager(address=('', port), authkey=auth_key)

cl1 = time.time_ns()
ssm.connect()
cl2 = time.time_ns()
shared_values = ssm.SharedValues()
cl3 = time.time_ns()
print('2-1: {} ms'.format((cl2 - cl1) / (10 ** 6)))
print('3-2: {} ms'.format((cl3 - cl2) / (10 ** 6)))

When I run the client code on a Debian buster instance running on Intel 8th-gen 2.3GHz i5 with 4 cores, I get:

2-1: 42.124515 ms
3-2: 132.311978 ms

Which is very slow. The python version is 3.7.3 although I tried different version with docker containers and got similar results.

But if I run the same code on a Macbook (macos 11.6) that has a similar CPU, I get:

2-1: 1.950359 ms
3-2: 3.264376 ms

What might be the cause of such dramatic response time difference?

Edit: The target machine isn't broken or anything. It's running a couple of QEMU virtual machines plus a few docker containers. So far the network latency did not happen on for instance, a sanic based HTTP server or other containerized socket servers. Copying a file over a network share will max out the LAN at about 100MB/s. So it's happening to Python's multiprocessing server in particular.

Edit: I also found this question which sort of exhibited the same behavior regarding a Python multiprocessing program running much faster on mac than linux.

Edit: Solved! I thought I was running Debian bullseye but it turned out I was running buster. After exhausting all other options I gave release upgrade a try. And the latency went away after the upgrade.

over 4 years ago · Santiago Trujillo
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!