Estoy usando un mapa de grupo de procesamiento múltiple en python 3.7 y recibo un error como el siguiente con el contenedor docker
multiprocessing.pool.MaybeEncoding Error: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'Estoy usando un código como el siguiente,
def foo(track): track_url = track[1] url = requests.get(track_url) hashes = fingerprint.fingerprint_worker( BytesIO(url.content) ) return True def worker: tracks = [[1, "track_url_1"], [2, "track_url_2"]] with multiprocessing.Pool(const.MULTIPROCESSING_POOL_SIZE) as pool: pool.map(foo, tracks) pool.close() pool.terminate() pool.join()MULTIPROCESAMIENTO_GRUPO_TAMAÑO = 1
El seguimiento completo de la pila de este error es,
Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1836, in __call__ return self.wsgi_app(environ, start_response) File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in wsgi_app response = self.make_response(self.handle_exception(e)) File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1403, in handle_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise raise value File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1817, in wsgi_app response = self.full_dispatch_request() File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1477, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1381, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 33, in reraise raise value File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/radio-streaming-fingerprint/app.py", line 37, in fingerprint_covers_tracks resp = fp.fingerprint_tracks(track, generate_fingerprints_with_pool) File "/radio-streaming-fingerprint/fingerprint.py", line 37, in fingerprint_tracks pool.map(generate_fingerprints_with_pool, tracks) File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/usr/local/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f8ff58e8110>'. Reason: 'PicklingError("Can't pickle <class 'MemoryError'>: it's not the same object as builtins.MemoryError")'No entiendo por qué recibo este error en mi entorno de producción, no en la configuración de mi proyecto local. Hay 4 GB de RAM en mi contenedor docker.
Por favor sugiérame una solución.
Gracias por adelantado.