Soy relativamente nuevo en Python y trato de implementar un módulo de multiprocesamiento para mi bucle for.
Tengo una matriz de URL de imágenes almacenadas en img_urls que necesito descargar y aplicar algo de visión de Google.
if __name__ == '__main__': img_urls = [ALL_MY_Image_URLS] runAll(img_urls) print("--- %s seconds ---" % (time.time() - start_time))Este es mi método runAll()
def runAll(img_urls): num_cores = multiprocessing.cpu_count() print("Image URLS {}",len(img_urls)) if len(img_urls) > 2: numberOfImages = 0 else: numberOfImages = 1 start_timeProcess = time.time() pool = multiprocessing.Pool() pool.map(annotate,img_urls) end_timeProcess = time.time() print('\n Time to complete ', end_timeProcess-start_timeProcess) print(full_matching_pages) def annotate(img_path): file = requests.get(img_path).content print("file is",file) """Returns web annotations given the path to an image.""" print('Process Working under ',os.getpid()) image = types.Image(content=file) web_detection = vision_client.web_detection(image=image).web_detection report(web_detection)Obtengo esto como advertencia cuando lo ejecuto y python falla
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.Este error ocurre debido a la seguridad adicional para restringir subprocesos múltiples en macOS High Sierra y versiones posteriores de macOS. Sé que esta respuesta es un poco tarde, pero resolví el problema usando el siguiente método:
Establezca una variable de entorno .bash_profile (o .zshrc para macOS reciente) para permitir secuencias de comandos o aplicaciones multiproceso bajo las nuevas reglas de seguridad de macOS High Sierra.
Abre una terminal:
$ nano .bash_profileAgregue la siguiente línea al final del archivo:
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESGuarde, salga, cierre la terminal y vuelva a abrir la terminal. Verifique que la variable de entorno ahora esté configurada:
$ envVerá un resultado similar a:
TERM_PROGRAM=Apple_Terminal SHELL=/bin/bash TERM=xterm-256color TMPDIR=/var/folders/pn/vasdlj3ojO#OOas4dasdffJq/T/ Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.E7qLFJDSo/Render TERM_PROGRAM_VERSION=404 TERM_SESSION_ID=NONE OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESAhora debería poder ejecutar su secuencia de comandos de Python con subprocesos múltiples.
Ejecutando MAC y z-shell y en mi archivo .zshrc tuve que agregar:
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YESy luego en la línea de comando:
source ~/.zshrcentonces funciono
las otras respuestas le dicen que configure OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES , ¡pero no haga esto! solo estás poniendo cinta adhesiva en la luz de advertencia. Es posible que necesite esto caso por caso para algún software heredado, ¡pero ciertamente no configure esto en su .bash_profile !
esto está arreglado en https://bugs.python.org/issue33725 (python3.8+) pero es una buena práctica usar
with multiprocessing.get_context("spawn").Pool() as pool: pool.map(annotate,img_urls)