¿Cómo puedo serializar una aplicación Flask o solicitar contexto, o un subconjunto de ese contexto (es decir, lo que sea que pueda serializarse con éxito) para poder acceder a ese contexto desde otro proceso, en lugar de un hilo?
Tengo algunas funciones que requieren acceso al contexto de solicitud de Flask, o al contexto de la aplicación, que quiero ejecutar en segundo plano.
Flask tiene un decorador @copy_current_request_context para envolver una función en una copia del contexto de la solicitud, para que pueda ejecutarla en un hilo diferente:
from threading import Thread from flask import Flask, request, copy_current_request_context app = Flask(__name__) @app.route('/') def index(): request.foo = 'bar' @copy_current_request_context def baz(): print(request.foo) thr = Thread(target=baz) thr.start() return 'ok'Si bien Flask no proporciona un decorador incorporado para copiar el contexto de la aplicación, proporciona la maquinaria para hacerlo; se describe una solución en Access Flask.g dentro de Greenlet
En primer lugar, esperaba resolver con Pickle. Esto es lo que usa concurrent.futures.ProcessPoolExecutor . Desafortunadamente, Pickle falla debido a la presencia de objetos de bloqueo de subprocesos dentro del contexto de la aplicación:
>>> with app.test_request_context('/'): ... pickle.dumps(appctx) ... Traceback (most recent call last): File "<stdin>", line 2, in <module> TypeError: can't pickle _thread.lock objectsTampoco puede copiar funciones decoradas/envueltas:
>>> with app.test_request_context('/'): ... bar = copy_current_request_context(pow) ... pickle.dumps(bar) ... Traceback (most recent call last): File "<stdin>", line 3, in <module> _pickle.PicklingError: Can't pickle <function pow at 0x110bee8c8>: it's not the same object as builtins.pow Luego probé dill . Pasa las dos pruebas anteriores, pero no puede serializar muchas otras cosas que podrían terminar en un contexto. Los contextos de las aplicaciones son especialmente propensos a que las extensiones se vinculen a sí mismas: SQLAlchemy es un gran ejemplo. Esto es lo que sucede cuando usa SQLAlchemy e intenta serializar el contexto de su aplicación:
>>> from flask_sqlalchemy import SQLAlchemy >>> db = SQLAlchemy(app) >>> with app.test_request_context('/'): ... from flask.globals import _app_ctx_stack ... appctx = _app_ctx_stack.top ... dill.dumps(appctx) ... Traceback (most recent call last): File "<stdin>", line 4, in <module> File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps dump(obj, file, protocol, byref, fmode, recurse)#, strictio) File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump pik.dump(obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump self.save(obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce save(state) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems save(v) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce save(state) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems save(v) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 816, in save_list self._batch_appends(obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 843, in _batch_appends save(tmp[0]) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1386, in save_function obj.__dict__), obj=obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce save(args) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1129, in save_cell pickler.save_reduce(_create_cell, (f,), obj=obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce save(args) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple save(element) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce save(state) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems save(v) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce save(state) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems save(v) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save self.save_reduce(obj=obj, *rv) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce save(state) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict StockPickler.save_dict(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict self._batch_setitems(obj.items()) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems save(v) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save f(self, obj) # Call unbound method with explicit self File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type StockPickler.save_global(pickler, obj) File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global (obj, module_name, name)) from None _pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.SignallingSession'>: it's not found as sqlalchemy.orm.session.SignallingSession Entonces: ¿hay alguna forma de evitar esto? Un pensamiento que tuve fue que bifurcar un nuevo proceso "justo a tiempo", es decir, inmediatamente antes de que quiera ejecutar mi tarea en segundo plano, puede significar que ni siquiera necesito copiar un contexto, ya debería tenerlo siempre que la bifurcación ocurre después de que se configura el contexto. Sin embargo, no sé cómo hacer que esto funcione con un ProcessPoolExecutor , donde los procesos son un grupo de larga duración. La única idea que tengo allí es forzar de alguna manera que los procesos de trabajo se cierren al final de cada tarea, pero estoy bastante seguro de que esto solo resultará en un grupo roto.
No estoy seguro de si esto es adecuado para usted, pero existe la posibilidad de usar el contexto de la aplicación Flask de otros procesos. Requiere que utilice el patrón de fábrica de aplicaciones .
# your_project/app.py def create_app(config_file): # Load the config file somehow # Create your app instance app = Flask(__name__) # Initialize your extensions db.init_app(app) # Mount routes, etc app.register_blueprint(...) # Return your app return app # your_project/wsgi.py from .app import create_app app = create_app("./config/local.cfg") Usted define una fábrica de aplicaciones y la usa para crear su aplicación como un punto de entrada en wsgi.py u otra ubicación conveniente, y luego puede acceder a esta instancia de la aplicación desde su secuencia de comandos en segundo plano que se ejecutaría en otro proceso.
# your_project/background_script.py from .wsgi import app with app.app_context(): # Do some stuffPor cierto, esta también es una forma de usar Flask con Celery, por ejemplo, como se describe en esta pregunta . Espero que esto te ayude.