In our app we have a Rails app that makes an API call to a Flask API. The API then schedules a task, and we are using Redis as the queue.
@app.route("/api/article", methods=["POST"])
def api_article():
app.logger.info("[route] /api/article")
.
.
res_id = celery.send_task("tasks.text_stats", args=[article], kwargs={}).id
@celery.task(name="tasks.text_stats")
def text_stats(article):
logger.info("text_stats")
What I am thinking is if the Redis database is accessible to both Rails and Flask, we could just create the task directly in Redis, which would then be picked up by the Celery worker.
Questions:
To solve this problem we can simply inspect in Redis how Celery pushes tasks to the LIST.
This is an example celery task that's been pushed to Redis by Celery:
{'body': 'W1siYjVmNDhmNDQtOWZiZS00MjdiLWE3NDMtZjc3MDgwMDQzN2ZiIl0sIHt9LCB7ImNhbGxiYWNrcyI6IG51bGwsICJlcnJiYWNrcyI6IG51bGwsICJjaGFpbiI6IG51bGwsICJjaG9yZCI6IG51bGx9XQ==', 'content-encoding': 'utf-8', 'content-type': 'application/json', 'headers': {'lang': 'py', 'task': 'my_python_function', 'id': '64148470-0dde-4e31-ab63-09c98955d24f', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '64148470-0dde-4e31-ab63-09c98955d24f', 'parent_id': None, 'argsrepr': "('my_string_argument_one', 'my_string_argument_2',)", 'kwargsrepr': '{}', 'origin': 'gen73965@mac.local'}, 'properties': {'correlation_id': '64148470-0dde-4e31-ab63-09c98955d24f', 'reply_to': '1a223e51-77c3-37d3-a870-baae15b3b741', 'delivery_mode': 2, 'delivery_info': {'exchange': '', 'routing_key': 'celery'}, 'priority': 0, 'body_encoding': 'base64', 'delivery_tag': 'f840868a-18c9-4b1f-bc95-5ff95666ca21'}}
The more important keys in the above is headers.task, which is the python function name you need to call, as well as headers.argsrepr and headers.kwargsrepr, which is a stringified tuple/dict representing the args/kwargs you need to pass to the function.
There are other keys in the above dict like routing_key (the celery queue), a couple of ids (you can autogenerate these), eta, that you could use to customize the behavior.
Once we understand how celery interacts with Redis, similar to this example, you can simply run LPUSH with Ruby's Redis client with the stringified python dict.
The more troubling part, really, is that celery expects the stringified LIST item to be in dict format, which has some differences with Ruby's to_json. You'll need to manually do that part (such as converting booleans to python bool)