The issue has appeared recently and the previously healthy container now enters a sleep loop when a shutit session is being created. The issue occurs only on Cloud Run and not locally.

Minimum reproducible code:

requirements.txt

Flask==2.0.1
gunicorn==20.1.0
shutit

Dockerfile

FROM python:3.9

# Allow statements and log messages to immediately appear in the Cloud Run logs
ENV PYTHONUNBUFFERED True

COPY requirements.txt ./
RUN pip install -r requirements.txt

# Copy local code to the container image.
ENV APP_HOME /myapp
WORKDIR $APP_HOME
COPY . ./

CMD exec gunicorn \
 --bind :$PORT \
 --worker-class "sync" \
 --workers 1 \
 --threads 1 \
 --timeout 0 \
 main:app

main.py

import os
import shutit
from flask import Flask, request

app = Flask(__name__)

# just to prove api works
@app.route('/ping', methods=['GET'])
def ping():
    os.system('echo pong')
    return 'OK'

# issue replication
@app.route('/healthcheck', methods=['GET'])
def healthcheck():
    os.system("echo 'healthcheck'")
    # hangs inside create_session
    shell = shutit.create_session(echo=True, loglevel='debug')
    # never shell.send reached 
    shell.send('echo Hello World', echo=True)
    # never returned
    return 'OK'

if __name__ == '__main__':
    app.run(host='127.0.0.1', port=8080, debug=True)

cloudbuild.yaml

steps:
  - id: "build_container"
    name: "gcr.io/kaniko-project/executor:latest"
    args:
      - --destination=gcr.io/$PROJECT_ID/borked-service-debug:latest
      - --cache=true
      - --cache-ttl=99h
  - id: "configure infrastructure"
    name: "gcr.io/cloud-builders/gcloud"
    entrypoint: "bash"
    args:
      - "-c"
      - |
        set -euxo pipefail

        REGION="europe-west1"
        CLOUD_RUN_SERVICE="borked-service-debug"

        SA_NAME="$${CLOUD_RUN_SERVICE}@${PROJECT_ID}.iam.gserviceaccount.com"

        gcloud beta run deploy $${CLOUD_RUN_SERVICE} \
          --service-account "$${SA_NAME}" \
          --image gcr.io/${PROJECT_ID}/$${CLOUD_RUN_SERVICE}:latest \
          --allow-unauthenticated \
          --platform managed \
          --concurrency 1 \
          --max-instances 10 \
          --timeout 1000s \
          --cpu 1 \
          --memory=1Gi \
          --region "$${REGION}"

cloud run logs that get looped:

Setting up prompt
In session: host_child, trying to send: export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
================================================================================
Sending>>> export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'<<<, expecting>>>['\r\nORIGIN_ENV:rkkfQQ2y# ']<<<
Sending in pexpect session (68242035994000): export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
Expecting: ['\r\nORIGIN_ENV:rkkfQQ2y# ']
export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
root@localhost:/myapp# export PS1_ORIGIN_ENV=$PS1 && PS1='OR''IGIN_ENV:rkkfQQ2y# ' && PROMPT_COMMAND='sleep .05||sleep 1'
Stopped sleep .05
Stopped sleep 1
pexpect: buffer: b'' before: b'cm9vdEBsb2NhbGhvc3Q6L3B1YnN1YiMgIGV4cx' after: b'DQpPUklHSU5fRU5WOnJra2ZRUTJ5IyA='
Resetting default expect to: ORIGIN_ENV:rkkfQQ2y# 
In session: host_child, trying to send: stty cols 65535
================================================================================
Sending>>> stty cols 65535<<<, expecting>>>ORIGIN_ENV:rkkfQQ2y# <<<
Sending in pexpect session (68242035994000): stty cols 65535
Expecting: ORIGIN_ENV:rkkfQQ2y# 
ORIGIN_ENV:rkkfQQ2y# stty cols 65535
stty cols 65535
Stopped stty cols 65535
Stopped sleep .05
Stopped sleep 1

Workarounds tried:

• Different regions: a few European(tier 1 and 2), Asia, US.
• Build with docker instead of kaniko
• Different CPU and Memory allocated to the container
• Minimum number of containers 1-5 (to ensure CPU is always allocated to the container)
• --no-cpu-throttling also made no difference
• Maximum number of containers 1-30
• Different GCP project
• Different Docker base images (3.5-3.9 + various shas ranging from a year ago to recent ones)