I am seeing two issues here.

1. Why the application is running fine when the healthcheck fails. here is the explanation from AWS Docs:

If a target group contains only unhealthy registered targets, the load balancer nodes route requests across its unhealthy targets. Health checks for your target groups

2. How could you fix the health check while the instances are draining because of failed healthchecks.

404 means that the health check URL is not found. Confirm the health check configuration. your health check URL should respond HTTP 200 OK response. If your instances are draining repeatedly, you can temporarily set the health check rule to match HTTP 404 until your instances becomes healthy. Once you figure out the correct health check URL, you can set that.

Hope this helps.

In my case IIS server and resolved with the below steps.

1. Check the security groups - whether we have opened the required ports from ALB SG to EC2 SG.

2. Login to server and check does IIS server's default site has 443 port opened if your health-check is on 443. (whatever port you are using for health checks).

Use the curl command to troubleshoot the issue.

1. If you would like to check on HTTPS use the below command to check the response. Use -k or --insecure to ignore the SSL issue.

   curl https://[serverIP] -k

2. For HTTP test use the below command.

   curl http://[serverIP]

I was dealing with this issue for 1 day. I finally realized that I had removed the default server configuration from nginx. this is needed to for de default path that the healthCheck checks