Backgound:

I'm currently hosting an ASP.NET application in Azure with the following specs:

• ASP .Net Core 2.2
• Using Flurl for HTTP requests
• Kestrel Webserver
• Docker (Linux - mcr.microsoft.com/dotnet/core/aspnet:2.2 runtime)
• Azure App Service on P2V2 tier app service plan

I have a a couple of background jobs that run on the service that makes a lot of outbound HTTP calls to a 3rd party service.

Issue:

Under a small load (approximately 1 call per 10 seconds), all requests are completed in under a second with no issue. The issue I'm having is that under a heavy load, when service can make up to 3/4 calls in a 10 second span, some of the requests will randomly timeout and throw an exception. When I was using RestSharp the exception would read "The operation has timed out". Now that I'm using Flurl, the exception reads "The call timed out".

Here's the kicker - If I run the same job from my laptop running Windows 10 / Visual Studios 2017, this problem does NOT occur. This leads me to believe I'm hitting some limit or running out of some resource in my hosted environment. Unclear if that is connection/socket or thread related.

Things I've tried:

• Ensure all code paths to the request are using async/await to prevent lockouts
• Ensure Kestrel Defaults allow unlimited connections (it does by default)
• Ensure Dockers default connection limits are sufficient (2000 by default, more than enough)
• Configuring ServicePointManager settings for connection limits

Here is the code in my startup.cs that I'm currently using to try and prevent this issue:

public class Startup
{
    public Startup(IHostingEnvironment hostingEnvironment)
    {
        ...

        // ServicePointManager setup
        ServicePointManager.UseNagleAlgorithm = false;
        ServicePointManager.Expect100Continue = false;
        ServicePointManager.DefaultConnectionLimit = int.MaxValue;
        ServicePointManager.EnableDnsRoundRobin = true;
        ServicePointManager.ReusePort = true;

        // Set Service point timeouts
        var sp = ServicePointManager.FindServicePoint(new Uri("https://placeholder.thirdparty.com"));
        sp.ConnectionLeaseTimeout = 15 * 1000; // 15 seconds  

        FlurlHttp.ConfigureClient("https://placeholder.thirdparty.com", cli => cli.Settings.ConnectionLeaseTimeout = new TimeSpan(0, 0, 15));
    }
}

Has anyone else run into a similar issue to this? I'm open to any suggestions on how to best debug this situation, or possible methods to correct the issue. I'm at a complete loss after researching this for several days.

Thank you in advance.