My server code (running on an AWS Lambda .net Core 3.1) suddenly stopped authenticating with my API server. It has been working for years, but today it stopped working in both production and development environments simultaneously. The Errors I'm getting are
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
The relevant source code is
WebRequest httpWebRequest2 = WebRequest.Create(APIurl);
httpWebRequest2.Method = "GET";
httpWebRequest2.Accept = "application/json"
httpWebRequest2.Timeout = 60 * 1000
httpWebRequest2.Headers.Add("Authorization", _APIToken);
WebResponse response = httpWebRequest2.GetResponseAsync().Result;
I'm using letsencrypt to generate certificates, and neither of the certificates -- for my development api url or production api url -- are expired. HTTPS queries succeed in every other platform I've tried.
At this point I don't even know what else to troubleshoot. I can give as much more information as is required.
This is an issue with Amazon Linux 2 that Lambda functions utilize to run their runtime environments. As of this posting Amazon Linux 2 utilizes OpenSSL 1.0 (specifically package openssl-1.0.2k-19.amzn2.0.6.x86_64). As of today, September 30th 2021, Let's Encrypt's older certificate from DST Root CA X3 expired has expired and a quirk of this is that the default preferred certificate chain from Let's Encrypt no longer works with OpenSSL 1.0.
You can work around this issue as noted in this post depending on how you generated your Let's Encrypt certificates. For example we utilize CertBot and needed to pass the --preferred-chain 'ISRG Root X1' argument to generate a compatible certificate for OpenSSL 1.0.
Hopefully the underlying runtime environment will be compatible with the default chain preference soon.
I've also found a temporary workaround to get Lambda functions up and running. I've only managed to get it working on the netcoreapp3.1 runtime - not the 2.1 one.
Essentially, I went and downloaded the "correct" intermediate R3 certificate from Let's Encrypt
I added it to the project-folders of my functions, set the file to be "copied to output directory":
<ItemGroup>
<None Update="lets-encrypt-r3.pem">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>
After this, I've added the following environment variable to my function:
"SSL_CERT_FILE" = "/var/task/lets-encrypt-r3.pem"
Making this change, the SSL errors went away. From my understanding, the SSL_CERT_FILE environment variable will instruct OpenSSL to use the given certificate for validation - although it might only use the given certificate, so test it on your own functions before pushing it to production :-) That being said, one of my functions connect to MailGun, which worked