I am trying to create an Kafka client app (both producer and consumer) using AWS managed Kafka instance (MSK). Also broker to broker communication and client to broker communication is configured as secure via TLS in cluster. CA is AWS Private CA as that's the only one supported by MSK for client to broker authentication via TLS.

Problem context: AWS official doc (https://docs.aws.amazon.com/msk/latest/developerguide/msk-authentication.html#msk-authentication-client) steps are more inclined towards Java world and deals with client truststore and keystore as jks. However .Net client doesn't use Java JKS container format (https://github.com/mhowlett/confluent-kafka-dotnet/tree/security/examples/Security).

• Server authenticating client : This part I was able to solve. As jks is just a data store so after creating Keystore by following above aws doc I ran few extra keytool and openssl commands to extract client certificate and key explicitly. I was able to produce and consume message successfully using this.

• However, for client to validate server I need to set ssl.ca.location to CA root certificate. From Private CA(the one used as CA with MSK instance) I have downloaded the root CA which by default is pem format (Note: This contained only certificate details and no key details). Converted it to .crt using below commands:

  add pem to truststore : keytool -keystore kafka.client.truststore.jks -alias CARoot -importcert -file Certificate.pem

  get cert from truststore : keytool -export -alias CARoot -keystore kafka.client.truststore.jks -rfc -file ca-root.crt

using the above ca-root.crt for ca location doesn't seem to be working. It throws an error broker validation failed. Client Machine is Windows but that shouldn't be problem as apart from passing cert location ssl.ca.location I additionally tried installing certificate in Machine but still no luck. I see below error:

SSL handshake failed: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed

Am I missing something with client validating server part ?

Update: I was able to solve broker validation as well. Above official AWS documentation relies on Java cert for Broker validation authentication. I extended it a bit to derive perm and further .crt from jks and used the generated .crt as CA for ssl.ca.location and it works. However there are way too many certificates in the one which is generated and it's not clear which one is used by for validation of MSK broker certificate. Also why the CA certificate downloaded from private AWS CA doesn't work? It would be really helpful if someone can help on these. Also please suggest if there is a alternative standard solution for the problem.