I have an EMR cluster running and I'm trying to link the notebook to github.
I have:
Attempted to link the notebook to a private repo with my user/pass secret, but I get the following error:
Unable to reach repository https://github.com/<my repo>. Ensure network and security groups have valid configurations. Ensure that the repository information provided is correct.
I see in the docs it talks about NAT and VPG options, both of which I do not have. Is that needed? The docs are really light on details/configuration.
https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-git-considerations.html
Any help is appreciated.
You need to take care of 4 things:
the git repo along with the target branch should exist
configure the credentials to access Git using the Secrets Manager
your EMR cluster mandatorily has to be in private subnet and NOT public subnet. You mandatorily need to have a NAT G/W in your VPC that this private subnet should use to reach out to the internet i.e. configure your route table for this private subnet to map 0.0.0.0/0 to the NAT G/W.
You need to have 2 Security Groups (SG):
i. A SG for the master instance
inbound rule - Allow TCP Port 18888 from any resources in the default EC2 security group for EMR Notebooks
outbound rule - None
ii. A SG for the notebook instance
i. A SG for the master instance
inbound rule - None
outbound rule - Allow TCP Port 18888 to any resources in the default EC2 security group for EMR Notebooks. Also allow the notebook to route traffic to the internet via the cluster e.g. HTTPS TCP 443 0.0.0.0/0
Source: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-managed-notebooks-security-groups.html
end-to-end, as follows :
1.) create "private" subnets.
from scratch, I created a dedicated vpc first : say 10.0.16.0/20.
Next, I create an internet gateway. Attach it to the above vpc.
In this VPC, create a "public" subnet first. say 10.0.17.0/24. It is public because of the above internet gateway entry in its route table. This is done by creating a route table under VPC section, and that route table has this internet gateway underneath. This route table is then attached to this subnet 10.0.17.0/24 so now it stands "public".
Now, create another subnet 10.0.20.0/24.
Create a NAT Gateway, and make an entry for it in this subnet's route table. This NAT Gateway, must flow into the public subnet created above, i.e 10.0.17.0/24.
Now, you private subnet 10.0.20.0/24 has internet access via NAT gateway interfacing the public subnet above.
2.) Enter "Security Groups". total 4 SGs are at play,
of which, ElasticMapReduceEditors-Editor and ElasticMapReduceEditors-Livy must have outbound to 0.0.0.0/0 allowed. The other two SGs are for Master and Slaves, i.e ElasticMapReduce-Master-Private and ElasticMapReduce-Slave-Private
3.) Make Private Git repositories, Under the settings section of your github profile, create a Personal Access Token. While furnishing details on creating an EMR Repository, add this Secret Value, save it. Your Notebook Service Role must have permission "GetSecretValue" on all the Repositories ie "r-*".
Also, Make sure that the branch specified must have to exist before you configure while creating / linking repositories.
The biggest problem with AWS is that they are great Salesmen but procrastinate when it comes to after-sales / maintaining so many products they make, Their documentations are terrible, lack in end-to-end examples and people have to rely on SO communities. I am sorry friend I am procrastinating too :p I would want to go get this into a Fine Blog and furnish the link in my this answer next!