I have an Intel Xeon Phi 64-core CPU with 16GB on-chip memory set as NUMA node 1. I want to bind a process running inside a Docker container to this NUMA node, but it errors out:
root@Docker$ sudo numactl --membind 1 python script.py
Error: set_mempolicy: Operation not permitted
setting membind: Operation not permitted
I also tried to bind the entire Docker mothership to NUMA node 1, but no luck:
localhost$ sudo numactl --membind 1 docker run xyz /bin/bash
This doesn't give any error, but shows no utilization of on-chip memory.
My question is: How can I bind a process to NUMA node 1 successfully? Thanks!
You need to get which cpus belongs to 1 numa node:
$ lscpu | grep node1
NUMA node1 CPU(s): 8-15,24-31
Create cpuset for docker:
$ cgcreate -g cpuset:docker
$ echo 8-15 > /sys/fs/cgroup/cpuset/docker/cpuset.cpus
If you want to run just certain containers on specific NUMA node:
cgcreate -g cpuset:my-cgroup
echo 8-15 > /sys/fs/cgroup/cpuset/my-cgroup/cpuset.cpus
Docker allows to put new containers under a different cgroup:
docker run -it --rm --cgroup-parent=/my-cgroup/ ubuntu bash
For more information see https://sthbrx.github.io/blog/2016/07/27/get-off-my-lawn-separating-docker-workloads-using-cgroups/ .
I had the same issue which got solved by creating a container using "--privileged" mode.
With this mode, we can bind mem and CPU nodes inside the container and use numa effectively.
Use the below command:
#docker run --privileged --name=test_numa --net=host -itd <image_id>
Try granting the deceptively-named CAP_SYS_NICE capability to the container, which allows the process to "set CPU affinity for arbitrary processes" [ref]:
docker run --cap-add SYS_NICE <other arguments>
This a safer approach than --privileged but should still enable what you need.