There are two locks on the PMD and PTE levels of the page table in the Linux kernel. Each time a thread/process allocates/maps a memory it should hold one of these locks to update the page table accordingly. It is obvious that as the number of threads increases the race for holding the locks increases as well. This may degrade the memory mapping throughput since many threads hold the spinlock.
What I'd like to measure for a task, is the worst-case overhead of these locks on memory mapping throughput but I have no idea how to measure it.
I tried using malloc in an infinite-loop as I increase the number of threads running the same loop. I check the /proc/{pid}/maps for each set of running threads to count the number of mapped memories. But I'm not sure if it is the correct way. Besides, this method consumes a lot of memory.
Is there any efficient way to measure the worst-case overhead of these locks?
A lot of the comments are correct, however I thought I might try my hand at a full response. Firstly, using malloc will not enable you to have explicit control over page mappings as a comment said the malloc part of stdlib will actually allocate a huge chunk of memory after the first allocation. Secondly when creating a new thread, this will use the same address-space, so there will be no additional mappings created.
I'm going to assume you want to do this from user space, because from kernel space, you can do a lot of things to make this exploration somewhat degenerate (for example you can just try and map pages to the same location). Instead you want to allocate anonymous pages using mmap. Mmap is an explicit call to create a Virtual Memory Entry so that when that particular page is accessed for the first time, the kernel can actually put some blank physical memory at that location. It is the first access to that location that causes the fault, and that first access which will actually use the locks in the PTE and PUD.
Ensuring Good Benchmarking Procedure:
What does this translate in each thread:
address = <per-thread address>
total = 0;
for(int i = 0; i < N; i++)
{
uint64_t* x = (uint64_t*) mmap((void*) address, 4096, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS, -1, 0); //Maps one page anonymously
assert(x);
*x ^= pseudo_rand(); // Accesses the page and causes the allocation
total += *x; // For fun
int res = munmap((void*) x, 4096); //Deallocates the page (similar locks)
assert(!res);
}
The big take aways are:
mmap and explicitly access the allocated location to actually control individual page allocation.