I am trying to have two processes communicate via shared memory. I noticed that if I run them with different users, the second one fails on shm_open() with permission denied, even if they are from the same group. But if I add the execute bit to the mode, the second process does not fail on the call to shm_open().

I would like to understand this behaviour, what exactly is the execute bit doing in the case of shm_open() ? Particularly, can a process execute shellcode in /dev/shm if the execute bit is set during the call to shm_open() ?

The man pages are the best docs I found about this. From what it says in O_CREAT:

Create the shared memory object if it does not exist. The user and group ownership of the object are taken from the corresponding effective IDs of the calling process, and the object's permission bits are set according to the low- order 9 bits of mode, except that those bits set in the process file mode creation mask (see umask(2)) are cleared for the new object

From my understanding it gets it's permission bits from mode (my umask is set top 0002, but changing to 0 doesn't seem to produce a different outcome). I keep O_CREAT in both programs because the manual states "Create the shared memory object if it does not exist", so I wanted to reuse the same code for both.

MWE:

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <stdlib.h>

int main() {
    const char* name = "com/page";
    int flags = O_CREAT | O_RDWR;
    int mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP;
    int fd = shm_open(name, flags, mode);
    if (fd < 1) {
        printf("shm_open() failed: %s", strerror(errno));
        exit(EXIT_FAILURE);
    }

    /* ... */

    shm_unlink(name);

    return EXIT_SUCCESS;
}

As it stands, the above snippet returns -1 with errno set to permission denied if the programs are run with different users, but both in the same group.

If I replace the 12th line above with:

int mode = S_IRWXU | S_RWXG

Then the call to shm_open() returns 0.

This seems related to when a directory lacks the execute bit, and we can't cd into it. I tried to use a path instead of a file name (with the folder structure in /dev/shm created before hand) but then shm_open() fails with invalid argument.