This works for me:

app.use(
        helmet.contentSecurityPolicy({
            directives: {
                "script-src": ["'self'"],
                upgradeInsecureRequests: null
            },
        })
    );

Setting upgradeInsecureRequests to null:

upgradeInsecureRequests: null

Solved: we can simply add upgradeInsecureRequests: []

This worked for me:

defaultDirectives = helmet.contentSecurityPolicy.getDefaultDirectives();
delete defaultDirectives['upgrade-insecure-requests'];

app.use( helmet() );
app.use(helmet.contentSecurityPolicy({
  directives: {
    ...defaultDirectives,
  },
}));

The delete part removes the upgrade-insecure-requests key in the defaultDirectives object.