It is evident that your extensionless URLs are currently dependent on MultiViews (mod_negotiation). However, it's not clear where the external redirect (that appends the .php extension to the URL you are seeing in the address bar) is coming from. MultiViews does not do this. So, either you have some other application logic that redirects to append the extension or you are seeing a cached response from an earlier (erroneous) redirect (although if disabling MultiViews results in a 404 then that would seem to rule out the "cache").

It's also possible you have directives elsewhere (in the server config) that might be doing this, after all, MultiViews must have been explicitly enabled somewhere - this is not enabled by default.

Moreover, I want it only for php extension. Because when user types domain.com/robots I don't want robots.txt to be displayed, but 404 page.

In that case, you cannot use MultiViews to manage your extensionless URLs. You need to use mod_rewrite and internally rewrite the URL instead to specifically target .php files only.

Try the following instead:

# Ensure that MultiViews is disabled
Options -Indexes -MultiViews

# Specify the actual file in the ErrorDocument, not simply "/404"
ErrorDocument 403 /404.php
ErrorDocument 404 /404.php

RewriteEngine On

# Rewrite extensionless URL to ".php" if it exists
RewriteCond %{DOCUMENT_ROOT}/$1.php -f
RewriteRule ^([^.]+)$ $1.php [L]

The regex ^([^.]+)$ assumes that your extensionless URLs do not otherwise contain a dot. The only dot occurs before the file extension .php - which is omitted in the request. This allows the directive to be more efficient and to avoid checking static resources etc. and to naturally avoid rechecking the request after it has been internally rewritten to append the .php extension (when applicable).

However, this may or may not resolve the redirect issue. This may depend on how the redirect is being triggered in the first place.

As an added bonus you can also redirect direct requests for .php files to remove the file extension. This is necessary if you are changing an existing URL structure where the .php URL might have been indexed by search engines and/or linked to by external third parties.

To do this you would add the following rule immediately after the RewriteEngine directive and before the internal rewrite (that appends the extension).

# Remove the ".php" extension on direct requests
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteRule (.+)\.php$ /$1 [R=301,L]

The condition that checks against the REDIRECT_STATUS environment variable ensures we only check direct requests from the client and not rewritten requests by the later directive.

Test first with a 302 (temporary) redirect to avoid potential caching issues.

However, if the earlier "redirect issue" has not already been resolved then this will likely result in a redirect-loop.