This occurs due to files names with the '~' prefix in site-packages. Please remove those files / folders. It will solve your problem.

You can list them with

ls -a /xx/xx/xx/lib/pythonx.x/site-packages | grep "^~"`

Then remove files/folders prefixed by '~'.

go to 'site-packages' directory, delete folders whose name start with ~ (tilde)

This happens due to multiple versions installation of libraries, at times conda or pip failed to uninstall the versions instead rename it to "~ip" so, suggest we find site-packages in our python installation and search for "~ip" and delete such folders which begin with ~ip. Then the problem will get resolved and we will not get any such warnings. Otherwise, it is only a warning because of such folders existing in the installation that we can ignore.

I faced the same issue. It got fixed by going to site-packages and deleting the folders starting with ~.

>pip install pyspark
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)
Requirement already satisfied: pyspark in c:\users\appdata\roaming\python\python39\site-packages (3.2.1)
Requirement already satisfied: py4j==0.10.9.3 in c:\users\anaconda3\lib\site-packages (from pyspark) (0.10.9.3)
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)
WARNING: Ignoring invalid distribution -yspark (c:\users\anaconda3\lib\site-packages)

Deleted the duplicate folder ~spark 0.2.1.

>pip install pyspark
Requirement already satisfied: pyspark in c:\users\appdata\roaming\python\python39\site-packages (3.2.1)
Requirement already satisfied: py4j==0.10.9.3 in c:\users\anaconda3\lib\site-packages (from pyspark) (0.10.9.3)