I have a dataframe df as follows:
C1 C2 C3
a1 a d
b1 b e
c1 c f
If I do df[df['C1'] == 'a1'] I get
C1 C2 C3
a1 a d
Is there some way of comparing things so that if I do df[df['C1'] == 'what should I use here'], I get all rows. i.e. I am looking for a dummy comparison variable so that comparison statement is ignored and all rows are returned.
C1 C2 C3
a1 a d
b1 b e
c1 c f
Edit: As the code is grandfathered, I need to use only == comparison statement. I can't use !=. The comparison variable is coming through a for loop and I can insert a dummy variable in the for loop if available.
This is what the original code is which I can't modify:
dfNew = []
for ii in ['a1', 'a2']:
for jj in ['a', 'b']:
dfNew.append(df[(df['C1'] == ii) & (df['C2'] == jj)])
What I am trying to achieve in dfNew is to also append values where if there was a dummy variable in ['a1', 'a2', 'dummy'] then only the other comparison operation is done. The number of columns is close to 50 so it will be impossible to code every pair to ignore in conditional operation.
Edit2:
How do you pass the dummy inside ['a1', 'a2', 'dummy'] ?
Dummy variable with a class.If you want a real Dummy variable, the official and cleanest way would be to create a class where __eq__ (equality check) always returns True:
class Dummy:
def __eq__(self, other):
return True
And now:
>>> df[df['C1'] == Dummy()]
C1 C2 C3
0 a1 a d
1 b1 b e
2 c1 c f
>>>
Always gives True.
The Dummy class returns True for all equality checks.
For the edit.
Just do:
['a1', 'a2', Dummy()]
unittest.mock.any which does this for us already.The unittest module (builtin module) already done this class for us. We could use that as well:
from unittest.mock import ANY
And now:
>>> df[df['C1'] == ANY]
C1 C2 C3
0 a1 a d
1 b1 b e
2 c1 c f
>>>
Would work too!
The source code of unittest.mock.ANY is the exact same as what I did:
Getting the source code of ANY:
import inspect
print(inspect.getsource(ANY.__class__))
Output:
class _ANY(object):
"A helper object that compares equal to everything."
def __eq__(self, other):
return True
def __ne__(self, other):
return False
def __repr__(self):
return '<ANY>'
It's the same thing as what I did :).
Why not this?
df[df['C1'] == df['C1']]
That will work for all values except for NaN, because NaN != NaN.
In fact, because of that very property (perhaps more acurately, definition) of NaN, you could do this (although the OP explicitly required solely the use of ==):
df[df['C1'] == np.nan]
(I test that, and, as expected, all rows were returned, including NaN ones.)