I have a DataFrame:
| name | age | |
|---|---|---|
| 0 | Paul | 25 |
| 1 | John | 27 |
| 2 | Bill | 23 |
I know that if I enter:
df[['name']] = df[['age']]
I'll get the following:
| name | age | |
|---|---|---|
| 0 | 25 | 25 |
| 1 | 27 | 27 |
| 2 | 23 | 23 |
But I expect the same result from the command:
df.loc[:, ['name']] = df.loc[:, ['age']]
But instead, I get this:
| name | age | |
|---|---|---|
| 0 | NaN | 25 |
| 1 | NaN | 27 |
| 2 | NaN | 23 |
For some reason, if I omit those square brackets [] around column names, I'll get exactly what I expected. That is the command:
df.loc[:, 'name'] = df.loc[:, 'age']
gives the right result:
| name | age | |
|---|---|---|
| 0 | 25 | 25 |
| 1 | 27 | 27 |
| 2 | 23 | 23 |
Why does two pairs of brackets with .loc result in NaN? Is it some sort of a bug or is it intended behaviour? I can't figure out the reason for such a behaviour.
That's because for the loc assignment all index axes are aligned, including the columns: Since age and name do not match, there is no data to assign, hence the NaNs.
You can make it work by renaming the columns:
df.loc[:, ["name"]] = df.loc[:, ["age"]].rename(columns={"age": "name"})
or by accessing the numpy array:
df.loc[:, ["name"]] = df.loc[:, ["age"]].values
When you use double brackets [[]] you are assigning a DataFrame. What you want is assign a (column) Series, and for that you use only one bracket [].
Here is some code:
import pandas as pd
df = pd.DataFrame({'name':['Paul','John','Bill'], 'age':[25,27,23]})
print('Inital Dataframe:\n',df)
df[['name']] = df[['age']]
print("\ndf[['name']] = df[['age']]\n",df)
print("df.loc[:, ['age']]:", type(df.loc[:, ['age']]))
print("df.loc[:, ['name']]:", type(df.loc[:, ['name']]))
df.loc[:, ['name']] = df.loc[:, ['age']]
print("\ndf.loc[:, ['name']] = df.loc[:, ['age']]\n",df)
print('=======================')
df = pd.DataFrame({'name':['Paul','John','Bill'], 'age':[25,27,23]})
print('Inital Dataframe:\n',df)
print("type(df.loc[:, 'age']):", type(df.loc[:, 'age']))
print("type(df.loc[:, 'name']):", type(df.loc[:, 'name']))
df.loc[:, 'name'] = df.loc[:, 'age']
print("\ndf.loc[:, 'name'] = df.loc[:, 'age']\n",df)
And the output:
Inital Dataframe:
name age
0 Paul 25
1 John 27
2 Bill 23
df[['name']] = df[['age']]
name age
0 25 25
1 27 27
2 23 23
df.loc[:, ['age']]: <class 'pandas.core.frame.DataFrame'>
df.loc[:, ['name']]: <class 'pandas.core.frame.DataFrame'>
df.loc[:, ['name']] = df.loc[:, ['age']]
name age
0 NaN 25.0
1 NaN 27.0
2 NaN 23.0
=======================
Inital Dataframe:
name age
0 Paul 25
1 John 27
2 Bill 23
type(df.loc[:, 'age']): <class 'pandas.core.series.Series'>
type(df.loc[:, 'name']): <class 'pandas.core.series.Series'>
df.loc[:, 'name'] = df.loc[:, 'age']
name age
0 25 25
1 27 27
2 23 23
However, here is another strange behaviour: Assigning the double brackets to difference variables, say df1 and df2, and then df1 = df2 works!
Here is some more code:
df = pd.DataFrame({'name':['Paul','John','Bill'], 'age':[25,27,23]})
print('Inital Dataframe:\n',df)
df1 = df.loc[:, ['name']]
df2 = df.loc[:, ['age']]
print("\ndf1 = df.loc[:, ['name']]\n",df1)
print("\ndf2 = df.loc[:, ['age']]\n",df2)
df1=df2
print("\ndf1=df2\ndf1:\n",df1)
And the output:
Inital Dataframe:
name age
0 Paul 25
1 John 27
2 Bill 23
df1 = df.loc[:, ['name']]
name
0 Paul
1 John
2 Bill
df2 = df.loc[:, ['age']]
age
0 25
1 27
2 23
df1=df2
df1:
age
0 25
1 27
2 23