I want to replace null values in one column with the values in an adjacent column ,for example if i have
A|B
0,1
2,null
3,null
4,2
I want it to be:
A|B
0,1
2,2
3,3
4,2
Tried with
df.na.fill(df.A,"B")
But didnt work, it says value should be a float, int, long, string, or dict
Any ideas?
Another Answer.
If the below df1 your dataframe
rd1 = sc.parallelize([(0,1), (2,None), (3,None), (4,2)])
df1 = rd1.toDF(['A', 'B'])
from pyspark.sql.functions import when
df1.select('A',
when( df1.B.isNull(), df1.A).otherwise(df1.B).alias('B')
)\
.show()
df.rdd.map(lambda row: row if row[1] else Row(a=row[0],b=row[0])).toDF().show()