DataFrame :
Df row1 : Ravi Computers 20
Df row2 : Jon Electronics 21
Df row3 : Sam arts 20
How can I write to write into s3 file as
Line1: Index:Ravi
Line2: Ravi Computers 20
Line3: Index:Jon
Line4: Jon Electronics 21
Line5: Index:Sam
Line6: Sam arts 20
For writing the Spark SQL DataFrame into 2 lines to S3, you have to map each row of DF in to respective string with a new line \n:
val df = sc.parallelize(Seq(("Ravi","Computers",20),("Jon","Electronics",21),
("Sam","arts",20))).toDF
df.map(r => s"Index:${r.getString(0)}\n${r.getString(0)} ${r.getString(1)} ${r.getInt(2)}").write.csv("s3n://........")
It will write the DF into expected output format:
Line1: Index:Ravi
Line2: Ravi Computers 20
Line3: Index:Jon
Line4: Jon Electronics 21
Line5: Index:Sam
Line6: Sam arts 20