I have a Spark Dataset similar to the example below:
0 1 2 3
+------+------------+--------------------+---+
|ItemID|Manufacturer| Category |UPC|
+------+------------+--------------------+---+
| 804| ael|Brush & Broom Han...|123|
| 805| ael|Wheel Brush Parts...|124|
+------+------------+--------------------+---+
I need to find the position of a column by searching the column header.
For Example:
int position=getColumnPosition("Category");
This should return 2.
Is there any Spark function supported on Dataset<Row> datatype to find the column index or any java functions which can run on Spark dataset?
You need to access the schema and read the field index as follows:
int position = df.schema().fieldIndex("Category");
I have used index attribute for lists:
df.columns.index(column_name)
You can consider this option (Scala implementation):
def getColumnPosition(dataframe: DataFrame, colName: String): Int = {
dataframe.columns.indexOf(colName)
}