I want to deconstruct a pandas DataFrame, using column headers as a new data-column and create a list with all combinations of the row index and columns. Easier to show than explain:
index_col = ["store1", "store2", "store3"]
cols = ["January", "February", "March"]
values = [[2,3,4],[5,6,7],[8,9,10]]
df = pd.DataFrame(values, index=index_col, columns=cols)
From this DataFrame I wish to get the following list:
[['store1', 'January', 2],
['store1', 'February', 3],
['store1', 'March', 4],
['store2', 'January', 5],
['store2', 'February', 6],
['store2', 'March', 7],
['store3', 'January', 8],
['store3', 'February', 9],
['store3', 'March', 10]]
Is there a convenient way to do this?
The structure that you want your data in is very messy, so this is probably the best method given the data you want.
# Results
res = []
# Nested loop: first for length of index col, then next for cols
for i in range(len(index_col)):
for j in range(len(cols)):
# Format of data
res.append([index_col[i], cols[j], values[i][j]])
# Return results
print(res)
return res
You can iterate over dataframe items using
data = []
for col, row in df.items():
for ind, val in row.reset_index().values:
data.append([ind, col, val])
data
You could avoid the second loop for sacrificing the order you requested the output in as it is a bit of a full breakdown of how the structure started.
df.unstack().swaplevel().reset_index().values.tolist()
#OR
df.reset_index().melt(id_vars="index").values.tolist()
# [['store1', 'January', 2],
# ['store2', 'January', 5],
# ['store3', 'January', 8],
# ['store1', 'February', 3],
# ['store2', 'February', 6],
# ['store3', 'February', 9],
# ['store1', 'March', 4],
# ['store2', 'March', 7],
# ['store3', 'March', 10]]
With following, the order of elements will match the output in the question.
df.transpose().unstack().reset_index().values.tolist()
# [['store1', 'January', 2],
# ['store1', 'February', 3],
# ['store1', 'March', 4],
# ['store2', 'January', 5],
# ['store2', 'February', 6],
# ['store2', 'March', 7],
# ['store3', 'January', 8],
# ['store3', 'February', 9],
# ['store3', 'March', 10]]
True Pandas-style:
lst = [[*k, v] for k, v in df.unstack().swaplevel().to_dict().items()]
temp = df.stack()
[[*ent, val] for ent, val in zip(temp.index, temp)]
[['store1', 'January', 2],
['store1', 'February', 3],
['store1', 'March', 4],
['store2', 'January', 5],
['store2', 'February', 6],
['store2', 'March', 7],
['store3', 'January', 8],
['store3', 'February', 9],
['store3', 'March', 10]]
I'd prefer stacking over unstacking then swapping the levels:
>>> df.stack().reset_index().to_numpy()
array([['store1', 'January', 2],
['store1', 'February', 3],
['store1', 'March', 4],
['store2', 'January', 5],
['store2', 'February', 6],
['store2', 'March', 7],
['store3', 'January', 8],
['store3', 'February', 9],
['store3', 'March', 10]], dtype=object)
>>>
Or with melt and ignore_index=False:
>>> df.melt(ignore_index=False).reset_index().to_numpy()
array([['store1', 'January', 2],
['store2', 'January', 5],
['store3', 'January', 8],
['store1', 'February', 3],
['store2', 'February', 6],
['store3', 'February', 9],
['store1', 'March', 4],
['store2', 'March', 7],
['store3', 'March', 10]], dtype=object)
>>>