0
174
ViewsPandas Merging 101
- How can I perform a (
INNER| (LEFT|RIGHT|FULL)OUTER)JOINwith pandas? - How do I add NaNs for missing rows after a merge?
- How do I get rid of NaNs after merging?
- Can I merge on the index?
- How do I merge multiple DataFrames?
- Cross join with pandas
merge?join?concat?update? Who? What? Why?!
... and more. I've seen these recurring questions asking about various facets of the pandas merge functionality. Most of the information regarding merge and its various use cases today is fragmented across dozens of badly worded, unsearchable posts. The aim here is to collate some of the more important points for posterity.
This Q&A is meant to be the next installment in a series of helpful user guides on common pandas idioms (see this post on pivoting, and this post on concatenation, which I will be touching on, later).
Please note that this post is not meant to be a replacement for the documentation, so please read that as well! Some of the examples are taken from there.
Table of Contents
For ease of access.
3 answers
Answer question0
A supplemental visual view of pd.concat([df0, df1], kwargs).
Notice that, kwarg axis=0 or axis=1 's meaning is not as intuitive as df.mean() or df.apply(func)
![on pd.concat([df0, df1])](https://i.stack.imgur.com/1rb1R.jpg)
0
In this answer, I will consider practical examples.
The first one, is of pandas.concat.
The second one, of merging dataframes from the index of one and the column of another one.
Considering the following DataFrames with the same column names:
Preco2018 with size (8784, 5)
Preco 2019 with size (8760, 5)
That have the same column names.
You can combine them using pandas.concat, by simply
import pandas as pd
frames = [Preco2018, Preco2019]
df_merged = pd.concat(frames)
Which results in a DataFrame with the following size (17544, 5)
If you want to visualize, it ends up working like this
(Source)
2. Merge by Column and Index
In this part, I will consider a specific case: If one wants to merge the index of one dataframe and the column of another dataframe.
Let's say one has the dataframe Geo with 54 columns, being one of the columns the Date Data, which is of type datetime64[ns].
And the dataframe Price that has one column with the price and the index corresponds to the dates
In this specific case, to merge them, one uses pd.merge
merged = pd.merge(Price, Geo, left_index=True, right_on='Data')
Which results in the following dataframe
0
Joins 101
These animations might be better to explain you visually. Credits: Garrick Aden-Buie tidyexplain repo
Inner Join
Outer Join or Full Join
Right Join
Left Join
