I am having trouble with a performance bottleneck trying to update many records across two tables at once. Currently, I have some Pandas DataFrames (new_records_df and modified_records_df) containing the records I would like to insert / update. See the following psuedocode:

if not new_records_df.empty:
    new_recs_data = new_records_df.T.to_dict().values()  # creates a list of dictionaries from the DataFrame
    new_recs = []
    for r in new_recs_data:
        new_rec = {'foo_id': foo_id,
                   'bar': bar}
        new_recs.append(new_rec)
    db_session.bulk_insert_mappings(Record, new_recs, return_defaults=True) # return_defaults inserts the id of the inserted record into the dictionary object
    new_related_recs = []
    for nr in new_recs:
        new_related_rec = {'rec_id': nr['id'],
                           'baz': baz}
        new_related_recs.append(new_rec)
    db_session.bulk_insert_mappings(RelatedRec, new_related_recs)

if not modified_records_df.empty:
    modified_rec_data = modified_records_df.T.to_dict().values()  # again, converting teh DataFrame to a list of dicts
    modified_recs = []
    for m in modified_rec_data:
        modified_rec = {'id': m['id'],
                        'zab': zab}
        modified_recs.append(modified_rec)
    db_session.bulk_update_mappings(RelatedRec, modified_recs)  # when a record is modified, only the RelatedRec object is updated. The Record object already exists and stays unmodified

problem is, with ~8k records the looping over the dictionaries takes ~20 seconds, while the actual insert / update of the database only takes about 4 seconds. I was hoping that there is a clever way that I can eliminate the for loop since this appears to be the bottleneck. My database is postgres and my driver is psycoppg2 2.6.2