I'm making an api call, and trying to store its response in Postgres using Django models. Here's what I have been doing:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
#handler is a object of a model
handler.api_response = response_data
handler.save()
But this used to fail, when my json had fields like 'field_name': '\x00\x00\x00\x00\x00'. It used to give following error :
DataError at /api/booking/reprice/
unsupported Unicode escape sequence
LINE 1: ... NULL, "api_status" = 0, "api_response" = '{"errorRe...
^
DETAIL: \u0000 cannot be converted to text.
CONTEXT: JSON data, line 1: ..."hoursConfirmed": 0, "field_name":...
How i tried to resolve this is by using the following:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
#handler is a object of a model
handler.api_response = json.loads(json.dumps(response_data).encode("unicode-escape").decode())
handler.save()
The initial issue was solved then. But recently, when i got a field with value 'field2_name': 'Hey "Whats up"'. This thing failed by giving error:
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 143 (char 142)
Probably because json.loads() got confused with " inside the value as an enclosing " and not an escaped ".
Now, If i print the initial response just after json.loads(response.content.decode('utf-8')) statement, it shows the field as \x00\x00\x00\x00\x00.
But output of following code:
response = requests.post(url='some.url.com', data=json.dumps(data), headers={'some': 'header'})
response_data = json.loads(response.content.decode('utf-8'))
print(json.dumps(response_data))
This shows the field as \\u0000\\u0000\\u0000\\u0000\\u0000.
How does \x00 change to \\u0000
And how do i save this field into postgres tables ? This is what i could think of.
json.loads(json.dumps(response_data).replace('\\u0000',''))
To add this statement before saving to postgres.
Is there a better way ?
Is the code response_data = json.loads(response.content.decode('utf-8')) wrong ? Or causing not to escape that particular character ?
FWIW, I recently ran into this and your proposed solution also worked for me. I think it's probably the simplest way to deal with it. Apparently this is the only character that can't go into a Postgres JSON column.
json.loads(json.dumps(response_data).replace('\\u0000',''))