I use manage.py dumpdata --format xml --some-more-parameters to export a full dump of the database to xml. The database is MS sql server and I'm using pyodbc as the driver. The dumpdata command is run using PowerShell and since Django 1.7 does not support a --output argument for the dumpdata command I redirect the output into a file using PowerShell.
Unfortunately the database contains unicode characters (e.g. country \xd6sterreich) and these characters are scrambled int the export file.
Here's what didn't work:
./manage.py dumpdata --format xml > export.xml
./manage.py dumpdata --format xml | out-file -encoding utf8 export.xml
./manage.py dumpdata -format xml | out-file -encoding ANY_OTHER_SUPPORTED_ENCODING export.xml
None of these commands work. Umlauts and accents are scrambled and additionally the > export.xml method adds an invalid BOM to the file which will result in ./manage.py loaddata export.xml aborting with an UnicodeDecode error message when I try to import this on another host.
Any suggestions on how I could export the data and preserve the special characters? The same problem exists when using the json or yaml serializers.
I was able to work around this problem using my own export script. The script below will dump the data and store it in a utf-8 encoded xml file called export_CURRENT-DATE-TIME.xml. call_command() calls the dumpdata command in Django. The script below should be equivalent to using dumpdata with the following arguments:
./manage.py dumpdata --natural --natural-foreign --natural-primary --format xml --indent 2
import sys
import codecs
import os
import django
from django.core.management import call_command
from StringIO import StringIO
from datetime import datetime
# setup access to django
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "PROJECT_NAME.settings")
django.setup()
# the actual export command
def do_work():
#print(u"\xd6sterreich")
call_command('dumpdata', use_natural_keys=True, use_natural_foreign_keys=True, use_natural_primary_keys=True, format='xml', indent=2)
# nasty hack to workaround encoding issues on windows
_stdout = sys.stdout
sys.stdout = StringIO()
do_work()
value = sys.stdout.getvalue().decode('utf-8')
sys.stdout = _stdout
with codecs.open('export_{}.xml'.format(datetime.now().strftime("%Y-%m-%d_%H-%M")), 'w', 'utf-8-sig') as f:
f.write(value)
print("export completed")