I am using Django logger for logging along with gunicorn workers. Currently, I am testing with 3 workers and I am using the TimedRotatingFileHandler. I configured when = m and interval = 1, I tried sending some 1000 requests and it is writing in the new file and overwriting the old file as well.
Here is my config from settings.py
'debug': {
'level': 'DEBUG',
'filename': BASE_DIR + '/Log/debug.log',
'class': 'logging.handlers.TimedRotatingFileHandler',
'when': 'M',
'interval': 1,
'formatter': 'verbose'
},
This will happen only if i use more than 1 gunicorn worker. But if I use 1 gunicorn worker this is not happening.
You shouldn't be using file-based handlers directly from multiple processes concurrently, which is what you get if you use gunicorn with this configuration. Instead, use a SocketHandler or QueueHandler to send logs from the webapp to a separate process which logs them to disk. The Python logging cookbook has more information and example code, which is too long to reproduce here.
Writing to a single file from multiple concurrent processes without them treading on each other isn't supported in Python (and a lot of other languages too).