I have written a pausable multi-thread downloader using requests and threading, however the downloads just can't complete after resuming, long story short, due to special network conditions the connections can often die during downloads requiring refreshing the connections.
You can view the code here in my previous question:
Python multi connection downloader resuming after pausing makes download run endlessly
I observed the downloads can go beyond 100% after resuming and won't stop (at least I haven't see them stop), mmap indexes will go out of bounds and lots of error messages...
I have finally figured out this is because the ghost of previous request, that makes the server mistakenly sent extra data from last connection that was not downloaded.
This is my solution:
s = requests.session()
r = s.get(
url, headers={'connection': 'close', 'range': 'bytes={0}-{1}'.format(start, end)}, stream=True)
r.close()
s.close()
del r
del s
In my testing, I have found that requests have two attributes named session, one Titlecase, one lowercase, the lowercase one is a function, and the other is a class constructor, they both create a requests.sessions.Session object, is there any difference between them?
And how can I set keep-alive to False?
The method found here won't work anymore:
In [39]: s = requests.session()
...: s.config['keep_alive'] = False
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-39-497f569a91ba> in <module>
1 s = requests.session()
----> 2 s.config['keep_alive'] = False
AttributeError: 'Session' object has no attribute 'config'
This method from here doesn't throw errors:
s = requests.session()
s.keep_alive = False
But I seriously doubt that it has any effect at all, it just adds a new boolean to the object and I don't think it is handled by the object.
I have seen a close method in requests.models.Response, does it have any effect in this case or I can just close the session?
And finally, using this method, is it guaranteed that the server will never send extra bytes from previous dead connections?
In general, with python, when there is some kind of 'handler' that is supposed to close after use, wrapping the use with with can limit the scope of the thing to a small code block.
ResponseData = None
with requests.get( Url, headers=Headers) as ResponseObject:
ResponseData = ResponseObject.text.encode('utf-8')
ResponseData = ResponseData.decode("utf-8")
#code down here does not have any idea what "ReponseObject" is.
#For some reason python is able to kill it more reilably after `with`
Not sure if this is a cannonical answer but it might help for you. The snippet works for me, but I avoid creating a session entirely. This trick has worked for me for countless other things that were supposed to close, but did not.
EDIT: Following up on session: I guess you can double with and see if it works?
with requests.session() as s:
with s.get(....) as r:
#try stuff here