I am scraping a ton of data from this website:
https://legacy-assignments.uspto.gov/assignments/q?db=pat&qt=pat&reel=&frame=&pat=6000244
Incrementing the patent number and going on, etc. It works great, but for every 10/500 pages, I get the following HTML. The issue is, it is inconsistent, where one page will work 90% of the time, but not 10% of the time. Additionally, I am able to access these pages on a browser with javascript completely blocked, so I am not sure if javascript is the issue.
When it does not work, it says 'please enable javascript,' html from that source is below. Where as if you take a look at the webpage, this html with the does not exist in the html when it is loaded. I cannot seem to understand why this webpage works on my browser without javascript, but I am getting a noscript redirect of some sort.
Any help would be fantastic! thank you!
<meta http-equiv="Pragma" content="no-cache">
<meta http-equiv="Expires" content="-1">
<meta http-equiv="CacheControl" content="no-cache">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel="shortcut icon" href="data:;base64,iVBORw0KGgo=">
<script>
(function(){
window["bobcmn"] = "1110101010101020000000120000000520000000022be9288a200000096300000000300000000300000006/TSPD/300000008TSPD_1013...
})();
</script>
<script type="text/javascript" src="/TSPD/08f32eeb83ab20001a095b38d5a74f26272ac31d7df8bfe5a458c6e278cdc42c4ab6aa99baef9d2c?type=7"></script>
<noscript>
Please enable JavaScript to view the page content.<br>Your support ID is: 17860431601783024671.
</noscript>
</head>
<body>
</body>
</html>
'''