I am developing an extension for parsing website data can convert it into csv file, for internal use.

The flow is here:

Step 1) User login to the system and finish verification Step 2) Paste the key to the extension, and use it to request another website Step 3) Parse the response Step 4) Generate CSV

Then my code like this:

// Button onClick function 

const code =[123,234,345...] // From User Input
const result =[]
for(x of target){

  const req = await fetch("https://www.example.com/?q="+123) // GET /123

  const res = await req.text() //  GET Response

  const parser = new DOMParser();
  const htmlDoc = parser.parseFromString(res, 'text/html'); // Parse HTML

  const data = {
     test: htmlDoc.querySelector("#sample1").innerText,
     test2: htmlDoc.querySelector("#sample2").value,
  }
  result.push(data)
  console.log(data)
}
console.log(result)




// Other Code change data to csv

The code works, I can get most of the data in the website and get the parsed data. However, I find some data is missing.

{

   test: "Sample!",
   test2: null,
}

So, I try to run document.querySelector("#sample2").value in the target website console, it works. So I view the response html code to figure out why it's not work in my script. Then I find that.

<div id="sample"> Hello </div>
<div id="sample2"></div>

<script>

// Some other library....

const value = "123"
// Init some UI
$("#sample2").innerText = value
.
.
.
const value2 = "456" + a
// Init some UI
$("#sample3").innerText = value2

</script>

The target website is sever-rendered, and the data is only hard-coded in the JavaScript Part. I think some different solution but seems not perfect. Here is my opinion:

Solution 1) Move the script to the python / node.js (Headless Browser)

• User login required sms 2fa and google captcha verification, it is too complicated to do on script.

Solution 2) Use InnerHTML to put the content into current tab and set timeout

• It is difficult to handle the javascript part error (like init the library multiple times or same variable name)

Solution 3) Regex

• cannot get the calculated result if the code like a = 1 + 2

Solution 4) Try Document.open() + write html + setTimeout

• Some resources in target html will check the domain (CORS)

Therefore, here are my questions:

1. Is it able to open an url (GET / POST) in background, then get the target tab's rendered data ? (like execute document.open("https://www.google.com") and get the google's dom)

2. If no, any other solution?