I am trying to write some code which captures limited information from webpages by using the following code in Google Apps Script:
var url = "myurlhere.com";
var str = UrlFetchApp.fetch(url).getContentText();
const RegexExp = /(?<=<span class=\"text-match\">).*?(?=<\/span>)/gi;
var result = str1.match(RegexExp);
And this works BEAUTIFULLY. The problem comes when the url page source is a client-side Javascript creation. I am crawling all over the internet, and basically everywhere I have looked, I get the same answer... "you need a headless browser", "use Node.js or Phantom.js".
This is irrelevant to me. I am not the owner of the HTML file, so I am either missing something or Node.js does not work here, and Phantom.js will not work because this is not my local machine.
My current thoughts toward a solution... create a pop-up using Apps Script (load the entire webpage, not just the Javascript source), and find a way to grab the generated HTML (the stuff you see when you "inspect" a page), but this is where I get stuck. I'm looking for an expert who may know best how to interface with a chrome tab and grab this information with Google Apps Script.
Current attempt progress:
function openURLdata() {
var js = " \
<script> \
window.open('myurlhere.com', 'URLdata', 'width=800, height=600'); \
google.script.host.close(); \
</script> \
";
var html = HtmlService.createHtmlOutput(js)
.setHeight(10)
.setWidth(100);
Logger.log(html.getContent());
SpreadsheetApp.getUi().showModalDialog(html, 'Now loading.'); // If you use this on Spreadsheet
}
Does anyone know if this is a step in the right direction? Does Google's server-side communications mean that I won't ever get a plain text interpretation of the HTML page that I need my data from? I have to get this task done in Google Sheets, and the target site does not have an API I can use.