I was wondering if anyone could help me with this.

My code:

from gc import callbacks
import re
import scrapy
from scrapy import Spider
from scrapy.http import FormRequest
from scrapy_splash import SplashRequest

class ProvinciaSpider(Spider):
    name = 'provincia'
    allowed_domains = ['aduanet.gob.pe']
    start_urls = ['http://www.aduanet.gob.pe/cl-ad-itconsmanifiesto/manifiestoITS01Alias?accion=cargaConsultaManifiesto&tipoConsulta=salidaProvincia']

    def parse(self, response):
        data ={ 'accion': 'consultaManifExpProvincia',
        'salidaPro': 'YES',
        'strMenu': '-',
        'strEmpTransTerrestre': '-',
        'CMc1_Anno': '2022',
        'CMc1_Numero': '354',
        'CG_cadu': '046',
        'viat': '1'}

        yield FormRequest('http://www.aduanet.gob.pe/cl-ad-itconsmanifiesto/manifiestoITS01Alias', formdata=data, callback=self.parse_form_page)

    def parse_form_page(self, response):
       #I extract and yield

Until here I haven't used splash and I'm already inside the page where I can extract without problem, the problem comes when I want to go to the next page and do the same thing.

XPath of the "next page" link: /html/body/form[1]/table[4]/tbody/tr/td/table[2]/tbody/tr/td[3]/a

which I reduced to this: /html/body/form[1]//td[3][@class="lnk7"]

Now, inside the def parse_form_page and after the extract and yield part, I put this script and yield SplashRequest.

 script = """function main(splash)
                assert(splash:go(splash.args.url))
                splash:wait(0.3)
                button = splash:select("/html/body/form[1]//td[3][class=lnk7] a")
                splash:set_viewport_full()
                splash:wait(0.1)
                button:mouse_click()
                splash:wait(1)
                return {url = splash:url(),
                        html = splash:html()}
            end"""


        yield SplashRequest(url ='http://www.aduanet.gob.pe/cl-ad itconsmanifiesto/manifiestoITS01Alias',
                            formdata=data,
                            callback =self.parse_form_page,
                            endpoint='execute',
                            args = {'lua_source': script})

I know this is wrong because is not working, it is still just extracting the first page. I think the script part is ok, but I'm not sure how to make the yield part, I've seen codes where they just put url= url. They enter the page with splash and it didn't have like a login page, so that's why I don't really know how to make this part.

I'm using splash for this part because the "next page" links are like this:

<a href="javascript:paginacion(10,1)">Siguiente</a>