爬虫翻页

查看 38|回复 0
作者:milu1123   
各位大大,,以下代码可以实现爬虫的模拟翻页吗?
如果可以,请问是如何实现呢??
[Asm] 纯文本查看 复制代码import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
def xiayiye():
    nxt = requests.get(url)
    nxt.encoding = nxt.apparent_encoding
    soup = BeautifulSoup(nxt.text, 'lxml')
    tiezi1 = soup.find("a", class_="nxt")
    fanhui = urljoin(url, tiezi1['href'])
    return fanhui
def main(url):
    resp = requests.get(url)
    resp.encoding = resp.apparent_encoding
    soup = BeautifulSoup(resp.text, 'lxml')
    tiezi = soup.find_all("a", class_="s xst")
    for s in tiezi:
        dic = {
        "name" : urljoin(url, s['href']),
        "text" : s.text + "\n",
        }
        print(dic)
        
    print(xiayiye())
if __name__ == '__main__':
    url = ''
    main(url)

爬虫, 翻页

您需要登录后才可以回帖 登录 | 立即注册

返回顶部