首先安装python环境(废话,去百度怎么安装,后期有时间补上),使用pip命令安装scrapy,再使用scrapy命令创建项目
pip install scrapy scrapy startproject projectname
projectname就是你要创建项目的名字
项目结构如下
![](http://121.40.42.148:1111/wp-content/uploads/2021/11/图片-9.png)
爬虫文件就写在spiders里面(__init__.py文件只是声明这个文件夹是一个python包)
首先创建一个py文件用来写爬虫,直接贴代码慢慢解释
import scrapy import urllib.parse import json import re class JobScrapy(scrapy.Spider): name = '51job' allowed_domains = ['www.51job.com','search.51job.com']## start_urls = ['https://search.51job.com/'] page = 1 pagesize= 0 jobtype=['0100','7700','7200','7300','7800','7400','2700','7900'] urls = 'https://search.51job.com/list/000000,000000,'+jobtype[0]+',00,9,99,+,2,' + \ str(page) + '.html?lang=c&postchannel=0000&workyear=99&cotype=99&d' \ 'egreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare=' url = "search.51job.com" def __init__(self, value ,fileName): self.value = value self.fileName = fileName self.fp = open("Over_"+fileName+".json", 'w', encoding='utf-8') def parse(self, response): urls = self.urls yield scrapy.Request(url=urls, callback=self.fond_parse, dont_filter=True)## dont_filter=True 允许爬取重复页面 def fond_parse(self, response): print(response)
首先解析这个类,继承了Spider 而它也就是爬虫的一个组件。
name属性是这个爬虫模块的名字,在启动爬虫是要与模块名对应
start_urls属性是开始爬取的第一个页面
allowed_domains属性指定了允许爬取的所有域名,不在此域名内的都会被过滤
parse方法是start_url爬取的回调函数,在这里处理(我初学的时候爬了首页,其实这个url应该就是目标页,然后直接取数据,懒得修改了)首页爬取的返回值,可以通过正则表达式,xpath定位等方法找到元素位置
yield scrapy.Request(url=urls, callback=self.fond_parse, dont_filter=True)
scrapy.Request 是一次普通请求默认get,可以修改为post也可以用FormRequest表单请求
dont_filter=True 允许爬取重复页
callback是回调方法
回调方法里面可以继续处理数据或者获取新的页面,比如爬取列表页面后去爬详情页面。后面一些处理后面再写
开一个新坑,目前51job详情页面爬取有滑动验证,有时间我会研究处理的,以及后续伪装ua
Comments | 121 条评论
Small girls are the easiest to throw around in bed https://is.gd/dBsd60
Would we snuggle or fuck? :) https://is.gd/dBsd60
I like watching myself masturbate, do you? https://is.gd/dBsd60
Insert your tongue here https://is.gd/dBsd60
Moving my shorts to the side, would you like a lick? https://is.gd/dBsd60
Tell your friends that I’m your girlfriend ;) https://snip.ly/6327oi
Can I sit on you? https://snip.ly/6327oi
Would you like me to wrap my boobs around your big hard cock? https://snip.ly/6327oi
I mighttttt be getting fucked in the ass tonight, we’ll see how it goes? https://snip.ly/6327oi
Need someone to play with these while I ride https://snip.ly/6327oi
How would you like to take me from behind daddy? https://snip.ly/6327oi
They are so swollen, volunteers for sucking my nipples?? Anyone??? https://tiny.cc/gz35vz
Waiting for you climb in to bed with me… https://tiny.cc/gz35vz
My pussy gets red like this when I’m horny https://tiny.cc/gz35vz
I wonder if it will fit :) https://tiny.cc/gz35vz