掌握使用requests发送带参数的GET请求
课程位置:04.第一个爬虫.pdf - 4.3 requests模块
核心方法:
requests.get(url) - 发送GET请求requests.get(url, params=params) - 带参数请求requests.get(url, headers=headers) - 携带请求头response.content.decode("utf-8") - 解码响应本页面提供商品搜索功能,请使用requests发送带参数的请求进行搜索。
keyword - 搜索关键词(如:iPhone、Mac、iPad)category - 商品分类(手机、电脑、平板、耳机)请使用代码发送带参数的请求来搜索商品
示例1:搜索关键词
import requests
url = 'https://req.haleibc.com/practice2'
params = {'keyword': 'iPhone'}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get(url, params=params, headers=headers)
print(response.content.decode("utf-8"))
示例2:按分类搜索
import requests
url = 'https://req.haleibc.com/practice2'
params = {'category': '手机'}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get(url, params=params, headers=headers)
print(response.content.decode("utf-8"))
示例3:组合搜索
import requests
url = 'https://req.haleibc.com/practice2'
params = {
'keyword': 'Pro',
'category': '电脑'
}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get(url, params=params, headers=headers)
print(response.content.decode("utf-8"))
1. 搜索关键词"iPhone",查看返回结果
2. 搜索分类"手机",查看返回结果
3. 组合搜索:关键词"Pro" + 分类"电脑"
4. 使用BeautifulSoup提取商品名称和价格
import requests
from bs4 import BeautifulSoup
url = 'https://req.haleibc.com/practice2'
params = {'keyword': 'iPhone'}
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
response = requests.get(url, params=params, headers=headers)
soup = BeautifulSoup(response.content, 'html.parser')
# 提取所有商品
products = soup.find_all('div', class_='book-card')
for product in products:
name = product.find('h3', class_='book-title').text
price = product.find('p', class_='book-price').text
print(f'{name}: {price}')