The news crawlers typically operate without restrictions from the target server, facing technical challenges mainly in managing and allocating crawl tasks, concurrent usage, and improving efficiency. However, in practice, crawlers targeting different sites often face many obstacles, one such being login challenges.
>
There was a time when logging in was simple — an account and its password would be POSTed to the server, and upon successful verification, access would be granted. That was a pleasant, uncomplicated era when servers were undefended, and users were not greedy. However, times have changed, and so have people’s intentions. Increasing numbers of individuals wish to collect data, and thus crawlers have multiplied; websites experience pressure from network requests and guard their data closely. The bustling world is for profit; the bustling actions are for profit. The current internet has become a battleground where profit is paramount, and deception runs deep.
Today, numerous websites have set up complex login barriers to prevent crawlers from harvesting large volumes, if not all, of their data. For instance, 12306 uses clickable image verification, Weibo employs distorted letter captchas, Zhihu uses upside-down character selection, while Bilibili requires a puzzle through slider movement for verification. These cumbersome verification processes involve human interaction to thwart automated logins, thereby preventing bulk automation by crawlers.
As everyone knows, the HTTP protocol is stateless, with user login states being tracked by cookies that are transmitted back and forth between the browser and server. Upon completing a login, cookies remain unchanged within a certain timeframe. Grabbing these cookies for crawler usage allows the crawler to maintain a logged-in state and proceed with subsequent scrapping, but this scraping can only last until the cookies expire.
Python study group [784758214] includes installation packages and learning video materials, from basics to advanced, and answers to questions. We hope this helps you quickly understand and learn Python.
1. Three Levels of Crawler Login
Completing the login process is best achieved through automation, so that once your program is written, you can sit back and relax; however, often things require a bit more attention, with logins needing slight human intervention. In summary, achieving login involves the following three levels:
- Automatic login can be accomplished with a simple POST of account credentials;
- Automated login is realized by simulating the login process with a program;
- Login requires human (intelligent) involvement for automation, with AI achieving automatic login;
The first level can be achieved with the requests module and a few lines of code; finding such considerate websites nowadays is like finding a needle in a haystack. The second level is challenging and sought after by those in the crawler community. The third level is handy for temporary quantitative data capture, where human input one-off captchas; using AI to recognize captchas can similarly reach a point of no human involvement, but then it’s beyond the crawler realm, requiring resources that are unimaginable for breaking various complex captchas via AI.
It thus appears that obtaining login state cookies mainly relies on simulating the login process or manually entering captchas.
2. Three Types of Tools for Crawler Login Analysis
Simulating login first requires analyzing the login process of the target website before proceeding with programmatic simulation. To analyze this process, you need tool assistance, and such tools include:
- Chrome Developer Tools (F12)
- Charles, Fiddler Web Debugging Proxy Tools
- Wireshark Packet Capture Tool
We’ve already introduced Chrome’s F12, which helps analyze understanding of site loading, although it falls short of specialized tools like Charles, etc.; Wireshark is a specialized tool for capturing packets, not limited to analyzing the HTTP protocol but capable of TCP, UDP, and more, though too complex for us to analyze login processes. Therefore, we choose specialized web (HTTP) debugging proxies like Charles and Fiddler.