The surge of AI‑driven scraper bots has entered a new phase: traffic now originates from millions of compromised residential and mobile devices rather than traditional datacenter proxies. As Jonathan Corbet noted, each compromised IP typically makes only a handful of requests, disguising itself as a regular browser by omitting images or CSS and using fabricated User‑Agent strings. Because the address disappears after a few hits, IP‑based blocking is ineffective and forces site operators to invest in more sophisticated defenses [1].
How residential proxy networks operate
Two distinct operator models drive the current flood. The first is outright criminal: malware installs a lightweight client on home routers, smartphones or streaming boxes, which then obeys commands from a central C2 server. Google’s takedown of the IPIDEA botnet in early 2025 proved that such networks can generate massive, low‑profile traffic, and its disruption briefly lowered scraper volume on LWN before the respite vanished [2]. A second, more public‑facing model offers “ethically sourced” residential IPs.
Companies like Bright Data market VPN‑based SDKs that turn a user’s device into a proxy node. Users unknowingly become part of a global botnet that can be rented for data‑harvesting operations. The business model monetises every megabyte of traffic, turning ordinary broadband lines into a commodity for AI model training [3].
Business impact and risk exposure
For site owners, the primary cost is operational. Defenses such as proof‑of‑work systems (e.g., Anubis) and commercial bot‑mitigation services add latency for legitimate users and require ongoing tuning. LWN’s own experience showed that while these measures prevented a service outage during its largest attack to date, they also forced the team to rewrite key request paths and allocate engineering hours to maintain a low‑friction user experience for genuine readers. Financially, the hidden expense can be significant.
A high‑traffic news site can see bandwidth spikes of several terabytes in a single hour, translating into $10 K–$20 K of additional cloud egress fees, not to mention the indirect cost of reduced SEO rankings if crawlers are blocked indiscriminately. Moreover, overly aggressive blocking can alienate search engines and the Internet Archive, inadvertently reinforcing the monopoly of dominant platforms.
Mitigation tactics that work today 1.
Selective rate‑limiting – Apply thresholds per subnet only after behavioural analysis, not on first request.
- Layered challenges – Use lightweight JavaScript challenges for suspected residential IPs while reserving full CAPTCHAs for persistent offenders.
- Traffic shaping – Prioritise API endpoints for logged‑in users and cache static assets aggressively to reduce load during attacks.
- Partner intelligence – Subscribe to threat‑intel feeds (e.g., Google’s updates on NetNut takedowns) to block known proxy prefixes early [4].
- User‑centric SDK policies – For in‑house apps, audit any third‑party libraries that claim to boost traffic or monetize connections; prohibit those that route traffic through residential networks.
Looking ahead
Google’s recent collaboration with the FBI to dismantle the NetNut network again demonstrates that law‑enforcement action can provide temporary relief, but the underlying economic incentive – cheap, trusted IPs for training massive language models – remains intact [5]. A sustainable fix will likely involve a combination of stronger app‑store vetting, clearer consent mechanisms for residential proxy SDKs, and perhaps a community‑driven open‑crawl that supplies high‑quality data without rewarding illicit scraping. Until such systemic changes occur, enterprises must treat residential‑proxy scraper traffic as a
