Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Server Farms 2025 Continuing discussion of hosting and data center IP ranges |
5 | not2easy | 5:09 pm Mar 1, 2025 | ||||
| AI UAs - Identifying the crawlers AI uses |
1 | not2easy | 5:38 pm Oct 18, 2023 | ||||
| At Home Without the Robots: 2023 edition |
5 | lucy24 | 11:39 am Jul 29, 2023 | ||||
| At Home with the Law-Abiding Robots (2021) |
2 | lucy24 | 8:32 pm Sep 6, 2021 | ||||
| At Home with the Robots: 2020 edition |
11 | lucy24 | 2:48 pm Aug 13, 2020 | ||||
| At Home with the Robots: 2019 edition |
3 | lucy24 | 3:52 am Mar 13, 2019 | ||||
| At Home with the Robots: 2017 edition Critique of the Year's Active User Agents |
15 | lucy24 | 3:52 pm May 18, 2017 | ||||
| User Agents of Programming Libraries and Command Line Tools Common user agents |
14 | incrediBILL | 8:54 pm Apr 29, 2008 | ||||
| IbouBot Something new |
1 | SumGuy | 12:08 am Aug 7, 2025 | ||||
| 0 | 12:00 am Jan 1, 1970 | ||||||
| Where is grok bot / xAI? Does grok use anonymous VPNs to crawl sites? |
7 | SumGuy | 12:05 am Aug 7, 2025 | ||||
| Seeing a lot more activity from applebot lately |
3 | SumGuy | 6:05 am Aug 2, 2025 | ||||
| Web hit from Residential ISP or international VPN? |
5 | SumGuy | 7:26 pm Jul 29, 2025 | ||||
| Search engines choking on phpBB |
13 | Kendo | 12:57 pm Jul 27, 2025 | ||||
| ACEVILLEPTELTD Singapore Scraper Farm |
5 | not2easy | 11:54 pm Jul 12, 2025 | ||||
| What is WordPress user agent? Showing up in visitor logs |
15 | MichaelBluejay | 7:22 pm Jul 7, 2025 | ||||
| Gemini-Deep-Research |
1 | SumGuy | 5:34 am Jul 7, 2025 | ||||
| Brightbot |
2 | lucy24 | 5:25 am Jul 7, 2025 | ||||
| Best method to block unwanted visits via Cpanel Site is routinely being overwhelmed with UAs(?) I don't care about |
6 | millimi1 | 10:40 am Jun 28, 2025 | ||||
| GoogleBot Spoofer 112.121.172.18 |
2 | Bewenched | 6:16 pm Jun 22, 2025 | ||||
| Ping probe from Anthropic IP |
4 | SumGuy | 12:00 am Jun 22, 2025 | ||||
| Softbank Data center or ISP? |
1 | SumGuy | 11:53 am Jun 12, 2025 | ||||
| Referer = www . baidu . com I flag these as bot/VPN |
2 | SumGuy | 4:05 am Jun 5, 2025 | ||||
| GIGENET - bot run wild 63.141.52.0/22 |
3 | Bewenched | 12:17 am Jun 5, 2025 | ||||
| AmazonBot ignoring robots.txt 3.208.0.0/12 |
2 | Bewenched | 10:59 pm Jun 4, 2025 |