Skip to content

List of possible bots to check #6315

@sgiehl

Description

@sgiehl

List of useragents from matomo-org/matomo-log-analytics#239 that are currently not detected as bot (or library):

  • Hatena-Favicon2 (http://www.hatena.ne.jp/faq/)
  • Mozilla/5.0 (X11; Linux x86_64; rv:20.0; Favicon; +https://github.com/ArthurHoaro/favicon) Gecko/20100101 Firefox/32.0
  • com.ddeville.llwebkit.favicon/158 CFNetwork/902.1 Darwin/17.7.0 (x86_64)
  • com.ddeville.llwebkit.favicon/158 CFNetwork/974.1 Darwin/18.0.0 (x86_64)
  • Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.170 Safari/537.36 Thumbshots/10.0.0.0
  • Thumbor/6.4.2
  • rethumb/v1 (http://rethumb.com)
  • FetchStream
  • Hatena::Fetcher/0.01 (master) Furl/3.13
  • Hatena::Fetcher/0.01 (master) Furl/3.06
  • safarifetcherd/604.1 CFNetwork/974.2.1 Darwin/18.0.0
  • safarifetcherd/604.1 CFNetwork/975.0.3 Darwin/18.2.0
  • safarifetcherd/604.1 CFNetwork/901.1 Darwin/17.6.0
  • safarifetcherd/604.1 CFNetwork/976 Darwin/18.2.0
  • safarifetcherd/604.1 CFNetwork/958.1 Darwin/18.0.0
  • HatenaBookmark/0.03 (compatible; entryimage-fetcher)
  • Mozilla/5.0 (compatible; ImageFetcher/7.0; +http://images.weserv.nl/)
  • Mozilla/5.0 (compatible; ImageFetcher/8.0; +http://images.weserv.nl/)
  • Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; BacklinkHttpStatus)
  • Hatena Star UserAgent/2
  • Hatena-Favicon2 (http://www.hatena.ne.jp/faq/)
  • Hatena::Fetcher/0.01 (master) Furl/3.13
  • Hatena::Fetcher/0.01 (master) Furl/3.06
  • HatenaBookmark/0.03 (Hatena::Bookmark; master;) Furl/3.13
  • HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)
  • Hatena::Scissors/0.01
  • HatenaBookmark/4.0 (Hatena::Bookmark; Scissors)
  • HatenaBookmark/0.03 (compatible; entryimage-fetcher)
  • Elytra/0.10.0 (Macintosh; Ubuntu/14.06) GCDHTTPRequest
  • Mozilla/4.0 (compatible; Win32; WinHttp.WinHttpRequest.5)
  • request.js
  • Zend_Http_Client
  • EventMachine HttpClient
  • HTTPClient/1.0 (2.6.0.1, ruby 2.0.0 (2015-12-16))
  • HTTPClient/1.0 (2.8.3, ruby 2.2.1 (2015-02-26))
  • Jakarta Commons-HttpClient/3.1
  • Mozilla/5.0 (compatible; Funnelback) RPT-HTTPClient/0.3-3E
  • AppEngine-Google; (+http://code.google.com/appengine; appid: s~readability-api-hrd)
  • AppEngine-Google; (+http://code.google.com/appengine; appid: e~finscience-1253)
  • AppEngine-Google; (+http://code.google.com/appengine; appid: s~cdn-dinoia)
  • GAE AppEngine-Google; (+http://code.google.com/appengine; appid: s~ga-mozilla-org-prod-001)
  • Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~mendoapp1)
  • Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~vodio-app)
  • Mozilla/5.0 AppEngine-Google; (+http://code.google.com/appengine; appid: e~finscience-1253)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~xiaohe18675)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~thgntk2)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~kindle-11)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedkin)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: s~ebook-rexdf)
  • Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
  • Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~simple-rss-proxy)
  • Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729) AppEngine-Google; (+http://code.google.com/appengine; appid: s~theajaxpost)
  • Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~twilinks123)
  • Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0) AppEngine-Google; (+http://code.google.com/appengine; appid: b~thatkindleear)
  • Evergreen (macOS; RSS Reader; https://ranchero.com/evergreen/)
  • Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/602.3.12 (KHTML, like Gecko) NetNewsWire/3.3.2
  • Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/534.57.7 (KHTML, like Gecko) NetNewsWire/3.3.2
  • NetNewsWire (macOS; RSS Reader; https://ranchero.com/netnewswire/)
  • NetNewsWire/3.3.2 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)
  • NetNewsWire/4.1.0 (Mac OS X; http://netnewswireapp.com/mac/; gzip-happy)
  • Evergreen (macOS; RSS Reader; https://ranchero.com/evergreen/)
  • Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36 AppEngine-Google; ( http://code.google.com/appengine; appid: s~simple-rss-proxy)
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) QuiteRss/0.18.12 Safari/538.1
  • NetNewsWire (macOS; RSS Reader; https://ranchero.com/netnewswire/)
  • rss/10 CFNetwork/976 Darwin/18.2.0
  • rss/5 CFNetwork/897.15 Darwin/17.5.0
  • rss/5 CFNetwork/902.2 Darwin/17.7.0
  • rss/5 CFNetwork/974.2.1 Darwin/17.7.0
  • rss/8 CFNetwork/974.2.1 Darwin/18.0.0
  • rss/8 CFNetwork/975.0.3 Darwin/17.7.0
  • ruby:net.bhaak.rss2html:1.0
  • Visor News Reader RSS/1.0 (admin@visorco.com)
  • Winds: Open Source RSS & Podcast app: https://getstream.io/winds/
  • RSSOwl/2.2.1.201312301314 (Windows; U; en)
  • Ruby
  • Ruby, Twurly v1.1 (https://twurly.org)
  • ruby:net.bhaak.rss2html:1.0
  • HTTPClient/1.0 (2.6.0.1, ruby 2.0.0 (2015-12-16))
  • HTTPClient/1.0 (2.8.3, ruby 2.2.1 (2015-02-26))
  • Mozilla/5.0 (Mixnode) AppleWebKit/537.36 (KHTML, like Gecko)
  • node-superagent/3.8.2
  • node-superagent/2.3.0
  • node-superagent/0.18.2
  • node.io
  • node.js
  • Node.js (linux; U; rv:v6.9.1) AppleWebKit/537.36 (KHTML, like Gecko)
  • http.rb/4.0.0
  • http.rb/3.3.0
  • http.rb/3.0.0
  • Mozilla/5.0 (compatible; adscanner/)
  • Mozilla/5.0 (compatible; DNS SSL/TLS HTTP HTML Website Security Scanner/0.2 beta; +https://www.htmlyse.com/)
  • httpscheck (unknown version) CFNetwork/897.15 Darwin/17.5.0 (x86_64)
  • httpscheck (unknown version) CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)
  • httpscheck (unknown version) CFNetwork/976 Darwin/18.2.0 (x86_64)
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 (compatible; linkCheckV3.0)
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) wkhtmltoimage Safari/538.1
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltopdf Safari/534.34
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) wkhtmltoimage Version/7.0 Safari/538.1
  • Mozilla/5.0 (X11; BSD Four) AppleWebKit/534.34 (KHTML, like Gecko) wkhtmltoimage Safari/534.34
  • Houdini%20Sweeper/3000 CFNetwork/893.13.1 Darwin/17.4.0 (x86_64)
  • Houdini%20Sweeper/3000 CFNetwork/897.15 Darwin/17.5.0 (x86_64)
  • GoScraper
  • MetadataScraper
  • Embed PHP Library
  • Embed PHP library
  • SnowHaze Search/1.0 support@snowhaze.com
  • WordupInfoSearch/1.0
  • WordupinfoSearch/1.0
  • Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36 http://notifyninja.com/monitoring
  • Iframely/0.8.5 (+http://iframely.com/;)
  • Iframely/1.2.2 (+http://iframely.com/;)
  • Iframely/1.2.5 (+http://iframely.com/;)
  • Iframely/1.2.7 (+http://iframely.com/;)
  • Iframely/1.0.4 (+http://iframely.com/;)
  • Iframely/1.2.7 (+https://iframely.com/;)
  • G-i-g-a-b-o-t
  • B-l-i-t-z-B-O-T/6.3 (Ubuntu 7.0; zh_TW;)
  • B-l-i-t-z-B-O-T/4.3 (AmigaOS 2.3; fr_LU;)
  • B-l-i-t-z-B-O-T/1.9 (Windows Vista 5.7; fr_CA;)
  • inbound.li parser
  • Mozilla/5.0 (compatible; WebDataStats/1.0 ; https://webdatastats.com/policy.html)
  • NetTrack Anonymous Web Statistics https://nettrack.info/support.php
  • CakePHP
  • Bots that Matomo catches in the server (but not the log analyzer)
  • Mozilla/5.0 (compatible; GoogleDocs; documents; +http://docs.google.com)
  • Mozilla/5.0 (compatible; GoogleDocs; +http://docs.google.com; +Google-Document-Conversion)
  • Mozilla/5.0 (compatible; GoogleDocs; apps-presentations; +http://docs.google.com)

Note: have not yet checked which of this useragents are actually bots or maybe valid mobile apps or other stuff....

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions