Now that #17497 has been done, there is another issue that most bots, especially the important Google and Bing bots, fail to call the tracker PHP endpoints matomo.php/piwik.php, while generally robots.txt and headers do allow it and some bots are successfully tracked.

Matomo, PHP and webserver do not report any errors, so the request seems to fail right at the crawler, probably related to a too long query string or invalid characters? Example URL shown by Google mobile-friendly test:

https://our.domain.com/matomo/piwik.php?action_name=<site_title>&idsite=1&rec=1&r=525912&h=7&m=3&s=1&url=<site_url>%2F&_id=d0bed9fe69afde68&_idn=1&_refts=0&send_image=0&cookie=1&res=412x732&pv_id=DAfyXy&pf_net=0&pf_srv=2&pf_tfr=0&pf_dm1=20

Further details:

• https://forum.matomo.org/t/google-and-bing-crawlers-fail-to-call-the-tracker-endpoint/41760?u=michaing
• Create robots.txt to prevent bots from indexing Matomo app #16795 (comment)

Expected Behavior

All bots that crawl the website are tracked and visible in Matomo.

Current Behavior

Many bots fail to call the tracker PHP endpoints and hence do not appear in Matomo.

Possible Solution

None found so far.

Steps to Reproduce (for Bugs)

1. Upgrade to latest Matomo v4.3 release candidate
2. Install BotTracker app
3. Watch AhrefsBot, Baiduspider, YandexBot and Wget being tracked
4. No other bot is tracked, despite respective accesses.
5. For Google crawler, the mobile-friendly test test reveals that the crawler is trying to perform the correct request (URL) but fails to do so with "Other error". Manually calling the same URL succeeds and shows a related Matomo visit/page view.

Context

The behaviour of search engine crawlers, which pages they crawl, how often etc, can be important to detect issues, optimise crawler settings/schedules, etc to balance server load and optimise SEO.

Your Environment

• Matomo Version: 4.3.0-rc2
• PHP Version: 8.0.5
• Server Operating System: Debian Bullseye
• Additionally installed plugins:

API, Actions, Annotations, BotTracker 2.01, BulkTracking, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CustomJsTracker, DBStats, DarkTheme 1.1.6, Dashboard, DevicePlugins, DevicesDetection, Diagnostics, Goals, Heartbeat, ImageGraph, Insights, Installation, Intl, LanguagesManager, Live, LogViewer 4.0.1, Login, Marketplace, Monolog, Morpheus, Overlay, PagePerformance, PrivacyManager, Proxy, Referrers, Resolution, SEO, SegmentEditor, SitesManager, Transitions, UserLanguage, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable