Skip to content

Introduce browser client hint detection #6989

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 54 commits into from
Mar 8, 2022
Merged

Introduce browser client hint detection #6989

merged 54 commits into from
Mar 8, 2022

Conversation

sgiehl
Copy link
Member

@sgiehl sgiehl commented Feb 8, 2022

Description:

As discussed in some other issues it might happen in the near future that some devices or clients might freeze the useragents in favor of client hints.

This is a proposal on how we could possibly implement a client hint detection in device detector
It's not yet finished and there are still a couple of things that might need to be discussed and improved.

I did not yet had much time to think about this or implement it, so if someone has a suggestion for a better solution feel free to suggest it here.

@sanchezzzhak Any feedback on that one?

@sanchezzzhak
Copy link
Collaborator

it would be useful to add one property $app
a value that can be filled from
$_SERVER['HTTP_X_REQUESTED_WITH']

image

even anonymous browsers can be identified by ID
example id https://play.google.com/store/apps/details?id=com.MaxSecureBrowse
image
useragent Mozilla/5.0 (Linux; Android 11; M2102J20SG Build/RKQ1.200826.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/90.0.4430.210 Mobile Safari/537.36

sanchezzzhak
sanchezzzhak previously approved these changes Feb 8, 2022
Copy link
Collaborator

@sanchezzzhak sanchezzzhak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it looks good

1 we need to added test for windows 11
#6928 (comment)

2 maybe worth adding to readme: if use client-hints, you need to send the header yourself (before output html)

header('Accept-CH: Sec-CH-UA-Full-Version, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Model, Sec-CH-UA-Arc', true);

@sgiehl
Copy link
Member Author

sgiehl commented Feb 8, 2022

@sanchezzzhak If you have some time free, feel free to push some adjustments & tests to this branch. I'm not yet sure if I will find much time this week to go on with this....

@liviuconcioiu
Copy link
Collaborator

@sanchezzzhak if $_SERVER['HTTP_X_REQUESTED_WITH'] is going to be added, I will suggest to be added also those:

$_SERVER['HTTP_DEVICE_STOCK_UA']
$_SERVER['HTTP_X_OPERAMINI_PHONE_UA']
$_SERVER['HTTP_X_UCBROWSER_UA']

Examples:

$_SERVER['HTTP_USER_AGENT'] $_SERVER['HTTP_DEVICE_STOCK_UA'] $_SERVER['HTTP_X_OPERAMINI_PHONE_UA']
Opera/9.80 (MAUI Runtime; Opera Mini/4.4.39014/191.265; U; hu) Presto/2.12.423 Version/12.16 OperaMini(Nokia 210FW40.01.11;Opera Mini/4.4.39014;hu) OperaMini(Nokia 210FW40.01.11;Opera Mini/4.4.39014;hu)
Opera/9.80 (MAUI Runtime; Opera Mini/4.4.33576/191.265; U; en) Presto/2.12.423 Version/12.16 OperaMini(MAUI_MRE;Opera Mini/4.4.33576;en) OperaMini(MAUI_MRE;Opera Mini/4.4.33576;en)
Opera/9.80 (MAUI Runtime; Opera Mini/4.4.33576/191.265; U; en) Presto/2.12.423 Version/12.16 OperaMini(Fucus/Unknown;Opera Mini/4.4.33576;en) OperaMini(Fucus/Unknown;Opera Mini/4.4.33576;en)
Opera/9.80 (J2ME/MIDP; Opera Mini/4.5.40312/191.265; U; nl) Presto/2.12.423 Version/12.16 Nokia301.1/2.0 (08.02) Profile/MIDP-2.1 Configuration/CLDC-1.1 Nokia301.1/2.0 (08.02) Profile/MIDP-2.1 Configuration/CLDC-1.1
Opera/9.80 (J2ME/MIDP; Opera Mini/4.5.40312/191.256; U; es) Presto/2.12.423 Version/12.16 Nokia302/5.0 (14.78) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 AppleWebKit/420+ (KHTML, like Gecko) Safari/420+ Nokia302/5.0 (14.78) Profile/MIDP-2.1 Configuration/CLDC-1.1 Mozilla/5.0 AppleWebKit/420+ (KHTML, like Gecko) Safari/420+
Opera/9.80 (SpreadTrum; Opera Mini/4.4.33961/191.265; U; en) Presto/2.12.423 Version/12.16 NativeOperaMini(Spreadtrum/HW Version: fp6531_bar;Native Opera Mini/4.4.33961;en) NativeOperaMini(Spreadtrum/HW Version: fp6531_bar;Native Opera Mini/4.4.33961;en)
Opera/9.80 (J2ME/MIDP; Opera Mini/4.1.14287/191.256; U; en) Presto/2.12.423 Version/12.16 Nokia2330c-2/2.0 (08.20) Profile/MIDP-2.1 Configuration/CLDC-1.1 Nokia2330c-2/2.0 (08.20) Profile/MIDP-2.1 Configuration/CLDC-1.1
$_SERVER['HTTP_USER_AGENT'] $_SERVER['HTTP_X_UCBROWSER_UA']
Mozilla/5.0 (Linux; U; Android 10; en-US; VOG-L29 Build/HUAWEIVOG-L29) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/78.0.3904.108 UCBrowser/13.4.0.1306 Mobile Safari/537.36 dv(VOG-L29);pr(UCBrowser/13.4.0.1306);ov(Android 10);ss(360715);pi(10802147);bt(GJ);pm(1);bv(1);nm(0);im(0);sr(2);nt(1);bi(355);fi(145);dn(32210210557-ff0aa3dd);ai(720089404747345);

@sanchezzzhak
Copy link
Collaborator

sanchezzzhak commented Feb 9, 2022

$_SERVER['HTTP_X_UCBROWSER_UA']
it looks interesting, it will be possible to make some hints of the screen size.
baidu, instagram - also contain something similar in useragent string

@sgiehl
Copy link
Member Author

sgiehl commented Feb 9, 2022

@sanchezzzhak I wouldn't start adding detection for screen sizes or similar for now. Guess there are also client hints for such stuff. Let's focus on bringing the client hints detection for the already detected stuff on the road. We can also consider parsing other header fields later...

@sanchezzzhak sanchezzzhak linked an issue Feb 16, 2022 that may be closed by this pull request
@sgiehl
Copy link
Member Author

sgiehl commented Feb 25, 2022

@sanchezzzhak I have added improved some more code. Now also the browser should be detected with client hints if possible.
Maybe you can have a quick look if everything makes sense so far. If you have some more real test data for client hints, feel free to extend the clienthints.yml fixture.

@liviuconcioiu
Copy link
Collaborator

Can you try the code exactly as it is here https://github.com/matomo-org/device-detector#usage and addheader('Accept-CH: Sec-CH-UA-Full-Version, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Model, Sec-CH-UA-Arc', true); at the top?

It should work.

@Nevercold
Copy link
Contributor

Nevercold commented Mar 10, 2022

<?php

header('Accept-CH: Sec-CH-UA-Full-Version, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version, Sec-CH-UA-Model, Sec-CH-UA-Arc', true);

use DeviceDetector\ClientHints;
use DeviceDetector\DeviceDetector;
use DeviceDetector\Parser\Device\AbstractDeviceParser;

require __DIR__ . "/../../vendor/autoload.php";

#AbstractDeviceParser::setVersionTruncation(AbstractDeviceParser::VERSION_TRUNCATION_NONE);

$userAgent = $_SERVER['HTTP_USER_AGENT'];
$clientHints = ClientHints::factory($_SERVER);
$dd = new DeviceDetector($userAgent, $clientHints);

$dd->parse();
$osInfo = $dd->getOs();
print_r($osInfo);


die();

Output: Array ( [name] => Windows [short_name] => WIN [version] => 10 [platform] => x64 [family] => Windows )
But why D:?

@Findus23
Copy link
Collaborator

I'm pretty sure that's the same issue I also had when updating devicedetector.lw1.at:

DeviceDetector\ClientHints::factory looks at the $_SERVER variables, but they are named differently on my setup (nginx) than what the factory expects. Therefore it doesn't detect any of the client hints (you can also see it above in

  "clientHints": {
    "architecture": "",
    "bitness": "",
    "mobile": false,
    "model": "",
    "platform": "",
    "platformVersion": "",
    "uaFullVersion": "",
    "fullVersionList": {}
  },
```)

@Nevercold
Copy link
Contributor

Nevercold commented Mar 10, 2022

Interesting.
So as web server I use Apache2 with PHP8.1-FPM
Maybe this is the reason.

(And Cloudflare with Proxy)

@Findus23
Copy link
Collaborator

I assume that's FPM-related.
I'll look into fixing that.

@liviuconcioiu
Copy link
Collaborator

liviuconcioiu commented Mar 10, 2022

Sometimes, the platform version is not sent on the first request, it needs a refresh. At least this is how Chrome behaves for example.

chrome

@Nevercold
Copy link
Contributor

I assume that's FPM-related. I'll look into fixing that.

i have disabled fpm, but it is still the same.

@Nevercold
Copy link
Contributor

and the headers are set, so very weird.

@Findus23
Copy link
Collaborator

It turns out @sgiehl was faster and already fixed the issue yesterdays:
eae6791

So make sure you are on the latest master branch.

And when in doubt do a var_dump($clientHints); and compare it with a var_dump($_SERVER); to see if the headers are correctly detected.

@sanchezzzhak
Copy link
Collaborator

example with the last commit dev-master
https://phpsandbox.io/e/x/mfnke?layout=EditorPreview&defaultPath=%2F&theme=dark&showExplorer=no&openedFiles=
I don't have windows11 to check

@Nevercold
Copy link
Contributor

It turns out @sgiehl was faster and already fixed the issue yesterdays: eae6791

So make sure you are on the latest master branch.

And when in doubt do a var_dump($clientHints); and compare it with a var_dump($_SERVER); to see if the headers are correctly detected.

It is recognized correctly, but still only gives out windows 10.
the current commit is pulled (the latest specified and saved in Composer.json )

object(DeviceDetector\ClientHints)#3 (8) {
--
  | ["architecture":protected]=>
  | string(0) ""
  | ["bitness":protected]=>
  | string(0) ""
  | ["mobile":protected]=>
  | bool(false)
  | ["model":protected]=>
  | string(2) """"
  | ["platform":protected]=>
  | string(9) ""Windows""
  | ["platformVersion":protected]=>
  | string(8) ""14.0.0""
  | ["uaFullVersion":protected]=>
  | string(14) ""99.0.4844.51""
  | ["fullVersionList":protected]=>
  | array(3) {
  | [0]=>
  | array(2) {
  | ["brand"]=>
  | string(12) " Not A;Brand"
  | ["version"]=>
  | string(2) "99"
  | }
  | [1]=>
  | array(2) {
  | ["brand"]=>
  | string(8) "Chromium"
  | ["version"]=>
  | string(2) "99"
  | }
  | [2]=>
  | array(2) {
  | ["brand"]=>
  | string(13) "Google Chrome"
  | ["version"]=>
  | string(2) "99"
  | }
  | }
  | }

@Nevercold
Copy link
Contributor

example with the last commit dev-master https://phpsandbox.io/e/x/mfnke?layout=EditorPreview&defaultPath=%2F&theme=dark&showExplorer=no&openedFiles= I don't have windows11 to check

same.
image

@liviuconcioiu
Copy link
Collaborator

liviuconcioiu commented Mar 10, 2022

@Nevercold Can you refresh phpsandbox page? Chrome sends the platform version on a second request. So this is the issue.

@Nevercold
Copy link
Contributor

@Nevercold Can you refresh phpsandbox page? Chrome sends the platform version on a second request. So this is the issue.

I already have a few times.
Its the same.

@liviuconcioiu
Copy link
Collaborator

On first request:

first request

On second request (page refresh)

second request

@Nevercold
Copy link
Contributor

Nevercold commented Mar 10, 2022

But also funny that it first says x64 and then x86.
Oh i see, first, he dont get the header.

@liviuconcioiu
Copy link
Collaborator

But also funny that it first says x64 and then x86.

@sgiehl Can you check this when you have time? It should get the platform from bitness.

@Nevercold
Copy link
Contributor

and in general, it does not work properly either. xD

@liviuconcioiu
Copy link
Collaborator

and in general, it does not work properly either. xD

This is a browser issue. Headers aren't sent by the browser on the first request. Probably to avoid fingerprinting or cloaking on the first attempt.

@sv3t0sl4v
Copy link
Contributor

and in general, it does not work properly either. xD

This is a browser issue. Headers aren't sent by the browser on the first request. Probably to avoid fingerprinting or cloaking on the first attempt.

Tricky! Private browsing or in general?

@Nevercold
Copy link
Contributor

and in general, it does not work properly either. xD

This is a browser issue. Headers aren't sent by the browser on the first request. Probably to avoid fingerprinting or cloaking on the first attempt.

But I have reloaded it many times, so it should be there then.
The version he does get. (14.0.0.)

@sanchezzzhak
Copy link
Collaborator

additional quotes are present, most likely, this does not allow determine the version

need fix

@Nevercold
Copy link
Contributor

Nevercold commented Mar 10, 2022

true, it takes "14.0.0" as version.
outputs at least $clientHints.

@Nevercold Nevercold mentioned this pull request Mar 10, 2022
11 tasks
@Nevercold
Copy link
Contributor

Fixed. #7050

Works now for me.

@sgiehl
Copy link
Member Author

sgiehl commented Mar 10, 2022

Client Hints are actually designed in a way that by default only some basic data (Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform) is provided in the request headers. If the server requests more details by sending a header to the client the next request will contain all information. So the first request can never detect windows 11 actually, as the platform version isn't available.

@Nevercold
Copy link
Contributor

Client Hints are actually designed in a way that by default only some basic data (Sec-CH-UA, Sec-CH-UA-Mobile, Sec-CH-UA-Platform) is provided in the request headers. If the server requests more details by sending a header to the client the next request will contain all information. So the first request can never detect windows 11 actually, as the platform version isn't available.

Thanks for the explanation :)

@nunoperalta
Copy link

Hey - Thanks very much for this.
Just realized this isn't included in 5.0.5

Any estimate date of when will the next version be released? :)
Thank you very much.

@sgiehl
Copy link
Member Author

sgiehl commented Mar 30, 2022

We are currently preparing the implementation in Matomo to see if any further changes are required here. Once that is done we will release a new major version of device detector. Guess within one or two weeks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Detecting Windows 11?
8 participants