First of all, I'd like to apologize for my misguided discussion about mTypicalPositiveRatio in the previous PR you had, #21. I was very new to chardet then and didn't fully understand how it worked.

How everything works is actually documented fairly well here.

As you can see in the original Mozilla paper, mTypicalPositiveRatio is the ratio of "the number of occurrences of the 512 most frequently used characters divided by the number of occurrences of the rest of the characters" for a typical document written in the language in question. It is a poorly named variable for sure, but has little to do with the confidence (except that we can determine the confidence based on how far a document's ratio is from what we expect).

Anyway, the ratio has to be calculated from a "typical" Turkish document (and currently we don't have a tool to do the calculation for you).

Could you explain how you came up with all of the tables in your PR? What files did you analyze to determine the ratio?

In the future I would like to have tools available to make it much simpler for people to add new encodings, but it is currently pretty difficult.

The tables, I found it in this project: https://github.com/PyYoshi/cChardet/blob/master/src/ext/libcharsetdetect/mozilla/extensions/universalchardet/src/base/LangTurkishModel.cpp

And convert it to python. I didn't analyze anything for ratio. First I tried same number from that project then I change it because It's not detecting Turkish with that ratio.

Thanks for pointing out that you adapted it from cChardet. It took me until this moment to realize that cChardet is actually wrapping uchardet-enhanced, which is a much-improved version of the original Mozilla code chardet is currently based on.

The ratio used by cChardet is correct; the problem is with the rest of chardet. Our other probers are reporting confidences that are too high. We should switch to treating uchardet-enhanced as the upstream version and pull in all of the changes present there. That would make it even simpler to merge with cChardet as we've discussed before in #19.

OK. I can revert back my ratio to default. Do you think revert back asap? Or wait your call?

I'd say revert it back to the default for now, and then I'll merge this in. Then we'll need to pull in the other updated tables and whatnot from cChardet/uchardet-enhanced.

Reverted. All yours now.

Thanks for the contribution.