Turkish ISO-8859-9 detection support #21

queeup · 2014-01-22T23:08:10Z

I didn't commit README.rst changes. I leave it to developers.

I have no idea about side effects. With this value it works good. More then 0.6..... always detects Turkish(iso-8859-9) as iso-8859-2.

hakanzy · 2014-01-31T13:20:56Z

Merge?

sigmavirus24 · 2014-01-31T14:22:08Z

@hakanzy I assume you're looking for the "Merge" button. If that's the case you need to be a maintainer on the repository. If instead you're giving an obscure direction, all pull requests need a proper amount of review before being accepted. The maintainers of chardet are busy right now and do not have the time to dedicate to review.

dan-blanchard · 2014-09-09T14:24:30Z

So I finally got a chance to look at this, and my initial thought was that things looked fine, but then I looked at the unit test results. We currently have the repo setup so that travis builds always look like they pass (because of known failures), which makes it difficult for people to see when their changes introduce new bugs.

Anyway, in the case of this PR, we go from having 27 unit test failures (see #13), to 29. If you look at the failures:

$ python test.py
.................................FFFFFF...................................................................FF.FFF......................................FF..............................................F.FF.FFFFF..................F..............................................F........................................FF..F...F...............F..F...............................................
======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1254, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/windows-1254-turkish/_chromium_windows-1254_with_no_encoding_specified.html

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected latin1, but got 'ISO-8859-2' in /home/travis/build/chardet/chardet/tests/latin1/_ude_2.txt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected latin1, but got 'TIS-620' in /home/travis/build/chardet/chardet/tests/latin1/_ude_4.txt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected latin1, but got 'ascii' in /home/travis/build/chardet/chardet/tests/latin1/_mozilla_bug638318_text.html

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected latin1, but got 'ISO-8859-2' in /home/travis/build/chardet/chardet/tests/latin1/_ude_3.txt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected latin1, but got 'IBM855' in /home/travis/build/chardet/chardet/tests/latin1/_ude_1.txt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1250, but got 'windows-1255' in /home/travis/build/chardet/chardet/tests/windows-1250-hungarian/bbc.co.uk.hu.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1250, but got 'ISO-8859-7' in /home/travis/build/chardet/chardet/tests/windows-1250-hungarian/objektivhir.hu.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1250, but got 'ISO-8859-2' in /home/travis/build/chardet/chardet/tests/windows-1250-hungarian/bbc.co.uk.hu.forum.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1250, but got 'IBM855' in /home/travis/build/chardet/chardet/tests/windows-1250-hungarian/bbc.co.uk.hu.pressreview.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1250, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/windows-1250-hungarian/bbc.co.uk.hu.learningenglish.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1252, but got 'ISO-8859-2' in /home/travis/build/chardet/chardet/tests/windows-1252/_mozilla_bug421271_text.html

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1252, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/windows-1252/github_bug_9.txt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.fin.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.mrk.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.mrt.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.bus.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.cmm.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.wld.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/naftemporiki.gr.spo.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-7, but got 'windows-1253' in /home/travis/build/chardet/chardet/tests/iso-8859-7-greek/disabled.gr.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-6, but got 'MacCyrillic' in /home/travis/build/chardet/chardet/tests/iso-8859-6-arabic/_chromium_ISO-8859-6_with_no_encoding_specified.html

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected windows-1256, but got 'MacCyrillic' in /home/travis/build/chardet/chardet/tests/windows-1256-arabic/_chromium_windows-1256_with_no_encoding_specified.html

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-2, but got 'ISO-8859-7' in /home/travis/build/chardet/chardet/tests/iso-8859-2-hungarian/escience.hu.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-2, but got 'windows-1251' in /home/travis/build/chardet/chardet/tests/iso-8859-2-hungarian/cigartower.hu.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-2, but got 'KOI8-R' in /home/travis/build/chardet/chardet/tests/iso-8859-2-hungarian/shamalt.uw.hu.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected iso-8859-2, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/iso-8859-2-hungarian/shamalt.uw.hu.mv.xml

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected utf-8, but got 'ISO-8859-2' in /home/travis/build/chardet/chardet/tests/utf-8/bom-utf-8.srt

======================================================================
FAIL: runTest (__main__.TestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 34, in runTest
    self.file_name))
AssertionError: Expected utf-8, but got 'ISO-8859-9' in /home/travis/build/chardet/chardet/tests/utf-8/_mozilla_bug306272_text.html

----------------------------------------------------------------------
Ran 389 tests in 162.607s

FAILED (failures=29)

you can see that with your change we would incorrectly predict ISO-8859-9 all over the place, so I think your threshold is too low.

queeup · 2014-09-12T18:14:45Z

What do you suggest? What do you mean about low threshold?

dan-blanchard · 2014-09-12T18:44:10Z

I believe the mTypicalPositiveRatio you changed in 1059680 is too low now, because it seems to predict Turkish for things that are not Turkish now.

sigmavirus24 · 2014-09-12T19:09:41Z

Yeah this is why I think we need to since with the upstream version of universal chardet from Mozilla. I expect they have updated models and ratios that would be far more accurate.

dan-blanchard · 2014-09-12T19:41:29Z

I completely agree. I'm working on a bunch of SKLL stuff at the moment, but I hope to get a chance to tackle this in the next month or so.

dan-blanchard · 2014-12-02T14:50:12Z

Sorry for the confusion, but because so many people were targeting master instead of develop, I renamed our branches in the following way:

master ➡️ stable
develop ➡️ master

Your pull request was unfortunately the only one to have the proper target of develop, so it got closed because it has a now non-existent target.

If you don't mind, could you please create a new PR with master as the target?

Also, @sigmavirus24, I've looked into the upstream changes a little bit, and there were very few substantial changes that I've found so far (except for them eliminating detection of a number of codecs that we would like to continue to support).

queeup added 4 commits January 22, 2014 19:38

Turkish ISO-8859-9 detection support

d0749bc

Turkish ISO-8859-9 test files

e96db0a

Turkish ISO-8859-9 subtitle test file

29f57de

Reduce mTypicalPositiveRatio value for better Turkish guess

1059680

I have no idea about side effects. With this value it works good. More then 0.6..... always detects Turkish(iso-8859-9) as iso-8859-2.

queeup mentioned this pull request Jan 22, 2014

Turkish (iso-8859-9) detecting. #20

Closed

dan-blanchard closed this Dec 2, 2014

queeup mentioned this pull request Dec 2, 2014

Turkish ISO-8859-9 detection support (New) #41

Merged

caronc mentioned this pull request Feb 2, 2016

Encoding doesn't work. caronc/nzb-subliminal#20

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Turkish ISO-8859-9 detection support #21

Turkish ISO-8859-9 detection support #21

Uh oh!

queeup commented Jan 22, 2014

Uh oh!

hakanzy commented Jan 31, 2014

Uh oh!

sigmavirus24 commented Jan 31, 2014

Uh oh!

dan-blanchard commented Sep 9, 2014

Uh oh!

queeup commented Sep 12, 2014

Uh oh!

dan-blanchard commented Sep 12, 2014

Uh oh!

sigmavirus24 commented Sep 12, 2014

Uh oh!

dan-blanchard commented Sep 12, 2014

Uh oh!

dan-blanchard commented Dec 2, 2014

Uh oh!

Uh oh!

Turkish ISO-8859-9 detection support #21

Turkish ISO-8859-9 detection support #21

Uh oh!

Conversation

queeup commented Jan 22, 2014

Uh oh!

hakanzy commented Jan 31, 2014

Uh oh!

sigmavirus24 commented Jan 31, 2014

Uh oh!

dan-blanchard commented Sep 9, 2014

Uh oh!

queeup commented Sep 12, 2014

Uh oh!

dan-blanchard commented Sep 12, 2014

Uh oh!

sigmavirus24 commented Sep 12, 2014

Uh oh!

dan-blanchard commented Sep 12, 2014

Uh oh!

dan-blanchard commented Dec 2, 2014

Uh oh!

Uh oh!