Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.74%. Comparing base (f380b1a) to head (43e4aba).
Report is 19 commits behind head on main.

@@           Coverage Diff            @@
##             main    #9173    +/-   ##
========================================
  Coverage   88.74%   88.74%            
========================================
  Files         321      320     -1     
  Lines       41853    41649   -204     
========================================
- Hits        37144    36963   -181     
+ Misses       4709     4686    -23     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

It maybe helpful to later add the option of using Meng Sheng Pinyin fonts or Hanzi Pinyin fonts as romanization to ascii can result in information loss for many tonal languages, as an example:

• "mother" (mā, 妈)
• "ant" (má, 蚂)
• "horse" (mǎ, 马)
• "scold" (mà, 骂)

There are libraries that will also do this, for example xpinyin. Support for other languages could be added as need arises.

Hiya @bkmgit , could you create a new issue with your comment? That greatly expands the scope of this rather simple approach.

Hit wrong GH button, re-opening

According to https://www.lookuptables.com/text/extended-ascii-table, it looks like all the accented characters are in the decimal range 128-154 in case we want to make an exception for them.

thanks for the review.

If the language is known, pyicu has options for transliteration, there is an example in the cheatsheet.

However, it maybe easier to do an NFC decomposition of each character and check if it contains an ascii letter, if all the NFC decompositions contain ascii characters, keep the name, otherwise use the ascii name. This could be done using unicodedata.

Ideally each person would be able to update this field since the readme of Unidecode indicates there will be many corner cases that will be difficult to cover with existing software.

Thanks for the insights, Benson, I had looked briefly and more naively at the unicodedata module. I agree it will be useful. I think Rich's test as implemented will inform us a lot as to where we get bitten by pointless extra text in practice.

(And, perfect being enemy of the good and all, merging this will fix the entirely non-Latin text cases that inspired the issue this addresses; follow-ups that deal with additional subtleties will be welcome)