Description

• adds scripts to prepare training/evaluation data for a language identification model and then train the langid model
• updates lang identifier class to use the new (v3) floret/fasttext model rather than the existing (v2) thinc/cld3
• adds additional datasets to training data, for more and better language coverage
• updates a couple existing datasets to newer versions
• adds floret (the explosion folks' wrapper around fasttext) as a package dependency

Motivation and Context

I wanted to update my home-brewed language identification code to use something a bit more standard and, ideally, faster / more accurate. I also want to use floret for other purposes, so this use case brings it into textacy for later.

How Has This Been Tested?

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation, and I have updated it accordingly.