https://github.com/tensorflow/datasets/blob/5952d3d60d60e1727786fa7a9a23d24bb463d4d6/tensorflow_datasets/text/c4.py#L101

The data is the output from running this code: https://github.com/tensorflow/datasets/blob/5952d3d60d60e1727786fa7a9a23d24bb463d4d6/tensorflow_datasets/text/c4.py

There are only a few 100 lines there, so it should be easy to get a high-level view of how it all hangs together.

The data is cleaned of HTML and other non-natural language. It is also deduplicated in a not very intelligent way: each paragraph (text separated by a newline) of each example is hashed and any paragraphs of hash collisions are removed.

I don't think "not very intelligent" is the correct description. We had to make some tough decisions to make the dataset reproducible outside of Google within reasonable cost bounds since we were unable to release the data directly. This also drove our use of CLD3 versus more accurate options.

Thanks Adam for the clarification. You're absolutely correct that full deduping can be hugely expensive, and it was a mistake to call the implemented method not intelligent given the reproducibility goals of C4.

The filenames contain the language, as detected by CLD3. You can download just one language with the instructions at #5056.

@dirkgr Thanks for quickly response, in file c4-af.tfrecord-00000-of-00064.json.gz, af means language type, so if I just need the af, use git lfs pull --include "multilingual/c4-af.*.json.gz"

There are a few differences. For the size, I counted only the contents of the "text" field, and I used ja as well as ja-Latn.

I don't know about the tokenizer, since I didn't run those numbers myself, but I would expect something called "ByteBPE" to create a lot of subword tokens, i.e., wordpieces, i.e., multiple tokens per word. That would inflate your token count quite a bit.

Sorry, this is not that kind of dataset. mC4 doesn't contain parallel data (though if you search long enough you might find documents that are translations of other documents).

I like to use a tool called jq to do that kind of thing. But you can also write a short Python script that reads the JSON and writes out the text.

I don't know. It'll be whatever the CLD3 language detector (https://github.com/google/cld3) detects as Portuguese. It's possible it doesn't make the distinction.