moc3 and caff: add decoder #747

Ronsor · 2023-08-18T03:20:27Z

This PR adds support for the proprietary MOC3 and CAFF formats used by the Live2D Cubism software (https://live2d.com/en/).

Notes:

MOC3 is a format for 2D rigged puppets, somewhat like Flash.
CAFF is an archive format usually found with the extensions .cmo3 and .can3.
No test data is included since there isn't any that I'm allowed to redistribute but can be found at https://www.live2d.com/en/download/sample-data/.
The MOC3 decoder is based on the specification I wrote: https://github.com/OpenL2D/moc3ingbird/blob/master/src/moc3.hexpat

wader

Had a quick look and think it looks great overall 👍 had some minor comments. I will take a deeper look and try the decoder myself later today.

BTW do you know if there are MIT licenses etc model files somewhere we can use for tests? or could we alternatively create some simple models in live2d and save and use those? the reason i really want test files is to be able to refactors and do changes with some kind of insurance to not break or change things in a bad way.

wader · 2023-08-18T07:25:43Z

format/caff/caff.go

+	obfsU64 := func(d *decode.D) uint64 { return d.U64() ^ (obfsKey<<32 | obfsKey) }
+	obfsBool := func(d *decode.D) bool { return obfsU8(d) != 0 }
+
+	// "Big Endian Base 128" - LEB128's strange sibling


😄 i guess strange enough to not include in *decode.D?

It could be, but dealing with the obfuscation makes it awkward and I don't know of any other format that uses it.

agree, lets keep it here

wader · 2023-08-18T07:27:37Z

format/moc3/moc3.go

+	moc3Version3_00_00: {Sym: "3_00_00", Description: "3.0.00 - 3.2.07"},
+	moc3Version3_03_00: {Sym: "3_03_00", Description: "3.3.00 - 3.3.03"},
+	moc3Version4_00_00: {Sym: "4_00_00", Description: "4.0.00 - 4.1.05"},
+	moc3Version4_02_00: {Sym: "4_02_00", Description: "4.2.00 - 4.2.02"},


If you want sym can be a number also but if it's major.patch.minor etc i guess it make no sense... maybe could be struct with "value" ints (d.FieldValue*) but this is probably fine also

I mainly chose the V3_00_00, etc. strings to match the enum from the spec.

wader · 2023-08-18T07:32:07Z

format/moc3/moc3.go

+	if !isBigEndian {
+		d.Endian = decode.LittleEndian
+	}
+	d.SeekRel(58 * 8)


Does this cause "unknown" fields? one way i've done is to explicitly add something like "unused" fields for bit ranges that i known to be unused. "unknown" fq adds automatically for ranges no field as "claimed"

format/caff/caff.go

wader · 2023-08-18T10:59:44Z

The i386 failure is just about test for probe order and help text is failing. Should be fixable with WRITE_ACTUAL=1 go test ./format and commit the changes to those tests

Richardn2002 · 2023-08-18T13:52:22Z

https://github.com/RavioliMavioli/archlive2d
This is a repo containing one moc3 file under the CC0 1.0 license.

https://github.com/shuhaiwen/live2dw
This is a repo containing tons of live2d models but:

The author does not specify the sources clearly, nor specify a license.
They are moc instead of moc3 files.

And I have to admit that I am not involved in the development of this PR, nor have experiences in the moc3 format, which means I do not know if these models are enough for testing.

-- EDIT --

https://github.com/imuncle/live2d
Tons of moc3 files under live2d_3/model/Azue Lane(JP). Still, the license for this repo is questionable, and I suspect these models are Azure Lane game properties.

wader · 2023-08-18T13:57:26Z

@Richardn2002 Hey! thanks that is good pointers. Ideally for tests some kind of simplest possible files that still use lots of features is best.

Maybe easiest is to download live2d, run it and save some simple files?

Richardn2002 · 2023-08-18T14:02:31Z

Yeah, would be nice to use Live2D with all functionalities possible to create a test model. I sadly have not purchased Live2D Cubism. I hope @Ronsor has one copy.

wader · 2023-08-18T14:05:03Z

Yeap would be great, but i see there is a free for 42 days version 🤔

…eserved space

Ronsor · 2023-08-18T14:55:40Z

https://github.com/shuhaiwen/live2dw This is a repo containing tons of live2d models but:

The author does not specify the sources clearly, nor specify a license.

They are moc instead of moc3 files.

Besides the old format, they seem to be taken from Live2D's previous sample downloads page.

…can't be decoded as a format

…lements

The obfuscation key is actually a signed integer, and thanks to two's complement arithmetic and sign extension, this is actually significant.

Ronsor · 2023-08-19T03:04:59Z

I made some improvements:

Included CAFF test data (test.cmo3)
Fixed bug with CAFF deobfuscation
Updated the MOC3 specification to add the new fields for version 5 (released a couple days ago).
Updated the MOC3 decoder to reflect this.
Included MOC3 test data (based on the Arch Chan MOC3, converted to version 5)
Improved generated tree structure for MOC3 with better names for array elements and grouping section fields together regardless of version

wader

Nice work on this, already looked good and gets better, have a look at my comments.

Would be great to cut down a bit on gap fields, make into explicit unknown/reserved/align/padding and whatnot is possible.

And great with test files, will make future maintenance much easier. Are they ok licensewise? test files are not shipped in the binary but i guess its good to keep the source distribution ok licensewise also.

wader · 2023-08-19T07:59:20Z

format/caff/caff.go

+	d.FieldStruct("archive_version", decodeVersion)
+	d.FieldUTF8("format_id", 4)
+	d.FieldStruct("format_version", decodeVersion)
+	obfsKey = d.FieldS32("obfuscate_key")


If signness is not important i would probably do d.FieldU32("obfuscate_key", scalar.UintHex) to display it as hex. Have look if there are more fields that would be better as hex like addresses, checksums etc

I found out the hard way that it needs to be signed otherwise obfsU64() can return bad data.

Aha i see 👍

wader · 2023-08-19T08:02:12Z

format/caff/caff.go

+	d.FieldStruct("preview_image", func(d *decode.D) {
+		d.FieldU8("image_format", imageFormatNames)
+		d.FieldU8("color_type", colorTypeNames)
+		d.SeekRel(2 * 8)


Could this and some of the other d.SeekRel be a unused/alignment/padding etc field? i usually try to have no or as few gaps as possibly but i understand it might hard when reversing a format.

BTW when i was talking about "unknown" fields earlier i was confused, they called "gap" fields now, was renamed some time ago so that decoders can use "unknown".

I'll change those to unusedN fields since I know they're unused here.

For moc3, I should probably handle the section alignment (seems to be 64 bytes).

wader · 2023-08-19T08:14:58Z

format/caff/caff.go

+					}
+				}
+
+				br := bitio.NewBitReader(rawBytes, -1)


Sorry for confusing APIs for doing this atm. Hopefully in the future there could be some helpers for dealing with compressed/massaged data

wader · 2023-08-19T08:22:33Z

format/moc3/moc3.go

+func decodeMOC3(d *decode.D) any {
+	d.FieldUTF8("magic", 4, d.StrAssert("MOC3"))
+	version := d.FieldU8("version", moc3VersionNames)
+	isBigEndian := d.FieldBoolFn("is_big_endian", func(d *decode.D) bool { return d.U8() != 0 })


Maybe want to use sym map to a bool? i usually try to make actual the decoded value and then sym the interpretation of the value. That way a user has access to both (toactual/tosym, default is sym valie if set, otherwise actual)

wader · 2023-08-19T08:35:01Z

format/moc3/moc3.go

@@ -0,0 +1,1766 @@
+package moc3
+
+// https://github.com/OpenL2D/moc3ingbird/blob/master/src/moc3.hexpat


👍 interesting to see two implementations, are the comparable completnesswise? and as someone who has worked with both, how do they compare? the go code feels a bit more verbose?

The Go code is largely autogenerated from the original implementation and then edited. They're at feature parity.

Some of the verbosity of the go decoder is probably due to some code duplication I can fix.

wader · 2023-08-19T08:38:59Z

format/moc3/moc3.go

+			sectionOffsets.countInfo = int64(d.FieldU32("count_info"))
+			sectionOffsets.canvasInfo = int64(d.FieldU32("canvas_info"))
+
+			d.FieldStruct("parts", func(d *decode.D) {


This code makes me think that maybe the decode API should have something similar to Read in encoding/binary hmm

FieldUnmarshalStruct(field string, out any)?

Yeap something like that. Auto camel_case names and maybe use struct tags to support some casting. Something for the future, no need for this PR.

Ronsor · 2023-08-19T13:26:53Z

Are they ok licensewise?

The MOC3 test is CC0 data, the CAFF test is my own work. Licensing should be no problem.

wader · 2023-08-19T14:33:29Z

Are they ok licensewise?

The MOC3 test is CC0 data, the CAFF test is my own work. Licensing should be no problem.

Great!

Ronsor · 2023-08-19T14:34:07Z

@wader What do you think is the best way to add alignment for arrays (at the end, not per field)?

wader · 2023-08-19T14:41:56Z

format/moc3/moc3.go

+				for i := int64(0); i < countInfo.parts; i++ {
+					d.FieldUTF8NullFixedLen("id", 64)
+				}
+			})


@Ronsor you mean alignment here for example? easiest is probably just an extra "ids_alignment" etc field after, other maybe more correct but less user friendly is to make "ids" into struct with "entries" and "alignment" i think

Yes, there. I'll try both ways to see which is more usable.

…t decoding

Ronsor · 2023-08-20T04:06:02Z

I settled on making each section a struct with both array and padding fields, and I refactored the code to be a lot less repetitive. Also fixed an issue with decoding version 5 files (the correction is reflected in the specification).

wader · 2023-08-20T16:03:07Z

LGTM! Think i'm ready to merge this once CI is green. Do you have something more you want to change or add?

The CI failure seems to be just some field that has been renamed

Ronsor · 2023-08-20T16:58:10Z

Whoops, forgot to update the test. I think it's ready to be merged now. 🥳

wader · 2023-08-20T17:10:00Z

Thanks! Great work and hope you find fq usefull! any plans for future contributions?

Ronsor · 2023-08-20T18:07:00Z

Right now there's about three other proprietary formats I plan to add decoders for in the near future.

wader · 2023-08-20T20:49:32Z

Right now there's about three other proprietary formats I plan to add decoders for in the near future.

Interesting! looking forward

BTW forgot to say that you can add documentation and authorship to a decoder by doing something like https://github.com/wader/fq/blob/master/format/apple/macho/macho.md and https://github.com/wader/fq/blob/master/format/apple/macho/macho.go#L17-L28 and it will show up in cli help and make doc will generate documentation.

Ronsor added 4 commits August 17, 2023 13:17

caff: initial implementation

da41a8d

caff: remove dead code

29084e3

moc3: initial implementation

092662e

moc3: remove dead code

20f02e7

wader reviewed Aug 18, 2023

View reviewed changes

Ronsor added 3 commits August 18, 2023 07:40

moc3: update certain array element names, explicitly mark unused or r…

6d10a25

…eserved space

caff: minor formatting changes

23e660f

moc3, caff: update tests and README

441fcd0

Ronsor added 8 commits August 18, 2023 09:25

caff: include uncompressed bits for proper decompressed entries that …

6a3fecd

…can't be decoded as a format

caff: update doc/formats.md

cc58c4b

moc3: nicer tree structure, use more meaningful names for array elements

3caf34e

moc3: consistency - scales array contains value elements, not scale e…

ce40fd1

…lements

moc3: add support for version 5

03ba71b

caff: obfuscation key is a signed integer, add test data

fa11572

The obfuscation key is actually a signed integer, and thanks to two's complement arithmetic and sign extension, this is actually significant.

moc3: add test data for new version 5

d3073c6

caff: run go fmt

4dd0f6d

wader reviewed Aug 19, 2023

View reviewed changes

Ronsor added 2 commits August 19, 2023 07:07

caff: eliminate gaps and specify unused fields

013cc2f

caff: run go fmt

b3759de

wader reviewed Aug 19, 2023

View reviewed changes

moc3: eliminate gaps and properly handle padding, fix version 5 forma…

e424e29

…t decoding

moc3: count_info: extra space is reserved, not normal alignment/padding

fac1e68

moc3: update test data

833b063

wader merged commit 2eae4c2 into wader:master Aug 20, 2023

		@@ -0,0 +1,1766 @@
		package moc3

		// https://github.com/OpenL2D/moc3ingbird/blob/master/src/moc3.hexpat

moc3 and caff: add decoder #747

moc3 and caff: add decoder #747

Uh oh!

Conversation

Ronsor commented Aug 18, 2023

Uh oh!

wader left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wader commented Aug 18, 2023

Uh oh!

Richardn2002 commented Aug 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wader commented Aug 18, 2023

Uh oh!

Richardn2002 commented Aug 18, 2023

Uh oh!

wader commented Aug 18, 2023

Uh oh!

Ronsor commented Aug 18, 2023

Uh oh!

Ronsor commented Aug 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wader left a comment

Choose a reason for hiding this comment

Uh oh!

wader Aug 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ronsor Aug 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ronsor commented Aug 19, 2023

Uh oh!

wader commented Aug 19, 2023

Uh oh!

Ronsor commented Aug 19, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wader left a comment •

edited

Loading

Richardn2002 commented Aug 18, 2023 •

edited

Loading

Ronsor commented Aug 19, 2023 •

edited

Loading

wader Aug 19, 2023 •

edited

Loading

Ronsor Aug 19, 2023 •

edited

Loading