Skip to content

Conversation

CuriousGeorgiy
Copy link
Member

@CuriousGeorgiy CuriousGeorgiy commented May 7, 2023

Closes #8147
Closes #8633

@CuriousGeorgiy CuriousGeorgiy force-pushed the gh-8147-tuple-formats-in-iproto-responses branch 25 times, most recently from 2049042 to bfcdce0 Compare May 8, 2023 14:13
@CuriousGeorgiy CuriousGeorgiy added the asan-ci Enables asan build tests for a pull request label Oct 16, 2023
@CuriousGeorgiy CuriousGeorgiy force-pushed the gh-8147-tuple-formats-in-iproto-responses branch 3 times, most recently from e000561 to da50539 Compare October 16, 2023 14:14
@Totktonada
Copy link
Member

OK regarding 'lua: handle MP_TUPLE extension in merge buffer source's next method' commit.

CuriousGeorgiy added a commit to CuriousGeorgiy/go-tarantool that referenced this pull request Oct 24, 2023
@CuriousGeorgiy CuriousGeorgiy force-pushed the gh-8147-tuple-formats-in-iproto-responses branch 3 times, most recently from 980faba to b1ff4cb Compare October 27, 2023 13:31
CuriousGeorgiy added a commit to CuriousGeorgiy/vshard that referenced this pull request Oct 27, 2023
Temporarily add tarantool/tarantool#8630 to test matrix.

NO_CHANGELOG=<temporary commit>
NO_DOC=<temporary commit>
NO_TEST=<temporary commit>
CuriousGeorgiy added a commit to CuriousGeorgiy/vshard that referenced this pull request Oct 27, 2023
Temporarily add tarantool/tarantool#8630 to test matrix.

NO_DOC=<temporary commit>
NO_TEST=<temporary commit>
@CuriousGeorgiy CuriousGeorgiy force-pushed the gh-8147-tuple-formats-in-iproto-responses branch from b1ff4cb to 8da0af0 Compare October 27, 2023 13:42
Add new MsgPack extension for tuples MP_TUPLE with the following structure:
+------+-----------------+----------+
|MP_EXT|      MP_UINT    | MP_ARRAY |
+------+-----------------+----------+
        ^                 ^
        format identifier tuple data

Needed for tarantool#8147

NO_CHANGELOG=<internal change>
NO_DOC=<internal change>
Extend the MsgPack object creation interface to push a MsgPack context
required for decoding tuples coming from IPROTO, the context ownership is
acquired by the new object to avoid copying (thus, a virtual `move`
function is added to the MsgPack context).

When a MsgPack context is owned by a MsgPack object, the lifetime of the
former is controlled by the latter, hence a virtual `destroy` function is
added to the MsgPack context.

Needed for tarantool#8147

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring
In scope of tarantool#8147, a new context-dependent extension for
box tuples, `MP_TUPLE`, is introduced. The buffer source uses a buffer with
raw MsgPack, which does not allow for passing the context required for
decoding `MP_TUPLE`, so, in order to decode it, we need to manually skip
the extension header and the tuple format identifier to get to the tuple
data and create a tuple. We can ignore the tuple format identifier (and the
tuple format that was originally sent for this tuple), since the format is
provided by the merger itself.

Needed for tarantool#8147

NO_CHANGELOG=<internal change>
NO_TEST=<tested by integration tests>
NO_DOC=<internal change>
Tuple formats are sent in a separate IPROTO_TUPLE_FORMATS response body
field, which is namely a mapping from format identifiers to serialized
tuple formats.

The algorithm for parsing IPROTO response body is the following:
1. Parse tuple formats and build a tuple format index.
2. Parse data passing the tuple format index in the decoding context.

Add new backward compatibility option `box_tuple_extension` which controls
the server's `IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION` bit: needed for
users that don't expect the new tuple extension to be returned in call and
eval responses instead of basic MsgPack types. This option will
also be used for `IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION` bit with the
same purpose.

Closes tarantool#8147

NO_DOC=<added in a later commit together with documentation of tuple
        formats in IPROTO `call` and `eval` arguments>
Extend the MsgPack port to store a pointer to a MsgPack context (the
lifetime of a MsgPack port is static, so we simply allocate it on the
stack), avoiding copying. This context can later be used for dumping
MsgPack to Lua, and possibly acquired by a MsgPack object when the
'MsgPack object' mode of `port_dump_lua` is used.

Needed for tarantool#8147

NO_CHANGELOG=refactoring
NO_DOC=refactoring
NO_TEST=refactoring
Since tuple format cannot be created in IPROTO thread, we only save the
tuple format data and later create a tuple format map from it when
processing `call` and `eval` requests.

Closes tarantool#8633

@TarantoolBot document
Title: Tuple formats in IPROTO

New MsgPack extension was added, `MP_TUPLE`, which is used exclusively
in IPROTO. It has the following structure:
+----------+-----------------+----------+
|MP_EXT = 7|      MP_UINT    | MP_ARRAY |
+----------+-----------------+----------+
            ^                 ^
            format identifier tuple data

New IPROTO features were added:
* `IPROTO_FEATURE_DML_TUPLE_EXTENSION = 7` — tuples in `IPROTO_DATA` are
   encoded as `MP_TUPLE` and tuple format is sent in response to DML
   requests.
* `IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION = 8` —tuples in `IPROTO_DATA`
  are encoded as `MP_TUPLE` and tuple format is sent in response to `call`
  and `eval` requests.
* `IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION = 9` — tuples in `IPROTO_TUPLE`
  of `call` and `eval` requests are encoded as `MP_TUPLE` with their
  formats encoded as described below.

The `IPROTO_FEATURE_DML_TUPLE_EXTENSION` feature can be enabled in
`net.box` by setting the `fetch_schema` option to `false` when creating a
new connection.

A new IPROTO key for bodies of responses with data and bodies of `call` and
`eval` requests: `IPROTO_TUPLE_FORMATS = 0x60` — a `MP_MAP` consisting of
`MP_UINT` keys (format identifiers) and `MP_STRING` values (tuple format
clauses).

A new backward compatibility option was added (and a new page,
`https://tarantool.io/compat/box_tuple_extension`, needs to be created for
it): `BOX_TUPLE_EXTENSION_BRIEF` — it controls the
`IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION` and
`IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION` feature bits. It was added for
clients that assume to receive `MP_ARRAY`s in `call` and `eval` requests
arguments and response data.

Please refer to the design document for [details](https://www.notion.so/tarantool/Schemafull-IPROTO-cc315ad6bdd641dea66ad854992d8cbf?pvs=4#7d86000c69cb4f8087a289f2caad73a0).
@CuriousGeorgiy CuriousGeorgiy force-pushed the gh-8147-tuple-formats-in-iproto-responses branch from 8da0af0 to ccea93d Compare October 27, 2023 13:46
@CuriousGeorgiy CuriousGeorgiy added full-ci Enables all tests for a pull request and removed integration-ci Enables integration tests for a pull request asan-ci Enables asan build tests for a pull request labels Oct 27, 2023
@alyapunov alyapunov merged commit d79e7a3 into tarantool:master Oct 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
full-ci Enables all tests for a pull request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support tuple formats in IPROTO call and eval request arguments Provide names in IPROTO responses
8 participants