-
Notifications
You must be signed in to change notification settings - Fork 387
box: support tuple formats in IPROTO request responses #8630
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
alyapunov
merged 12 commits into
tarantool:master
from
CuriousGeorgiy:gh-8147-tuple-formats-in-iproto-responses
Oct 27, 2023
Merged
box: support tuple formats in IPROTO request responses #8630
alyapunov
merged 12 commits into
tarantool:master
from
CuriousGeorgiy:gh-8147-tuple-formats-in-iproto-responses
Oct 27, 2023
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2049042
to
bfcdce0
Compare
e000561
to
da50539
Compare
OK regarding 'lua: handle MP_TUPLE extension in merge buffer source's next method' commit. |
3 tasks
CuriousGeorgiy
added a commit
to CuriousGeorgiy/go-tarantool
that referenced
this pull request
Oct 24, 2023
3 tasks
980faba
to
b1ff4cb
Compare
CuriousGeorgiy
added a commit
to CuriousGeorgiy/vshard
that referenced
this pull request
Oct 27, 2023
Temporarily add tarantool/tarantool#8630 to test matrix. NO_CHANGELOG=<temporary commit> NO_DOC=<temporary commit> NO_TEST=<temporary commit>
CuriousGeorgiy
added a commit
to CuriousGeorgiy/vshard
that referenced
this pull request
Oct 27, 2023
Temporarily add tarantool/tarantool#8630 to test matrix. NO_DOC=<temporary commit> NO_TEST=<temporary commit>
b1ff4cb
to
8da0af0
Compare
Add new MsgPack extension for tuples MP_TUPLE with the following structure: +------+-----------------+----------+ |MP_EXT| MP_UINT | MP_ARRAY | +------+-----------------+----------+ ^ ^ format identifier tuple data Needed for tarantool#8147 NO_CHANGELOG=<internal change> NO_DOC=<internal change>
Extend the MsgPack object creation interface to push a MsgPack context required for decoding tuples coming from IPROTO, the context ownership is acquired by the new object to avoid copying (thus, a virtual `move` function is added to the MsgPack context). When a MsgPack context is owned by a MsgPack object, the lifetime of the former is controlled by the latter, hence a virtual `destroy` function is added to the MsgPack context. Needed for tarantool#8147 NO_CHANGELOG=refactoring NO_DOC=refactoring NO_TEST=refactoring
In scope of tarantool#8147, a new context-dependent extension for box tuples, `MP_TUPLE`, is introduced. The buffer source uses a buffer with raw MsgPack, which does not allow for passing the context required for decoding `MP_TUPLE`, so, in order to decode it, we need to manually skip the extension header and the tuple format identifier to get to the tuple data and create a tuple. We can ignore the tuple format identifier (and the tuple format that was originally sent for this tuple), since the format is provided by the merger itself. Needed for tarantool#8147 NO_CHANGELOG=<internal change> NO_TEST=<tested by integration tests> NO_DOC=<internal change>
Tuple formats are sent in a separate IPROTO_TUPLE_FORMATS response body field, which is namely a mapping from format identifiers to serialized tuple formats. The algorithm for parsing IPROTO response body is the following: 1. Parse tuple formats and build a tuple format index. 2. Parse data passing the tuple format index in the decoding context. Add new backward compatibility option `box_tuple_extension` which controls the server's `IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION` bit: needed for users that don't expect the new tuple extension to be returned in call and eval responses instead of basic MsgPack types. This option will also be used for `IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION` bit with the same purpose. Closes tarantool#8147 NO_DOC=<added in a later commit together with documentation of tuple formats in IPROTO `call` and `eval` arguments>
Extend the MsgPack port to store a pointer to a MsgPack context (the lifetime of a MsgPack port is static, so we simply allocate it on the stack), avoiding copying. This context can later be used for dumping MsgPack to Lua, and possibly acquired by a MsgPack object when the 'MsgPack object' mode of `port_dump_lua` is used. Needed for tarantool#8147 NO_CHANGELOG=refactoring NO_DOC=refactoring NO_TEST=refactoring
Since tuple format cannot be created in IPROTO thread, we only save the tuple format data and later create a tuple format map from it when processing `call` and `eval` requests. Closes tarantool#8633 @TarantoolBot document Title: Tuple formats in IPROTO New MsgPack extension was added, `MP_TUPLE`, which is used exclusively in IPROTO. It has the following structure: +----------+-----------------+----------+ |MP_EXT = 7| MP_UINT | MP_ARRAY | +----------+-----------------+----------+ ^ ^ format identifier tuple data New IPROTO features were added: * `IPROTO_FEATURE_DML_TUPLE_EXTENSION = 7` — tuples in `IPROTO_DATA` are encoded as `MP_TUPLE` and tuple format is sent in response to DML requests. * `IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION = 8` —tuples in `IPROTO_DATA` are encoded as `MP_TUPLE` and tuple format is sent in response to `call` and `eval` requests. * `IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION = 9` — tuples in `IPROTO_TUPLE` of `call` and `eval` requests are encoded as `MP_TUPLE` with their formats encoded as described below. The `IPROTO_FEATURE_DML_TUPLE_EXTENSION` feature can be enabled in `net.box` by setting the `fetch_schema` option to `false` when creating a new connection. A new IPROTO key for bodies of responses with data and bodies of `call` and `eval` requests: `IPROTO_TUPLE_FORMATS = 0x60` — a `MP_MAP` consisting of `MP_UINT` keys (format identifiers) and `MP_STRING` values (tuple format clauses). A new backward compatibility option was added (and a new page, `https://tarantool.io/compat/box_tuple_extension`, needs to be created for it): `BOX_TUPLE_EXTENSION_BRIEF` — it controls the `IPROTO_FEATURE_CALL_RET_TUPLE_EXTENSION` and `IPROTO_FEATURE_CALL_ARG_TUPLE_EXTENSION` feature bits. It was added for clients that assume to receive `MP_ARRAY`s in `call` and `eval` requests arguments and response data. Please refer to the design document for [details](https://www.notion.so/tarantool/Schemafull-IPROTO-cc315ad6bdd641dea66ad854992d8cbf?pvs=4#7d86000c69cb4f8087a289f2caad73a0).
8da0af0
to
ccea93d
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #8147
Closes #8633