Skip to content

Conversation

kwvg
Copy link
Collaborator

@kwvg kwvg commented Feb 23, 2024

Additional Information

Breaking Changes

None. Changes are restricted to BIP324 cryptosuite, tests and associated logic.

Checklist:

Go over all the following points, and put an x in all the boxes that apply.

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas (note: N/A)
  • I have added or updated relevant unit/integration/functional/e2e tests (note: N/A)
  • I have made corresponding changes to the documentation (note: N/A)
  • I have assigned this pull request to a milestone (for repository code-owners and collaborators only)

@kwvg kwvg added this to the 20.1 milestone Feb 23, 2024
@kwvg kwvg force-pushed the p2p_primitives branch 2 times, most recently from 4abb0bb to 8db1f7c Compare February 27, 2024 14:22
PastaPastaPasta added a commit that referenced this pull request Feb 27, 2024
, bitcoin#21604, bitcoin#23859, partial bitcoin#21798, bitcoin#26341 (auxiliary backports: part 13)

d033cd5 partial bitcoin#26341: add BIP158 false-positive element check in rpc_scanblocks.py (Kittywhiskers Van Gogh)
eab94ac merge bitcoin#23859: Add missing suppressions for crypto_diff_fuzz_chacha20.cpp (Kittywhiskers Van Gogh)
0e8d4a1 partial bitcoin#21798: Create a block template in tx_pool targets (Kittywhiskers Van Gogh)
ad71db2 merge bitcoin#21604: Document why no symbol names can be used for suppressions (Kittywhiskers Van Gogh)
c116d84 merge bitcoin#21599: Replace file level integer overflow suppression with function level suppression (Kittywhiskers Van Gogh)
8a0dc8c merge bitcoin#21586: Add missing suppression for signed-integer-overflow:txmempool.cpp (Kittywhiskers Van Gogh)
53cf0d5 merge bitcoin#21000: Add UBSan suppressions needed for fuzz tests to not warn under -fsanitize=integer (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Dependency for #5902
  * [bitcoin#21586](bitcoin#21586) was listed as DNM in the backports Google Sheet as listed below as it is `reverted as bitcoin#23059`
    ![Listing of bitcoin#21586 as DNM](https://github.com/dashpay/dash/assets/63189531/413c116a-b3cb-4b58-becd-0d731b0e4b0b)
    * [bitcoin#23059](bitcoin#23059) actually reverts [bitcoin#22925](bitcoin#22925) (which deals with `addrman.cpp`), not [bitcoin#21586](bitcoin#21586) (which deals with `txmempool.cpp`).

  ## Breaking Changes

  None, changes are limited to fuzzing, functional tests and undefined behavior exclusions

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: 7b6c3fcb462edc9bbcc12a0c4eab052d3ef3a6048281b162dd5650f311f6e1bae7a958adf1c03aa0dea09c56c1ddbc99d98a44023fd81f83b2325fd79dc32e80
Copy link

This pull request has conflicts, please rebase.

PastaPastaPasta added a commit that referenced this pull request Feb 28, 2024
, bitcoin#21969, bitcoin#23653, bitcoin#23438, bitcoin#24190, bitcoin#24253, bitcoin#24231, bitcoin#26258, bitcoin#28012, partial bitcoin#25001, bitcoin#25296, bitcoin#23595, bitcoin#27927 (serialization updates)

2b26a87 merge bitcoin#28012: Allow FastRandomContext::randbytes for std::byte, Allow std::byte serialization (Kittywhiskers Van Gogh)
4eeafa2 partial bitcoin#27927: Allow std::byte and char Span serialization (Kittywhiskers Van Gogh)
cb2fa83 test: place the std::ostream operator<< definition in namespace std (Kittywhiskers Van Gogh)
cf4522f partial bitcoin#23595: Add ParseHex<std::byte>() helper (Kittywhiskers Van Gogh)
e4091aa partial bitcoin#25296: Add DataStream without ser-type and ser-version (Kittywhiskers Van Gogh)
eab031a merge bitcoin#26258: Remove unused CDataStream::rdbuf method (Kittywhiskers Van Gogh)
95b5850 partial bitcoin#25001: Modernize util/strencodings and util/string: string_view and optional (Kittywhiskers Van Gogh)
5fe72bb merge bitcoin#24231: Fix read-past-the-end and integer overflows (Kittywhiskers Van Gogh)
24af372 merge bitcoin#24253: Remove broken and unused CDataStream methods (Kittywhiskers Van Gogh)
baf8dd6 merge bitcoin#24190: Fix sanitizer suppresions in streams_tests (Kittywhiskers Van Gogh)
e933d78 merge bitcoin#23438: Use spans of std::byte in serialize (Kittywhiskers Van Gogh)
d3b2822 merge bitcoin#23653: Generalize/simplify VectorReader into SpanReader (Kittywhiskers Van Gogh)
2c32a09 merge bitcoin#21969: Switch serialize to uint8_t (Kittywhiskers Van Gogh)
0a08dbf merge bitcoin#21824: Replace deprecated char with uint8_t in serialization (Kittywhiskers Van Gogh)
d0b4e56 merge bitcoin#21966: Remove double serialization; use software encoder for fee estimation (Kittywhiskers Van Gogh)
1d6aafe merge bitcoin#21817: Replace &foo[0] with foo.data() (Kittywhiskers Van Gogh)
d9a8ce2 trivial: move GetSerializeSize away from Stream (Un)serialize functions (Kittywhiskers Van Gogh)

Pull request description:

  ## Additional Information

  * Dependency for #5902

  * [bitcoin#24231](bitcoin#24231) is merged after [bitcoin#24253](bitcoin#24253) due to a MinGW bug ([comment](bitcoin#24231 (comment)))

  * [bitcoin#25001](bitcoin#25001) is listed as unmerged despite being committed upstream as bitcoin@132d5f8

  * [bitcoin#25296](bitcoin#25296) is listed as unmerged despite being committed upstream as bitcoin@79e007d

  * [bitcoin#21966](bitcoin#21966) was partially backported in [dash#4197](#4197) as f946c68, including only 2be4cd9.
    The excluded commits have been backported, marking the pull request as fully merged.

  * [bitcoin#23438](bitcoin#23438) was partially backported in [dash#5574](#5574) as de54b87, including only fa65bbf.
     The excluded commits have been backported, marking the pull request as fully merged.

  * [bitcoin#27927](bitcoin#27927) opened a fresh can of hell thanks to being (possibly?) the first pull request to include `std::byte` `BOOST_CHECK`'s to the unit test suite. For reasons still unbeknownst to me, it refused to compile, despite being perfectly happy when checked-out as a commit directly and built as-is from upstream.

    The compile error was like this (edited for brevity):
    ```
      CXX      test/test_dash-serialize_tests.o
    In file included from test/serialize_tests.cpp:13:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/unit_test.hpp:18:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/test_tools.hpp:46:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/tools/old/impl.hpp:24:
    /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/tools/detail/print_helper.hpp:53:13: error: static assertion failed due to requirement 'boost::has_left_shift<std::basic_ostream<char, std::char_traits<char>>, std::byte, boost::binary_op_detail::dont_care>::value': Type has to implement operator<< to be printable
                BOOST_STATIC_ASSERT_MSG( (boost::has_left_shift<std::ostream,T>::value),
                ^                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    [...]
    <scratch space>:206:1: note: expanded from here
    BOOST_PP_REPEAT_1
    ^
    test/serialize_tests.cpp:347:9: note: in instantiation of function template specialization 'boost::test_tools::tt_detail::check_frwd<boost::test_tools::tt_detail::equal_impl_frwd, std::byte, std::byte>' requested here
            BOOST_CHECK_EQUAL(out.at(0), std::byte{'a'});
            ^
    [...]
    In file included from test/serialize_tests.cpp:13:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/unit_test.hpp:18:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/test_tools.hpp:46:
    In file included from /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/tools/old/impl.hpp:24:
    /src/dash/depends/x86_64-pc-linux-gnu/include/boost/test/tools/detail/print_helper.hpp:55:18: error: invalid operands to binary expression ('std::ostream' (aka 'basic_ostream<char>') and 'const std::byte')
                ostr << t;
                ~~~~ ^  ~
    /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/cstddef:130:5: note: candidate function template not viable: no known conversion from 'std::ostream' (aka 'basic_ostream<char>') to 'byte' for 1st argument
        operator<<(byte __b, _IntegerType __shift) noexcept
        ^
    [...]
    5 warnings and 2 errors generated.
    make[2]: *** [Makefile:17842: test/test_dash-serialize_tests.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    make[2]: Leaving directory '/src/dash/src'
    make[1]: *** [Makefile:18525: all-recursive] Error 1
    make[1]: Leaving directory '/src/dash/src'
    make: *** [Makefile:802: all-recursive] Error 1
    ```
    * No such error was present on Bitcoin.

      It is true, that no `std::ostream& operator<<(std::ostream& a, const std::byte& b)` is present by default but attempting to `grep` for any specializations didn't show anything _relevant_ that Dash didn't have. Searching on GitHub didn't help either.
    * Then, assuming that perhaps Boost's assertion logic may have changed, upgraded the version of Boost to match the pull request at the time, Boost 1.81. That also did not do anything (and actually, caused a trailing slashes unit test to fail but doesn't cause any problem in Bitcoin because they got rid of their `boost::filesystem` usage by then).
    * If that isn't it, then let's try building Bitcoin with Dash's depends. It built successfully, ran successfully. The problem isn't in the dependency, it's in the codebase.
    * Since it seemed to be `std::byte` related, pull requests that are related to `std::byte` serialization were backported and `std::byte` serialization related changes needed for [dash#5902](#5902) were cherry-picked. That's why this pull request came to be. But it didn't help this particular issue (though it did smooth out the cherry-picks).
    * Running out of ideas, `gdb` is used to step through `serialize_tests`'s `class_methods` and understand why `BOOST_CHECK_EQUAL(out.at(0), std::byte{'a'});` is valid in Bitcoin but not Dash by finding the elusive `operator<<`. This is where things go from bad to worse.

      Turns out, when you build with `clang`, `gdb` loses the ability to do breakpoints by file. So, an attempt is made to use `lldb` (which btw, is called `lldb-16`, running `lldb` with yield an error if you're using the develop container) and it refuses to work, erroring `personality set failed: Operation not permitted`.

      Turns out, the [`docker-compose.yml`](https://github.com/dashpay/dash/blob/37f43e4e56de0d510110a7f790df34ea77748dc9/contrib/containers/develop/docker-compose.yml) needs the following additions:

      ```
        cap_add:
          - SYS_PTRACE
        security_opt:
          - "seccomp:unconfined"
      ```

      After making these changes, `lldb` works and then we resume trying to find `operator<<`. After too many hours and nimbly alternating between `next` and `step`, tried making a return to `gdb` (compiling with `gcc` this time with the appropriate `CXXFLAGS`) hoping for different results and a while later, realized that it cannot step through Boost's headers (it doesn't recognize the filenames) and then recompile it with `clang` and return to `lldb`.

      This was a wild goose chase.

    * After a lot of futile efforts to find the operator by stepping through `BOOST_CHECK_EQUAL`, a basic example addressing the static assertion (that a left shift operator must exist of `<type>` (here `std::byte`) for `std::ostream`) was added in

      ```c++
      std::ostream& ostr = std::cout;
      ostr << std::byte{'a'};
      ```

      ...and it compiled in both codebases.

      So the left shift that Boost is asserting doesn't exist does exist but it isn't being detected for some reason. Upon hovering the `<<`, VSCode highlighted the source of the definition as [`setup_common.h`](https://github.com/dashpay/dash/blob/96ac317c2714afcac1ff4e2df309f17149f42776/src/test/util/setup_common.h#L27-L32) thanks to the comment above it.

      Diffing between Bitcoin and Dash revealed the secret, the `operator<<` definition was placed under namespace `std` by [bitcoin#23497](bitcoin#23497) in bitcoin@f7086fd (see [change](https://github.com/bitcoin/bitcoin/blame/f7086fd8ff084ab0dd656d75b7485e59263bdfd8/src/test/util/setup_common.h#L29-L35)).

      That change has now been made in a separate commit.

  ## Breaking Changes

  Changes in serialization APIs will make backports predating [bitcoin#23438](bitcoin#23438) annoying but will _not_ change how data is stored on disk.

  ## Checklist:
    _Go over all the following points, and put an `x` in all the boxes that apply._
  - [x] I have performed a self-review of my own code
  - [x] I have commented my code, particularly in hard-to-understand areas **(note: N/A)**
  - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)**
  - [x] I have made corresponding changes to the documentation **(note: N/A)**
  - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_

Top commit has no ACKs.

Tree-SHA512: 1d7c67f5fe282f78e24cb720828e5f1f48b6926006b903c28399938532cc5c470c175b00c8b80e9662c4467c1201e09ae6d1cd9b95e8b20ace5e4410c72c472e
Copy link

This pull request has conflicts, please rebase.

@kwvg kwvg marked this pull request as ready for review March 1, 2024 17:31
@kwvg kwvg requested review from knst, UdjinM6 and PastaPastaPasta March 1, 2024 17:33
Comment on lines +813 to +954
std::vector<std::byte> total_tag(Poly1305::TAGLEN);
total_ctx.Finalize(total_tag);
Copy link
Member

@PastaPastaPasta PastaPastaPasta Mar 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using vector instead of array; does our serialization not support arrays? disregard; was looking at wrong lines

PastaPastaPasta
PastaPastaPasta previously approved these changes Mar 1, 2024
Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK for merging via merge commit

src/bip324.cpp Outdated
{
// Determine salt (fixed string + network magic bytes)
const auto& message_header = Params().MessageStart();
std::string salt = std::string{"dash_v2_shared_secret"} + std::string(std::begin(message_header), std::end(message_header));
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need to deviate from bip324 here? Isn't having different network magic bytes enough already? I think we could simply go with the original salt, drop 566ba2a and it should just work.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The salt is composed of a fixed message and network bytes. Changing network bytes will therefore change the salt. We could reuse Bitcoin's salt but that would involve hardcoding their network bytes. I don't really see a good reason to do that.

Since we're already changing the salt due to the difference in network bytes, I went ahead and changed the fixed message too... either ways, without hardcoding Bitcoin's network magic, we'd have to regenerate values. This has been done before, the message magic for instance, was changed for Dash (with its former name).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, right... will have to update test vectors anyway... hmm.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has been done before, the message magic for instance, was changed for Dash (with its former name).

Yes, I remember that and that's exactly why I wanted to avoid making the same mistake here :D

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm leaning towards following bip324 and simply using our bytes. @PastaPastaPasta ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I agree with Udjin here; I am open to further discussion

The BIP even outlines the salt as something parameterized; I'm receptive to an argument that this may make us non-compatible with bip-324; I guess we should create a dip which specifies the modified salt?

def initialize_v2_transport(peer, ecdh_secret, initiating):
    # Include NETWORK_MAGIC to ensure a connection between nodes on different networks will immediately fail
    prk = HKDF_Extract(Hash=sha256, salt=b'bitcoin_v2_shared_secret' + NETWORK_MAGIC, ikm=ecdh_secret)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The BIP even outlines the salt as something parameterized
...

def initialize_v2_transport(peer, ecdh_secret, initiating):
    # Include NETWORK_MAGIC to ensure a connection between nodes on different networks will immediately fail
    prk = HKDF_Extract(Hash=sha256, salt=b'bitcoin_v2_shared_secret' + NETWORK_MAGIC, ikm=ecdh_secret)

It explicitly states that NETWORK_MAGIC is what does the job here, not the (bitcoin_v2_shared_secret) prefix. Also, creating a DIP only to say that our version uses another string feels weird... like, it's not such a huge improvement to have its own DIP imo :D

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm; @kwvg any input here? any reason we should change the static (non-network bytes component) shared secret? If having different network bytes is enough then...? why change this too?

Copy link
Collaborator Author

@kwvg kwvg Mar 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An earlier version of this PR changed didn't change the fixed message. The values were recomputed solely due to the difference in network bytes. On remembering the change of MESSAGE_MAGIC and taking it as precedent, the fixed message was changed and values recomputed again.

I don't think simply changing the fixed message would require issuing a DIP. It seems unusual for any (unlikely) third party implementation that supports Bitcoin forks and BIP324 to expect that the fixed message would be the same no matter what.

Retaining the Bitcoin branding for something like PSBT makes sense since the benefit of renaming it is nonexistent compared to the trouble it'd give to developers. Retaining it here in the fixed message doesn't cause harm or benefit.

I didn't know that changing MESSAGE_MAGIC was regrettable. It does lock you into a value that will remain the same regardless of future rebranding but I lean towards the belief a magic value should be unique (not just unique enough).

But I don't feel strongly on the matter.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@UdjinM6, the fixed message has been changed back to bitcoin_v2_shared_secret in the latest push (diff).

@kwvg kwvg requested a review from UdjinM6 March 1, 2024 18:29
knst
knst previously approved these changes Mar 2, 2024
Copy link
Collaborator

@knst knst left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@UdjinM6 UdjinM6 modified the milestones: 20.1, 21 Mar 5, 2024
@kwvg kwvg dismissed stale reviews from knst and PastaPastaPasta via da40c7d March 5, 2024 18:38
@kwvg kwvg force-pushed the p2p_primitives branch from 9aee22f to da40c7d Compare March 5, 2024 18:38
@kwvg kwvg requested a review from knst March 5, 2024 18:39
@kwvg kwvg requested a review from PastaPastaPasta March 5, 2024 18:39
PastaPastaPasta
PastaPastaPasta previously approved these changes Mar 5, 2024
Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

force push diff here looks good. https://github.com/dashpay/dash/compare/9aee22f2067bc9f103d582e3ab8c6dac4a182c8f..da40c7da5fc7624e99782dd6f18cb84deb0f6927;

@kwvg can you please rebase against latest develop to fix the "merge into master CI"?

Copy link

github-actions bot commented Mar 5, 2024

This pull request has conflicts, please rebase.

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Range diff looks pretty good git range-diff 7701948873a229a15d0dc1a2d71c6ced07d0858a..da40c7d 27736114a1383576689a51af1f7b5058a5f85aa2..b60c493

1:  a3e7ceb5bf = 1:  d7482eb8a6 merge bitcoin#27985: Add support for RFC8439 variant of ChaCha20
2:  74d8bd6c6c = 2:  ff542199bc merge bitcoin#27993: Make poly1305 support incremental computation + modernize
3:  5feb595d01 ! 3:  1b1924e3d5 merge bitcoin#28008: BIP324 ciphersuite
    @@ Commit message
     
      ## src/Makefile.am ##
     @@ src/Makefile.am: BITCOIN_CORE_H = \
    +   base58.h \
    +   batchedlogger.h \
        bech32.h \
    -   bip39.h \
    -   bip39_english.h \
     +  bip324.h \
        blockencodings.h \
        bloom.h \
    @@ src/Makefile.bench.include: bench_bench_dash_SOURCES = \
     
      ## src/Makefile.test.include ##
     @@ src/Makefile.test.include: BITCOIN_TESTS =\
    +   test/base64_tests.cpp \
        test/bech32_tests.cpp \
        test/bip32_tests.cpp \
    -   test/bip39_tests.cpp \
     +  test/bip324_tests.cpp \
        test/block_reward_reallocation_tests.cpp \
        test/blockchain_tests.cpp \
4:  5fd5bc3658 = 4:  7c5edf772a merge bitcoin#28267: BIP324 ciphersuite follow-up
5:  b9aa8a5d7d = 5:  c2aa01cf1d merge bitcoin#28374: python cryptography required for BIP 324 functional tests
6:  da40c7da5f = 6:  b60c493265 merge bitcoin#28100: more Span<std::byte> modernization & follow-ups

Re-reviewed the non-equal commit and it looks good

Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@PastaPastaPasta PastaPastaPasta merged commit e84e3d9 into dashpay:develop Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants