Skip to content

Conversation

ajtowns
Copy link
Contributor

@ajtowns ajtowns commented Aug 24, 2023

This PR changes -acceptnonstdtxn=1 so that it also skips the non-mandatory script checks, allowing txs that (eg) use OP_NOPx or OP_SUCCESS into the mempool. This remains only available on test nets.

@DrahtBot
Copy link
Contributor

DrahtBot commented Aug 24, 2023

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
Concept ACK darosior, moonsettler, sipa, instagibbs

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #29280 (Implement OP_CHECKTEMPLATEVERIFY by reardencode)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Copy link
Member

@darosior darosior left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK. Makes sense to me.

@moonsettler
Copy link

Concept ACK!

@ajtowns
Copy link
Contributor Author

ajtowns commented Aug 24, 2023

Added a commit to ensure that the mempool is at least enforcing the script rules that will be required for the next block. Not sure that's sensible: it only makes a difference if there's a rule in GetBlockScriptFlags that's not in MANDATORY_SCRIPT_VERIFY_FLAGS, and if there is such a rule, even with this patch, you could add a tx to the mempool prior to the rule becoming active, in which case it wouldn't magically be removed from the mempool when it became active. There's already a double check in the mining code (test_block_validity) which would prevent PoW being wasted on a block that included an invalid tx because it was somehow made it into the mempool.

@sipa
Copy link
Member

sipa commented Aug 24, 2023

Concept ACK

@instagibbs
Copy link
Member

have -acceptnonstdtxn=1 skip the non-mandatory script checks,

Can you give a bit of motivation for this second part?

@ajtowns ajtowns force-pushed the 202303-acceptnonstdscript branch from b76d2d6 to a656ecc Compare August 25, 2023 06:26
@ajtowns
Copy link
Contributor Author

ajtowns commented Aug 25, 2023

have -acceptnonstdtxn=1 skip the non-mandatory script checks,

Can you give a bit of motivation for this second part?

Mostly that STANDARD_SCRIPT_VERIFY_FLAGS violations sounds like something that acceptnonstdtxn should allow. But some particular examples:

  • When investigating the btcd bugs that crashed lnd, I tried using an OP_SUCCESS to overcome the standard limits, but was blocked by these checks. IIRC I manually disabled them.
  • There was a discussion about being able to see inquisition txs on signet in mempool.space prior to them being mined, without having to run inquisition node software. -acceptnonstdtxn seemed like a way to make this possible, except it doesn't actually work. https://twitter.com/fiatjaf/status/1632710159163170821
  • It makes it hard to mess around with things on regtest https://twitter.com/4moonsettler/status/1694024772403834963

@@ -2016,33 +2042,33 @@ static unsigned int GetBlockScriptFlags(const CBlockIndex& block_index, const Ch
// mainnet.
// For simplicity, always leave P2SH+WITNESS+TAPROOT on except for the two
// violating blocks.
uint32_t flags{SCRIPT_VERIFY_P2SH | SCRIPT_VERIFY_WITNESS | SCRIPT_VERIFY_TAPROOT};
MandatoryFlags flags{MandatoryFlag<SCRIPT_VERIFY_P2SH | SCRIPT_VERIFY_WITNESS | SCRIPT_VERIFY_TAPROOT>};
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably this commit should be in its own PR since it touches block consensus code. I just wanted to make sure that there is a path forward to ensuring we don't introduce mempool bugs in future by updating consensus without also updating MANDATORY_SCRIPT_VERIFY_FLAGS. This gives compile time errors:

validation.cpp:2070:11: error: no viable overloaded '|='
    flags |= SCRIPT_VERIFY_CLEANSTACK;

or

validation.cpp:2013:5: error: static_assert failed due to requirement '(16384U & MANDATORY_SCRIPT_VERIFY_FLAGS) == 16384U' "MandatoryFlag not included in MANDATORY_SCRIPT_VERIFY_FLAGS"
....
validation.cpp:2071:14: note: in instantiation of variable template specialization '(anonymous namespace)::MandatoryFlag<16384>' requested here
    flags |= MandatoryFlag<SCRIPT_VERIFY_NULLFAIL>;

const auto it{consensusparams.script_flag_exceptions.find(*Assert(block_index.phashBlock))};
if (it != consensusparams.script_flag_exceptions.end()) {
flags = it->second;
flags &= it->second;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This prevents script_flag_exceptions from adding rules; it can now only remove from the P2SH, SEGWIT, TAPROOT set.

@instagibbs
Copy link
Member

concept ACK on first half,
~0 on latter commits

@ajtowns ajtowns force-pushed the 202303-acceptnonstdscript branch from a656ecc to 091d52b Compare August 28, 2023 05:28
@ajtowns
Copy link
Contributor Author

ajtowns commented Aug 28, 2023

concept ACK on first half, ~0 on latter commits

Split the first part out into #28354

@ajtowns ajtowns changed the title policy: Update -acceptnonstdtxn behaviour policy: Allow non-standard scripts with -acceptnonstdtxn=1 Aug 28, 2023
@ChrisMartl
Copy link

@ajtowns: what would be the behavior regarding non-standard scripts when -acceptnonstdtxn=0 ?

@ariard
Copy link

ariard commented Aug 30, 2023

Mostly that STANDARD_SCRIPT_VERIFY_FLAGS violations sounds like something that acceptnonstdtxn should allow. But some particular examples:

I think this is valuable for experimentation to allow non-standard scripts with -accetnonstdtxn=1, though I wonder if we wouldn’t wish more granularity e.g allow OP_SUCCESS / OP_NOP without bypassing IsStandardTx checks. I wonder if we might wish partial policy loosening deployment, e.g if a OP_NOP is used as a new data carrier though all the other checks on sanity of spent inputs / transaction size / etc are conserved. Related discussions in #27926.

@Retropex
Copy link

I'm not sure I understand what this brings to Bitcoin could you elaborate?

@moonsettler
Copy link

I'm not sure I understand what this brings to Bitcoin could you elaborate?

it's a rationalization of node behavior. removes some logical inconsistencies related to standardness checks and relay policy and gives you better control over your own node.

sometimes you remove weirdness instead of adding new functionality.

@instagibbs
Copy link
Member

I don't know if it's particularly weird to "protect" upgrade hooks in testnets, but I also know that people can just go do their own thing and then mine a block with violations of these rules, hence my ~0

@ajtowns ajtowns force-pushed the 202303-acceptnonstdscript branch from 091d52b to f09bb66 Compare September 1, 2023 20:29
Copy link
Contributor Author

@ajtowns ajtowns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to replace the template magic enforcing consistency between GetBlockScriptFlags and MANDATORY_SCRIPT_VERIFY_FLAGS with a comment.

template<uint32_t FLAG>
struct MandatoryFlagChecked
{
static_assert((FLAG & MANDATORY_SCRIPT_VERIFY_FLAGS) == FLAG, "MandatoryFlag not included in MANDATORY_SCRIPT_VERIFY_FLAGS");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think there's a reasonable way of doing this with C++20 consteval:

class MandatoryFlags {
private:
    uint32_t value{0};
    struct CheckFlag {
        uint32_t flag;
        consteval CheckFlag(uint32_t f) {
            if ((f & MANDATORY_SCRIPT_VERIFY_FLAGS) != f) {
                throw "Block script flag not included in MANDATORY_SCRIPT_VERIFY_FLAGS";
            }
            flag = f;
        }
    };
public:
    explicit MandatoryFlags(const CheckFlag& flags) { value |= flags.flag; }
    void operator|=(const CheckFlag& flags) { value |= flags.flag; }
    void operator&=(uint32_t flags) { value &= flags; }
    operator uint32_t() const { return value; }
};

then declare MandatoryFlags flags; and you can just say flags |= SCRIPT_VERIFY_DERSIG; as normal, but if you give a non-mandatory flag, you get:

validation.cpp:2072:14: error: call to consteval function '(anonymous namespace)::MandatoryFlags::CheckFlag::CheckFlag' is not a constant expression
    flags |= SCRIPT_VERIFY_MINIMALIF;
             ^
validation.cpp:2018:17: note: subexpression not valid in a constant expression
                throw "Block script flag not included in MANDATORY_SCRIPT_VERIFY_FLAGS";
                ^

and saying flags |= var also gives you a decent error:

validation.cpp:2049:18: error: call to consteval function '(anonymous namespace)::MandatoryFlags::CheckFlag::CheckFlag' is not a constant expression
        flags |= var;
                 ^
validation.cpp:2049:18: note: initializer of 'var' is not a constant expression
validation.cpp:2046:20: note: declared here
        const auto var = it->second;
                   ^

@ajtowns
Copy link
Contributor Author

ajtowns commented Sep 1, 2023

I think this is valuable for experimentation to allow non-standard scripts with -accetnonstdtxn=1, though I wonder if we wouldn’t wish more granularity e.g allow OP_SUCCESS / OP_NOP without bypassing IsStandardTx checks. I wonder if we might wish partial policy loosening deployment, e.g if a OP_NOP is used as a new data carrier though all the other checks on sanity of spent inputs / transaction size / etc are conserved.

I think any time you want more granularity, you should just define the new standardness rule and implement it. Examples:

Using -acceptnonstdtxn is useful for experimenting before you get to that point (or while you're not confident that the code implementing it is really safe to run, but you want to see the txs that use those rules anyway).

@ajtowns ajtowns marked this pull request as ready for review September 1, 2023 20:43
@luke-jr
Copy link
Member

luke-jr commented Sep 5, 2023

I think we should distinguish at least between simply non-standard transactions, and ones that use opcodes reserved for future functionality. Maybe only bypass the latter for -acceptnonstdtxn=2 or something?

@moonsettler
Copy link

@luke-jr that thought crossed my mind too. so long the option is available granularity is not a bad thing at all.

@ajtowns
Copy link
Contributor Author

ajtowns commented Sep 6, 2023

I think we should distinguish at least between simply non-standard transactions, and ones that use opcodes reserved for future functionality. Maybe only bypass the latter for -acceptnonstdtxn=2 or something?

I guess the difference would be:

  • malleability:
    • SCRIPT_VERIFY_MINIMALDATA
    • SCRIPT_VERIFY_CLEANSTACK
    • SCRIPT_VERIFY_MINIMALIF
    • SCRIPT_VERIFY_NULLFAIL
    • SCRIPT_VERIFY_LOW_S
  • upgrades:
    • SCRIPT_VERIFY_DISCOURAGE_UPGRADABLE_NOPS
    • SCRIPT_VERIFY_DISCOURAGE_UPGRADABLE_WITNESS_PROGRAM
    • SCRIPT_VERIFY_DISCOURAGE_UPGRADABLE_TAPROOT_VERSION
    • SCRIPT_VERIFY_DISCOURAGE_OP_SUCCESS
    • SCRIPT_VERIFY_DISCOURAGE_UPGRADABLE_PUBKEYTYPE
  • weirdness:
    • SCRIPT_VERIFY_CONST_SCRIPTCODE
    • SCRIPT_VERIFY_STRICTENC
    • SCRIPT_VERIFY_WITNESS_PUBKEYTYPE

Not really seeing the point in differentiating between them though; all the examples I have are where you want to mess with the upgradable things.

@ariard
Copy link

ariard commented Sep 9, 2023

I think any time you want more granularity, you should just define the new standardness rule and implement it.

Yes this is one policy rules design viewpoint. The pitfall I can think of is in term of DoS protection of full-nodes. Let’s say in the future we discover a new DoS vector on new deployed segwit versions (e.g grafroot spends) and max transaction size. Max transaction size (MAX_STANDARD_TX_WEIGHT) is currently enforced IsStandardTx, lowering this limit would a plausible fix to this DoS vector, however it would come at the ecosystem cost of breaking all the data carriage use-case relying on current MAX_STANDARD_TX_WEIGHT. Those node operators if they have high-performance full-nodes would have an interest to keep those loose default, and we’ll start to have mempool partitioning in the ecosystem (sounds bad for decentralization), I think.

All of that hypothetical, though it sounds the most-granular policy loosening allowing the use-case can be a better policy rules design thumb rule. I wonder if here a new acceptnonstdscripts=1 would fit ? If it’s for experimentation-only, the current code changes could be also signet / testnet /regtest only.

…n=0 node

Prepare for updating -acceptnonstdtxn to allow txns that violate
STANDARD_SCRIPT_VERIFY_FLAGS but not MANDATORY_SCRIPT_VERIFY_FLAGS by
checking the non-mandatory flags with node that enforces standardness.
@instagibbs
Copy link
Member

Leaning on -0 now for this PR.

Unfortunately I think mining pools may start hacking things off with their modifications, including flipping standardness switch in mainnet. From what I can tell, Marathon was doing exactly this feedback was given to introduce back some limits, such as upgrade hooks.

@ajtowns
Copy link
Contributor Author

ajtowns commented Apr 10, 2024

Note that the concept aks in the drahtbot report were mostly/entirely for the "don't do this on testnet by default" aspect of this PR which was split out into #28354.

@maflcko
Copy link
Member

maflcko commented Apr 10, 2024

Maybe reopen as a fresh pull request, if still relevant, so that there is a clean and easy to follow discussion?

@ajtowns
Copy link
Contributor Author

ajtowns commented Apr 10, 2024

Closed and re-opened fresh as #29843

@bitcoin bitcoin locked and limited conversation to collaborators Apr 10, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.