Overhaul the beacon system #1678
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
75% on the AppCache; nice :) |
This replaces the AppCache BEACON section with a strongly-typed API for beacon management. It refactors the placeholder BeaconRegistry class to a service that stores and provides access to beacon data and adds a few new data structures to represent that data.
This updates the following core areas to use the new BeaconRegistry API: - Block acceptance pipeline and validation - Scraper - RPCs that access the full beacon collection
This moves the responsibility for beacon advertisement and management of the node's local beacon to the Researcher API.
After converting the application to the BeaconRegistry API, these legacy functions are no longer used.
These exceptions replace a workaround for beacon signature verification using previous beacon keys (beaconalt). A bug caused some nodes to sign blocks using an old beacon key. The original workaround stored all keys for previous beacons in another map (BEACONALT AppCache section). Using block exceptions instead for the handful of these blocks saves memory.
For the most part, areas that used these functions replaced them with optimized routines. We retain a version of SignBlockWithCPID() to use in the voting code until we refactor that area as well.
We leave a scoped copy of it in the superblock tests for the legacy format compatibility verification.
An issue observed on testnet resulted in exclusion of a CPID from a superblock when that wallet automatically renewed its beacon during the superblock formation period. The new advertisement replaced the beacon with a timestamp later than the beacon consensus window used by the scrapers to create a list of CPIDs to include for statistics processing. This change disables beacon renewal immediately prior to superblock formation to avoid the condition above.
jamescowens
requested changes
May 18, 2020
src/main.cpp
Outdated
Comment on lines
3733
to
3746
| // Attempt to advertise or renew a beacon automatically if the wallet is | ||
| // unlocked and funded. | ||
| // | ||
| if (TimerMain("send_beacon", 180)) { | ||
| NN::Researcher::Get()->AdvertiseBeacon(); | ||
| const NN::ResearcherPtr researcher = NN::Researcher::Get(); | ||
|
|
||
| // Do not perform an automated renewal for participants with existing | ||
| // beacons before a superblock is due. This avoids overwriting beacon | ||
| // timestamps in the beacon registry in a way that causes the renewed | ||
| // beacon to appear ahead of the scraper beacon consensus window. | ||
| // | ||
| if (!researcher->Eligible() || !NN::Quorum::SuperblockNeeded()) { | ||
| researcher->AdvertiseBeacon(); | ||
| } |
There was a problem hiding this comment.
I think this is fine. The simplest way to fix it. Otherwise we have to pin beacons per the ladder, which is far more difficult, and would probably use more memory. The downside of this is that beacon renewal is inconvenient due to not working with the wallet unlocked for staking only, and if someone unlocks the wallet with the intent to renew, this could be confusing.
The proper pattern for beacon renewal should be a wizard screen that pops up when the beacon is first eligible to be renewed. That screen should not come up during the dead zone defined above, even if the beacon is newly eligible for renewal. Also, we should be able to invoke the initial beacon registration/beacon renewal by right clicking on the new beacon icon I added, and/or selecting a menu item called beacon registration/renewal. If someone does this during the dead zone, it should say something like, "You are attempting to renew your beacon too close to the next superblock. Please try again in approximately x hours." If not in the dead zone, then we should ask to fully unlock the wallet, then do the beacon process, and then re-prompt to lock the wallet/unlock for staking only.
There was a problem hiding this comment.
I agree with the above. For now, this only affects automatic renewals for existing beacons if the wallet is unlocked. Users can still manually advertise a beacon renewal right before a superblock if they want to. I accounted for this condition in the upcoming PR for the GUI.
|
Fixed some documentation and spelling in the force-push. |
jamescowens
approved these changes
May 19, 2020
jamescowens
added a commit
that referenced
this pull request
Sep 4, 2020
Added - Backport newer uint256 types from Bitcoin #1570 (@cyrossignol) - Implement project level rain for rainbymagnitude #1580 (@jamescowens) - Upgrade utilities (Update checker and snapshot downloader/application) #1576 (@iFoggz) - Provide fees collected in the block by the miner #1601 (@iFoggz) - Add support for generating legacy superblocks from scraper stats #1603 (@cyrossignol) - Port of the Bitcoin Logger to Gridcoin #1600 (@jamescowens) - Implement zapwallettxes #1605 (@jamescowens) - Implements a global event filter to suppress help question mark #1609 (@jamescowens) - Add next target difficulty to RPC output #1615 (@cyrossignol) - Add caching for block hashes to CBlock #1624 (@cyrossignol) - Make toolbars and tray icon red for testnet #1637 (@jamescowens) - Add an rpc call convergencereport #1643 (@jamescowens) - Implement newline filter on config file read in #1645 (@jamescowens) - Implement beacon status icon/button #1646 (@jamescowens) - Add gridcointestnet.png #1649 (@caraka) - Add precision to support magnitudes less than 1 #1651 (@cyrossignol) - Replace research accrual calculations with superblock snapshots #1657 (@cyrossignol) - Publish example gridcoinresearch.conf as a md document to the doc directory #1662 (@jamescowens) - Add options checkbox to disable transaction notifications #1666 (@jamescowens) - Add support for self-service beacon deletion #1695 (@cyrossignol) - Add support for type-specific contract fee amounts #1698 (@cyrossignol) - Add verifiedbeaconreport and pendingbeaconreport #1696 (@jamescowens) - Add preliminary testing option for block v11 height on testnet #1706 (@cyrossignol) - Add verified beacons manifest part to superblock validator #1711 (@cyrossignol) - Implement beacon, vote, and superblock display categories/icons in UI transaction model #1717 (@jamescowens) - neuralnet: Add integrity checking to researcher accrual snapshot registry #1727 (@jamescowens) - Add workaround for scrypt assembly on macOS #1740 (@cyrossignol) - gui: Build onboarding/beacon wizard #1739 (@cyrossignol) - doc: Add CONTRIBUTING.md from bitcoin #1723 (@div72) - rpc: Implement inspectaccrualsnapshot and parseaccrualsnapshotfile #1744 (@jamescowens) - scraper: Add disk based state backing for verified beacon list in scraper #1751 (@jamescowens) - Add ability to recover beacon in block version 11+ #1768 (@cyrossignol) - refactor: Add transaction context to contract handlers #1777 (@cyrossignol) - gui: Add context for when BOINC is attached to a pool #1775 (@cyrossignol) - doc: Clarify what to do if PR in multiple categories (for CONTRIBUTING.md) #1798 (@RoboticMind) - qt: Add option to choose not to start the wallet minimized #1804 (@jamescowens) - superblock: Add check for OutOfSyncByAge to SuperblockValidator::Validate #1806 (@jamescowens) - contract: Standardize contract validation and add block context #1808 (@cyrossignol) - add seed.gridcoin.pl to default config #1812 (@wilkart) - gui: Implement sidestake send display #1813 (@jamescowens) - gui: Add pool/investor pages to researcher wizard #1819 (@cyrossignol) - ci: Port lint scripts from Bitcoin #1823 (@div72) - doc: Create basic readme in contrib #1826 (@RoboticMind) - gui: Implement TransactionRecord::Message #1829 (@jamescowens) - rpc: Add private_key_available to beaconstatus #1833 (@a123b) - gui: Validate email address in researcher wizard #1840 (@a123b) - rpc: Add "getrawwallettransaction" RPC function #1842 (@cyrossignol) - consensus: Set block version 11 threshold height for mainnet #1862 (@cyrossignol) Changed - Upgrade LevelDB from v1.17 to v1.20 #1562 (@cyrossignol) - Re-enable scrypt optimizations #1450 (@denravonska) - Derive CScript from prevector type (optimization) #1554 (@cyrossignol) - Disable quorum for grandfathered blocks to speed up sync #1568 (@cyrossignol) - Refactor hashBoinc for binary claim contexts #1558 (@cyrossignol) - integrated_scraper_2 branch tracking PR #1559 (@jamescowens) - Upgrade depends - OpenSSL to 1.1.1d #1581 (@jamescowens) - Ubuntu 19.10 fixes #1590 (@denravonska) - Force a re-parse of legacy claims in generated blocks #1592 (@cyrossignol) - Improve the "versionreport" RPC output #1595 (@cyrossignol) - Overhaul the core tally and accrual system #1583 (@cyrossignol) - Overhaul the superblock quorum system #1597 (@cyrossignol) - Add more data to the "superblocks" RPC output #1599 (@cyrossignol) - Update Windows Build doc #1606 (@barton2526) - Change the order of calls in gridcoinresearchd.cpp to optimize rpc shunt path #1610 (@jamescowens) - Change staking tooltip to display frequency #1611 (@jamescowens) - Enhancements to ETTS #1442 (@jamescowens) - Standardize money values as integers #1614 (@cyrossignol) - Clean up and optimize legacy coin age code #1616 (@cyrossignol) - Some scraper cleanups #1620 (@jamescowens) - Reorganize accrual code and fix 6-month cutoff #1630 (@cyrossignol) - Update Copyright years #1633 (@barton2526) - Change team whitelist delimiter to <> for CPID detection #1634 (@cyrossignol) - Change team whitelist separator to <> to accomodate more team names #1632 (@jamescowens) - Change Curl download speed type to support older environments #1640 (@cyrossignol) - Optimize logo SVGs used for tray icons #1638 (@cyrossignol) - Tweak consolidateunspent rpc function #1644 (@jamescowens) - ETTS and staking icon enhancements #1650 (@jamescowens) - Implement new transaction fees for block version 11 #1652 (@jamescowens) - Optimize in-memory storage of superblock data #1653 (@cyrossignol) - Miscellaneous superblock API improvements and housekeeping #1654 (@cyrossignol) - Update openssl to 1.1.1f compatibility #1660 (@jamescowens) - Optimize bdb to avoid synchronous flush of database #1659 (@jamescowens) - Add support for CPID input to "lifetime" RPC function #1668 (@cyrossignol) - Overhaul the contract handling system #1669 (@cyrossignol) - Make the autostart mainnet/testnet aware #1671 (@jamescowens) - Remove slashes from User Agent in peers tab #1674 (@div72) - Refactor contracts for polymorphic binary payloads #1676 (@cyrossignol) - Overhaul the beacon system #1678 (@cyrossignol) - Replace boost::optional<T&> with non-owning pointers #1680 (@cyrossignol) - Optimize proof-of-stake validation #1681 (@cyrossignol) - Updated Slack link #1683 (@NeuralMiner) - Update build-unix.md #1686 (@Quezacoatl1) - Replace deprecated QT methods #1693 (@Pythonix) - Made protocol.h more similar to bitcoin #1688 (@Pythonix) - Touch up some details for block version 11 #1697 (@cyrossignol) - More tweaks for block version 11 #1700 (@cyrossignol) - Finish the conversion to the BCLog class based logger #1699 (@jamescowens) - Move claim version transitional code in miner for proper signature #1712 (@cyrossignol) - doc: Update threads in coding.txt #1730 (@div72) - qt: Include QPainterPath in trafficgraphwidget.cpp #1733 (@div72) - doc: Update doc/build-unix.md #1731 (@div72) - gui: Show peers tab on connections icon click #1734 (@div72) - refactor: Change return type of IsMine to isminetype && move wallet files to wallet directory #1722 (@div72) - build: Updates boost to 1.73.0 for depends #1673 (@jamescowens) - doc: Update Unit Test Readme #1743 (@RoboticMind) - wallet: Change Assert To Error Message In kernel.cpp #1748 (@RoboticMind) - scraper: Shorten display representation of verification codes #1754 (@cyrossignol) - log: Change ".B." to Clear Message #1758 (@RoboticMind) - util: Fix braindamage in GetDefaultDataDir() #1737 (@jamescowens) - scraper: Improve scraper processing of beacon verifications #1760 (@jamescowens) - scraper: Add instrumentation to convergencereport #1763 (@jamescowens) - rpc: Improve rpc stress test script #1767 (@tunisiano187) - Generalize enum serialization #1770 (@cyrossignol) - scraper: Improve handling of ETags in http class and tweak verified beacon logic #1776 (@jamescowens) - scraper: Improve ProcessNetworkWideFromProjectStats and other tweaks #1778 (@jamescowens) - researcher: Automate beacon advertisement for renewals only #1781 (@cyrossignol) - gui: Tweak behavior of beacon page in researcher wizard #1784 (@cyrossignol) - Prepare for block version 11 hard-fork on testnet #1787 (@cyrossignol) - scraper: Modify UpdateVerifiedBeaconsFromConsensus #1791 (@jamescowens) - gui: Optimize OverviewPage::updateTransactions() #1794 (@jamescowens) - ci: Adopt ci changes from Bitcoin #1795 (@div72) - consensus: switch snapshot accrual calculation to integer arithmetic #1799 (@cyrossignol) - voting: Overhaul the voting system #1809 (@cyrossignol) - contract: Optimize contract replay after chain reorganization #1815 (@cyrossignol) - contract: Reimplement transaction messages as contracts #1816 (@cyrossignol) - staking: Sign claim contracts with coinstake transaction #1817 (@cyrossignol) - gui: Change research wizard text #1820 (@div72) - net: Update protocol version and clean up net messaging #1824 (@cyrossignol) - rpc, wallet: Corrections to GetAmounts #1825 (@jamescowens) - gui: Tweak some minor researcher wizard details #1830 (@cyrossignol) - gui: Change GetEstimatedStakingFrequency text #1836 (@jamescowens) - scraper: Scraper global statistics cache optimization #1837 (@jamescowens) - doc: Update Vulnerability Response Process #1843 (@RoboticMind) - scraper: Optimization of manifest and parts sharing between ConvergedScraperStatsCache, mapManifest, and mapParts #1851 (@jamescowens) - consensus: Update Checkpoints #1855 (@barton2526) - docs: Update docs to build off master #1856 (@barton2526) - gui: Fix and improve GUI combo box styles #1858 (@cyrossignol) - build: Tweak Gridcoin installer for Fern release #1863 (@jamescowens) Removed - Remove old research age checks (rebase #1365) #1572 (@cyrossignol) - Remove PrimaryCPID check from diagnostics dialog #1586 (@cyrossignol) - Remove missed label for PrimaryCPID from diagnostics #1588 (@cyrossignol) - Remove legacy quorum messaging system (@neural network) #1589 (@cyrossignol) - Remove old remnants of legacy smart contract experiments #1594 (@cyrossignol) - Remove block nonce for version 11 #1622 (@cyrossignol) - Delete obsolete contrib/Installer and Upgrader directories #1623 (@jamescowens) - Remove redundant LoadAdminMessages() calls #1625 (@cyrossignol) - Remove some legacy informational RPC commands #1658 (@cyrossignol) - Remove informational magnitude field from binary claims #1661 (@cyrossignol) - Remove fDebug3,4, and net and convert to BCLog::LogFlags #1663 (@jamescowens) - Remove qt5.7.1 depends support build System #1665 (@iFoggz) - Remove unused jQuery library #1679 (@cyrossignol) - Remove unused NetworkTimer() function and global state #1701 (@cyrossignol) - Refactor claim context objects into contracts #1704 (@cyrossignol) - Clean old assets up #1718 (@div72) - Remove legacy "rain" RPC (not by-project rain) #1742 (@cyrossignol) - Temporarily disable voting system on testnet #1769 (@cyrossignol) - gui: Remove legacy GUI transaction description for contracts #1772 (@cyrossignol) - gui: Remove transaction fee setting #1780 (@cyrossignol) - trivial: Cleanup unused legacy functions #1793 (@cyrossignol) - mining, rpc: Remove kernel-diff-best and kernel-diff-sum #1796 (@jamescowens) - refactor: Remove libs subdirectory #1802 (@div72) - scraper: cleanup unused/unnecessary functions #1803 (@jamescowens) - gui: Remove useless "Detach databases at shutdown" #1810 (@jamescowens) - test: Remove testnet condition for standard transactions #1814 (@cyrossignol) - consensus: Remove transitional testnet code #1854 (@cyrossignol) Fixed - Fix "Owed" amount in output of "magnitude" RPC method #1569 (@cyrossignol) - Add support for paths with special characters on Windows #1571 (@cyrossignol) - Fix lingering peers.dat temp files and clean up remaining paths #1582 (@cyrossignol) - Fix incorrect beacon length warning in GUI transaction list #1585 (@cyrossignol) - Fix default config file line endings on Windows #1587 (@cyrossignol) - Reenable Travis builds for MacOS #1591 (@jamescowens) - Correct peer detail info background color #1593 (@jamescowens) - Fix exception in debug3 mode #1598 (@cyrossignol) - Fix deadlock in "getmininginfo" RPC function #1596 (@cyrossignol) - Fix accuracy of statistics in "network" RPC output #1602 (@cyrossignol) - Fix heights for quorum vote weight calculations #1604 (@cyrossignol) - Fix deadlock in log archiver when rename fails #1607 (@cyrossignol) - Fix a spurious segmentation fault during client load on Windows with fast CPUs #1608 (@jamescowens) - Fix lock order debugging and potential deadlocks #1612 (@jamescowens) - Add dependencies #1613 (@Scalextrix) - Fix std namespace pollution #1617 (@denravonska) - Add missing condition for newbie accrual computer #1618 (@cyrossignol) - Track first reward blocks in research accounts #1619 (@cyrossignol) - Fix lingering beacon warning after advertisement #1627 (@cyrossignol) - Fix accrual calculation for new, zero-magnitude CPIDs #1636 (@cyrossignol) - Fix diagnostics, add ETTS test, fix tooltipcolor, add missing lock, and add email=investor check #1647 (@jamescowens) - Fix help message of two RPC methods #1656 (@div72) - Fix legacy accrual for newbie with non-zero past reward #1667 (@cyrossignol) - Fix GUI autostart on Windows for paths with wide characters #1670 (@cyrossignol) - Qualify boost bind placeholders with their full namespace #1672 (@Ponce) - Fix suffix when copying txids #1677 (@div72) - Unnecessary if-statement removed #1685 (@Pythonix) - Fix consolidatemsunspent Help Message #1687 (@Pythonix) - Fix gettransaction help message #1691 (@Pythonix) - Fix GetNewMint To Look for Stakes #1692 (@RoboticMind) - Suppress deprecated copy warnings for Qt with GCC 9+ #1702 (@cyrossignol) - Fix exclusion error on stats processing and misplaced ENDLOCK logging entry #1710 (@jamescowens) - Removed unnecessary comparison #1708 (@Pythonix) - Fixed typo #1707 (@Pythonix) - Fix out-of-bounds exception for peers tab version slashes #1713 (@cyrossignol) - Fix transition for v1 superblocks when reorganizing #1714 (@cyrossignol) - Touch up transition to version 2 transactions #1715 (@cyrossignol) - Avoid mutating transactions in ConnectBlock() #1716 (@cyrossignol) - Skip beacon advertisement when already pending #1726 (@cyrossignol) - Fix Windows cross-compilation in newer environments #1728 (@cyrossignol) - Fix out-of-bounds access in IsMineInner() #1736 (@cyrossignol) - Fix a couple of block version 11 issues #1738 (@cyrossignol) - Fix null pointer dereference in GUI researcher model #1741 (@cyrossignol) - accrual: Reset research accounts when rebuilding accrual snapshots #1745 (@cyrossignol) - scraper: Correct update for verified beacons #1747 (@jamescowens) - accrual: Refactor tally initialization for snapshot rebuild #1749 (@cyrossignol) - rpc: Fix "cpid" field in "beaconconvergence" RPC output #1750 (@cyrossignol) - accrual: Fix snapshot accrual superblock state transitions #1752 (@cyrossignol) - scraper: Correct stale verified beacon logic #1753 (@jamescowens) - rpc: Correct possible divide by zero in getblockstats #1755 (@jamescowens) - gui: Fix issues with researcher wizard flow #1756 (@cyrossignol) - wallet: Stop Error When Starting From Zero #1759 (@RoboticMind) - Don't count empty email as explicit investor #1761 (@cyrossignol) - accrual: Fix snapshot accrual superblock state transitions #1764 (@cyrossignol) - rpc: Cleanup Help Message and Fix Typo #1771 (@RoboticMind) - scraper: Fix scraper etag header case sensitivity #1773 (@cyrossignol) - consensus: Use explicit time to check if superblock needed #1774 (@cyrossignol) - gui: Fix scroll area dark theme styles #1785 (@cyrossignol) - rpc, gui: Fix three divide by zero possibilities #1789 (@jamescowens) - rpc: Fix balance pre-check in "rainbymagnitude" RPC #1792 (@cyrossignol) - accrual: Fix outdated comment and correct grammar #1800 (@RoboticMind) - gui: Fix stuck cursor on labels #1801 (@div72) - beacon: Fix research wizard beacon renewal status #1805 (@cyrossignol) - gui: Fix translations for port numbers #1818 (@cyrossignol) - util: Create parent directory #1821 (@div72) - mining: Fix coinstake/claim signature order #1828 (@cyrossignol) - voting: Remove double increment in loop #1831 (@cyrossignol) - neuralnet, scraper: Fix compilation with gcc5 and older libcurl #1832 (@a123b) - wallet: Fix smallest coin selection for contracts #1841 (@cyrossignol) - gui: Fix display of polls with no votes yet #1844 (@cyrossignol) - gui: add indentation to diagnostic status bar labels #1849 (@jamescowens) - voting, gui: Fix formatting and alignment of vote shares and percent #1850 (@jamescowens) - wallet, rpc: Fix for self-transactions in listtransactions #1852 (@jamescowens) - accrual: Clear any accrual snapshots when syncing from pre-v11 #1853 (@cyrossignol) - accrual: Fix reset of accrual directory if starting sync below research age height #1857 (@jamescowens) - gui: Fix researcher wizard layout on macOS with native theme #1860 (@cyrossignol)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This refactors the
AppCacheBEACONsection into a strongly-typed API for beacon management. It updates the placeholderBeaconRegistryclass included in #1669 into a service that stores and provides access to beacon contract data and adds some new data structures to represent that data. These replace the Base64-encoded, semicolon-delimited string format for runtime storage.This PR introduces a
Beaconclass that implements theIConractPayloadinterface for binary serialization of beacon contracts after block version 11. This payload removes unnecessary fields present in the legacy string-based beacon format and includes just the CPID and public key. The application now uses the public key to produce the addresses used for the rain functions instead of relying on the addresses returned by the problematicDefaultWalletAddress()function. The new format significantly reduces the size of beacon contract messages.This work also removes the
BEACONALTAppCachesection that provided a workaround for an old bug caused by legacy nodes that signed blocks with previous beacon keys. I added exceptions for these historical blocks. This change—and switching the in-memory format for active beacons—reduces the memory overhead of beacon storage by more than 75%.I fixed an issue that caused wallets to repeatedly generate a series of invalid beacon keys when attempting to renew beacons while locked. I also added mitigation for an issue observed on testnet that resulted in exclusion of a CPID from a superblock when that wallet automatically renewed its beacon during the superblock formation period.