Skip to content

Conversation

sipsorcery
Copy link
Contributor

@sipsorcery sipsorcery commented Nov 8, 2019

#17364 attempted to save a couple of minutes by skipping the vcpkg steps if the vcpkg install directory was already cached.

The discussion in #15382 highlights the approach used in #17364 does not accommodate adding a new package.

This PR improves the approach to individually check whether each vcpg package is installed rather than checking for the existence of the vcpkg install directory.

This PR moves the list of required vcpkg packages into a separate file and uses changes to that file to invalidate the appveyor cache. Whenever the cache is invalidated the vcpkg sources will be updated, the vcpkg binary built and the required packages installed from the latest port files.

@maflcko
Copy link
Member

maflcko commented Nov 8, 2019

Would it be possible and easier to put the install script into a separate file and invalidate the cache when that script changes?

@sipsorcery
Copy link
Contributor Author

@MarcoFalke yep I'm pretty sure that would be possible. My concern would be starting down the path of an elaborate multi-script build config. That could get scary for us Windows devs.

Is the main goal to have a way of triggering updates without having to touch the appveyor config? What do you think about something simpler like putting the list of packages in a separate text file?

@maflcko
Copy link
Member

maflcko commented Nov 8, 2019

The installation step is already a separate (one-line) script in build_msvc/README.md, so moving it to a file (or just the list of packages to a file) wouldn't make it worse. Also, it moves the hard coded required packages names to a single place.

@sipsorcery sipsorcery force-pushed the check-missing-vcpkgs branch 2 times, most recently from c2db0a5 to d34028f Compare November 8, 2019 18:08
@DrahtBot
Copy link
Contributor

DrahtBot commented Nov 8, 2019

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #15382 (util: add runCommandParseJSON by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipsorcery sipsorcery changed the title Appveyor improvement - adds check for all required vcpkg packages WIP: Appveyor improvement - adds check for all required vcpkg packages Nov 9, 2019
@sipsorcery sipsorcery force-pushed the check-missing-vcpkgs branch 3 times, most recently from ab676f7 to 244a556 Compare November 10, 2019 10:11
@sipsorcery sipsorcery changed the title WIP: Appveyor improvement - adds check for all required vcpkg packages Appveyor improvement - adds check for all required vcpkg packages Nov 10, 2019
@sipsorcery sipsorcery changed the title Appveyor improvement - adds check for all required vcpkg packages Appveyor improvement - text file for vcpkg package list Nov 10, 2019
@sipsorcery
Copy link
Contributor Author

This PR is now ready for review. Title and description have been updated.

@maflcko
Copy link
Member

maflcko commented Nov 11, 2019

ACK 29eb039

maflcko pushed a commit that referenced this pull request Nov 11, 2019
29eb039 Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)

Pull request description:

  #17364 attempted to save a couple of minutes by skipping the `vcpkg` steps if the vcpkg install directory was already cached.

  The discussion in #15382 highlights the approach used in #17364 does not accommodate adding a new package.

  ~~This PR improves the approach to individually check whether  each vcpg package is installed rather than checking for the existence of the vcpkg install directory.~~

  This PR moves the list of required vcpkg packages into a separate file and uses changes to that file to invalidate the appveyor cache. Whenever the cache is invalidated the vcpkg sources will be updated, the vcpkg binary built and the required packages installed from the latest port files.

ACKs for top commit:
  MarcoFalke:
    ACK 29eb039

Tree-SHA512: 0c2a170f4e4b47ca0f9cef14f1e3892001b441a6d84f50bf5fd8a26bc4cdbd9358dfce7ef180d37150262e849650e9857d6b2bcd686964b963c3de6cd708a2f3
@maflcko maflcko merged commit 29eb039 into bitcoin:master Nov 11, 2019
@Sjors Sjors mentioned this pull request Nov 11, 2019
1 task
sipsorcery added a commit to sipsorcery/bitcoin that referenced this pull request Jan 5, 2020
fanquake pushed a commit to fanquake/bitcoin that referenced this pull request Jan 5, 2020
@fanquake fanquake mentioned this pull request Jan 6, 2020
laanwj added a commit that referenced this pull request Jan 8, 2020
99b5407 scripts: fix check-symbols & check-security argument passing (fanquake)
4330a1e Update msvc build for Visual Studio 2019 v16.4 (Aaron Clauson)
b0f9b8e Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)
cd7b3b2 Updated appveyor config:  - Update build image from Visual Studio 2017 to Visual Studio 2019.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added commands to update vcpkg port files (this does not update already installed packages).  - Updated vcpkg package list as per #17309.  - Removed commands setting common project file options. Now done via common.init.vcxproj include.  - Changed msbuild verbosity from normal to quiet. Normal rights a LOT of logs and impacts appveyor job duration. Updated msvc project configs:  - Updated platform toolset from v141 to v142.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added ignore for linker warning building bitcoin-qt program.  - Added missing util/str.cpp class file to test_bitcoin project file. (Aaron Clauson)
112144d Add missing typeinfo includes (Wladimir J. van der Laan)
1a6a534 net: Log to net category for exceptions in ProcessMessages (Wladimir J. van der Laan)
c0dc728 test: fix bitcoind already running warnings on macOS (fanquake)
5276b0e util: Add missing headers to util/fees.cpp (Hennadii Stepanov)
4d7875c rpc: require second argument only for scantxoutset start action (Andrew Chow)
bda2f5b cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice (Harris)
d14ab7c gui: disable File->CreateWallet during startup (fanquake)
b9f1bc0 wallet: unbreak with boost 1.72 (Jan Beich)

Pull request description:

  Backports the following PRs to the 0.19 branch:
  * #17654 - Unbreak build with Boost 1.72.0
  * #17695 - gui: disable File->CreateWallet during startup
  * #17687 - cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice
  * #17728 - rpc: require second argument only for scantxoutset start action
  * #17450 - util: Add missing headers to util/fees.cpp
  * #17488 - test: fix "bitcoind already running" warnings on macOS
  * #17762 - Log to net category for exceptions in ProcessMessages
  * #17364 - Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes
  * #17416 - Appveyor improvement - text file for vcpkg package list
  * #17736 - Update msvc build for Visual Studio 2019 v16.4
  * #17857 - scripts: fix symbol-check & security-check argument passing

  Fixes #17856.

ACKs for top commit:
  sipsorcery:
    ACK (tested: Windows 10 & msvc build) 99b5407.

Tree-SHA512: 91313de56fb0825e70a4be30ba0bf561b8c26d7dcf60549185df4f5e3524099398c828bb46faae807b631634d1afd5a1d397fb41e61ecfa0d746e4bf10b923cb
MarkLTZ added a commit to litecoinz-core/litecoinz that referenced this pull request Mar 13, 2020
[0.19] Backports bitcoin#17858
Unbreak build with Boost 1.72.0 bitcoin#17654
cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice bitcoin#17687
rpc: require second argument only for scantxoutset start action bitcoin#17728
wallet: Fix origfee return for bumpfee with feerate arg bitcoin#17643
test: fix "bitcoind already running" warnings on macOS bitcoin#17488
net: Log to net category for exceptions in ProcessMessages bitcoin#17762
Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes bitcoin#17364
Appveyor improvement - text file for vcpkg package list bitcoin#17416
Update msvc build for Visual Studio 2019 v16.4 bitcoin#17736
scripts: fix symbol-check & security-check argument passing bitcoin#17857
qt: Periodic translations update for 0.19 branch
IsUsedDestination should count any known single-key address bitcoin#17621
init: Stop indexes on shutdown after ChainStateFlushed callback. bitcoin#17897
qt: Translations update pre-rc1
wallet: Reset reused transactions cache bitcoin#17843
Squashed 'src/univalue/' changes from 7890db9..98261b1
0.19: Update univalue subtree bitcoin#18100
qt: Pre-rc2 translations update
MarkLTZ added a commit to litecoinz-core/litecoinz that referenced this pull request Mar 13, 2020
[0.19] Backports bitcoin#17858
Unbreak build with Boost 1.72.0 bitcoin#17654
cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice bitcoin#17687
rpc: require second argument only for scantxoutset start action bitcoin#17728
wallet: Fix origfee return for bumpfee with feerate arg bitcoin#17643
test: fix "bitcoind already running" warnings on macOS bitcoin#17488
net: Log to net category for exceptions in ProcessMessages bitcoin#17762
Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes bitcoin#17364
Appveyor improvement - text file for vcpkg package list bitcoin#17416
Update msvc build for Visual Studio 2019 v16.4 bitcoin#17736
scripts: fix symbol-check & security-check argument passing bitcoin#17857
qt: Periodic translations update for 0.19 branch
IsUsedDestination should count any known single-key address bitcoin#17621
init: Stop indexes on shutdown after ChainStateFlushed callback. bitcoin#17897
qt: Translations update pre-rc1
wallet: Reset reused transactions cache bitcoin#17843
Squashed 'src/univalue/' changes from 7890db9..98261b1
0.19: Update univalue subtree bitcoin#18100
qt: Pre-rc2 translations update
[0.19] Further 0.19 backports bitcoin#18218
build: don't embed a build-id when building libdmg-hfsplus bitcoin#18004
linuxsh2 pushed a commit to linuxsh2/dash that referenced this pull request Sep 16, 2021
…ge list

29eb039 Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)

Pull request description:

  bitcoin#17364 attempted to save a couple of minutes by skipping the `vcpkg` steps if the vcpkg install directory was already cached.

  The discussion in bitcoin#15382 highlights the approach used in bitcoin#17364 does not accommodate adding a new package.

  ~~This PR improves the approach to individually check whether  each vcpg package is installed rather than checking for the existence of the vcpkg install directory.~~

  This PR moves the list of required vcpkg packages into a separate file and uses changes to that file to invalidate the appveyor cache. Whenever the cache is invalidated the vcpkg sources will be updated, the vcpkg binary built and the required packages installed from the latest port files.

ACKs for top commit:
  MarcoFalke:
    ACK 29eb039

Tree-SHA512: 0c2a170f4e4b47ca0f9cef14f1e3892001b441a6d84f50bf5fd8a26bc4cdbd9358dfce7ef180d37150262e849650e9857d6b2bcd686964b963c3de6cd708a2f3
Munkybooty pushed a commit to Munkybooty/dash that referenced this pull request Dec 9, 2021
99b5407 scripts: fix check-symbols & check-security argument passing (fanquake)
4330a1e Update msvc build for Visual Studio 2019 v16.4 (Aaron Clauson)
b0f9b8e Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)
cd7b3b2 Updated appveyor config:  - Update build image from Visual Studio 2017 to Visual Studio 2019.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added commands to update vcpkg port files (this does not update already installed packages).  - Updated vcpkg package list as per bitcoin#17309.  - Removed commands setting common project file options. Now done via common.init.vcxproj include.  - Changed msbuild verbosity from normal to quiet. Normal rights a LOT of logs and impacts appveyor job duration. Updated msvc project configs:  - Updated platform toolset from v141 to v142.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added ignore for linker warning building bitcoin-qt program.  - Added missing util/str.cpp class file to test_bitcoin project file. (Aaron Clauson)
112144d Add missing typeinfo includes (Wladimir J. van der Laan)
1a6a534 net: Log to net category for exceptions in ProcessMessages (Wladimir J. van der Laan)
c0dc728 test: fix bitcoind already running warnings on macOS (fanquake)
5276b0e util: Add missing headers to util/fees.cpp (Hennadii Stepanov)
4d7875c rpc: require second argument only for scantxoutset start action (Andrew Chow)
bda2f5b cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice (Harris)
d14ab7c gui: disable File->CreateWallet during startup (fanquake)
b9f1bc0 wallet: unbreak with boost 1.72 (Jan Beich)

Pull request description:

  Backports the following PRs to the 0.19 branch:
  * bitcoin#17654 - Unbreak build with Boost 1.72.0
  * bitcoin#17695 - gui: disable File->CreateWallet during startup
  * bitcoin#17687 - cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice
  * bitcoin#17728 - rpc: require second argument only for scantxoutset start action
  * bitcoin#17450 - util: Add missing headers to util/fees.cpp
  * bitcoin#17488 - test: fix "bitcoind already running" warnings on macOS
  * bitcoin#17762 - Log to net category for exceptions in ProcessMessages
  * bitcoin#17364 - Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes
  * bitcoin#17416 - Appveyor improvement - text file for vcpkg package list
  * bitcoin#17736 - Update msvc build for Visual Studio 2019 v16.4
  * bitcoin#17857 - scripts: fix symbol-check & security-check argument passing

  Fixes bitcoin#17856.

ACKs for top commit:
  sipsorcery:
    ACK (tested: Windows 10 & msvc build) 99b5407.

Tree-SHA512: 91313de56fb0825e70a4be30ba0bf561b8c26d7dcf60549185df4f5e3524099398c828bb46faae807b631634d1afd5a1d397fb41e61ecfa0d746e4bf10b923cb
Munkybooty pushed a commit to Munkybooty/dash that referenced this pull request Dec 9, 2021
99b5407 scripts: fix check-symbols & check-security argument passing (fanquake)
4330a1e Update msvc build for Visual Studio 2019 v16.4 (Aaron Clauson)
b0f9b8e Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)
cd7b3b2 Updated appveyor config:  - Update build image from Visual Studio 2017 to Visual Studio 2019.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added commands to update vcpkg port files (this does not update already installed packages).  - Updated vcpkg package list as per bitcoin#17309.  - Removed commands setting common project file options. Now done via common.init.vcxproj include.  - Changed msbuild verbosity from normal to quiet. Normal rights a LOT of logs and impacts appveyor job duration. Updated msvc project configs:  - Updated platform toolset from v141 to v142.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added ignore for linker warning building bitcoin-qt program.  - Added missing util/str.cpp class file to test_bitcoin project file. (Aaron Clauson)
112144d Add missing typeinfo includes (Wladimir J. van der Laan)
1a6a534 net: Log to net category for exceptions in ProcessMessages (Wladimir J. van der Laan)
c0dc728 test: fix bitcoind already running warnings on macOS (fanquake)
5276b0e util: Add missing headers to util/fees.cpp (Hennadii Stepanov)
4d7875c rpc: require second argument only for scantxoutset start action (Andrew Chow)
bda2f5b cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice (Harris)
d14ab7c gui: disable File->CreateWallet during startup (fanquake)
b9f1bc0 wallet: unbreak with boost 1.72 (Jan Beich)

Pull request description:

  Backports the following PRs to the 0.19 branch:
  * bitcoin#17654 - Unbreak build with Boost 1.72.0
  * bitcoin#17695 - gui: disable File->CreateWallet during startup
  * bitcoin#17687 - cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice
  * bitcoin#17728 - rpc: require second argument only for scantxoutset start action
  * bitcoin#17450 - util: Add missing headers to util/fees.cpp
  * bitcoin#17488 - test: fix "bitcoind already running" warnings on macOS
  * bitcoin#17762 - Log to net category for exceptions in ProcessMessages
  * bitcoin#17364 - Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes
  * bitcoin#17416 - Appveyor improvement - text file for vcpkg package list
  * bitcoin#17736 - Update msvc build for Visual Studio 2019 v16.4
  * bitcoin#17857 - scripts: fix symbol-check & security-check argument passing

  Fixes bitcoin#17856.

ACKs for top commit:
  sipsorcery:
    ACK (tested: Windows 10 & msvc build) 99b5407.

Tree-SHA512: 91313de56fb0825e70a4be30ba0bf561b8c26d7dcf60549185df4f5e3524099398c828bb46faae807b631634d1afd5a1d397fb41e61ecfa0d746e4bf10b923cb
Munkybooty pushed a commit to Munkybooty/dash that referenced this pull request Dec 9, 2021
99b5407 scripts: fix check-symbols & check-security argument passing (fanquake)
4330a1e Update msvc build for Visual Studio 2019 v16.4 (Aaron Clauson)
b0f9b8e Moves vcpkg list to a text file and updates the appveyor job and readme to use it. (Aaron Clauson)
cd7b3b2 Updated appveyor config:  - Update build image from Visual Studio 2017 to Visual Studio 2019.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added commands to update vcpkg port files (this does not update already installed packages).  - Updated vcpkg package list as per bitcoin#17309.  - Removed commands setting common project file options. Now done via common.init.vcxproj include.  - Changed msbuild verbosity from normal to quiet. Normal rights a LOT of logs and impacts appveyor job duration. Updated msvc project configs:  - Updated platform toolset from v141 to v142.  - Updated Qt static library from Qt5.9.7 to Qt5.9.8.  - Added ignore for linker warning building bitcoin-qt program.  - Added missing util/str.cpp class file to test_bitcoin project file. (Aaron Clauson)
112144d Add missing typeinfo includes (Wladimir J. van der Laan)
1a6a534 net: Log to net category for exceptions in ProcessMessages (Wladimir J. van der Laan)
c0dc728 test: fix bitcoind already running warnings on macOS (fanquake)
5276b0e util: Add missing headers to util/fees.cpp (Hennadii Stepanov)
4d7875c rpc: require second argument only for scantxoutset start action (Andrew Chow)
bda2f5b cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice (Harris)
d14ab7c gui: disable File->CreateWallet during startup (fanquake)
b9f1bc0 wallet: unbreak with boost 1.72 (Jan Beich)

Pull request description:

  Backports the following PRs to the 0.19 branch:
  * bitcoin#17654 - Unbreak build with Boost 1.72.0
  * bitcoin#17695 - gui: disable File->CreateWallet during startup
  * bitcoin#17687 - cli: fix Fatal LevelDB error when specifying -blockfilterindex=basic twice
  * bitcoin#17728 - rpc: require second argument only for scantxoutset start action
  * bitcoin#17450 - util: Add missing headers to util/fees.cpp
  * bitcoin#17488 - test: fix "bitcoind already running" warnings on macOS
  * bitcoin#17762 - Log to net category for exceptions in ProcessMessages
  * bitcoin#17364 - Updates to appveyor config for VS2019 and Qt5.9.8 + msvc project fixes
  * bitcoin#17416 - Appveyor improvement - text file for vcpkg package list
  * bitcoin#17736 - Update msvc build for Visual Studio 2019 v16.4
  * bitcoin#17857 - scripts: fix symbol-check & security-check argument passing

  Fixes bitcoin#17856.

ACKs for top commit:
  sipsorcery:
    ACK (tested: Windows 10 & msvc build) 99b5407.

Tree-SHA512: 91313de56fb0825e70a4be30ba0bf561b8c26d7dcf60549185df4f5e3524099398c828bb46faae807b631634d1afd5a1d397fb41e61ecfa0d746e4bf10b923cb
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants