Skip to content

Conversation

fanquake
Copy link
Member

@fanquake fanquake commented Mar 16, 2020

Something has changed in the mingw-w64 headers such that we
no-longer compile when using 7.0.0.

util/time.cpp: In function 'std::__cxx11::string FormatISO8601DateTime(int64_t)':
util/time.cpp:84:9: error: 'gmtime_r' was not declared in this scope
     if (gmtime_r(&time_val, &ts) == nullptr) {
         ^~~~~~~~
util/time.cpp: In function 'std::__cxx11::string FormatISO8601Date(int64_t)':
util/time.cpp:97:9: error: 'gmtime_r' was not declared in this scope
     if (gmtime_r(&time_val, &ts) == nullptr) {

Looking at time.h, it seems that gmtime_r() is only available when
_POSIX_C_SOURCE is defined. This must have been the case for 6.0.0
(which we compile fine using), but no-longer seems to be for 7.0.0?

I've checked that adding -D_POSIX_C_SOURCE=200112L to our compile
flags does fix the issue above.

However, an alternative solution seems to be to just use gmtime_s()
instead, when compiling with mingw-w64, as gmtime_r() just wraps
gmtime_s() anyways
.

I've tested this change crosss-compiling on Debian Bullseye (mingw-w64 7.0.0)
and Buster (mingw-w64 6.0.0).

@hebasto
Copy link
Member

hebasto commented Mar 16, 2020

Concept ACK.

@hebasto
Copy link
Member

hebasto commented Mar 16, 2020

However, an alternative solution seems to be to just use gmtime_s()
instead, when compiling with mingw-w64, as gmtime_r() just wraps gmtime_s() anyways.

Does this alternative solution has any downsides?

@practicalswift
Copy link
Contributor

Concept ACK

@DrahtBot
Copy link
Contributor

Gitian builds

File commit 8662387
(master)
commit 5794c4e
(master and this pull)
bitcoin-0.19.99-aarch64-linux-gnu-debug.tar.gz 184a2e3f976d1afe... 69032278b4fb3194...
bitcoin-0.19.99-aarch64-linux-gnu.tar.gz aa5c3d4eb36052a2... e6af093a3940130b...
bitcoin-0.19.99-arm-linux-gnueabihf-debug.tar.gz 88a802223576828c... cdcd2eaf388b7995...
bitcoin-0.19.99-arm-linux-gnueabihf.tar.gz f5f9101a04efcbc4... 7f05a1cb48114e2e...
bitcoin-0.19.99-osx-unsigned.dmg 4e7a5cafd7f83e18... 8a1c33df35629b92...
bitcoin-0.19.99-osx64.tar.gz c8d48a19665cd75e... 8497d3b315cb488e...
bitcoin-0.19.99-riscv64-linux-gnu-debug.tar.gz c12106ddc8c14bb1... f7c7ae68c0e5c8cf...
bitcoin-0.19.99-riscv64-linux-gnu.tar.gz 3e860dcefdc35267... f2d979e190f23d7d...
bitcoin-0.19.99-win64-debug.zip 967c32a4058a802b... 2745f915464f60be...
bitcoin-0.19.99-win64-setup-unsigned.exe 8254f4e5a1f09c6a... d5298e31d6527c3e...
bitcoin-0.19.99-win64.zip d5fb1ac9cc809c7c... f08f040863159612...
bitcoin-0.19.99-x86_64-linux-gnu-debug.tar.gz 5644bfd3c6efa7ba... 3a8ebf737fc31954...
bitcoin-0.19.99-x86_64-linux-gnu.tar.gz 0d4ba03883044861... 6a94bc2d5266cdf9...
bitcoin-0.19.99.tar.gz 3abdc69dfe4d5f23... 48d1a69f4afb58a6...
bitcoin-core-linux-0.20-res.yml f3815dcc9396f613... c5c21257a6a86019...
bitcoin-core-osx-0.20-res.yml 9e00cf2496e87ee7... e09498c9b06fd794...
bitcoin-core-win-0.20-res.yml acd91b8db1409e99... 69ca4551b3c51e2e...
linux-build.log 08ce938f565b6375... 062edb1ef74dbde8...
osx-build.log c7851524186d3b65... 1452c13376551126...
win-build.log 3f7133b0888ce7a0... 92a02991bb54c1b5...
bitcoin-core-linux-0.20-res.yml.diff 0fe8e86861a8f3d9...
bitcoin-core-osx-0.20-res.yml.diff d4bbb2d32700c589...
bitcoin-core-win-0.20-res.yml.diff 9970ee113bb8bb83...
linux-build.log.diff 50af512810c062fc...
osx-build.log.diff 8cfc3a33373d8b5c...
win-build.log.diff 78f307674781bbb3...

@dongcarl
Copy link
Contributor

Did a bit more digging. Here's what I found:

This commmit is the one that caused the change in behaviour.

There's a very short mailing list thread about it here.

My conversation with the author of the commit on IRC:

15:02 <dongcarl> jacekc: After your "Don't define _POSIX_THREAD_SAFE_FUNCTIONS." patch, which landed in 7.0.0, what is the right way to get `gmtime_r` in my application?
15:02 * dongcarl waits patiently
15:04 <jacekc> dongcarl: ideally, I'd suggest to use gmtime_s
15:04 <dongcarl> jacekc: Why's that?
15:05 <jacekc> because that's what the platform you target offers
15:06 <jacekc> but if you prefer mingw wrappers, you may define _POSIX_THREAD_SAFE_FUNCTIONS or _POSIX_C_SOURCE
15:07 <dongcarl> jacekc: I think I see what you're saying. I'm assuming the mingw wrappers don't offer thread safety, even though the name is gmtime_r?
15:07 * dongcarl is thankful for the patient response
15:09 <jacekc> both _s and _r variants offer thread safety
15:09 <jacekc> _r variant is just an inline wrapper in time.h aroud _s
15:12 <dongcarl> jacekc: I see. Out of curiosity, what's the difference between defining _POSIX_THREAD_SAFE_FUNCTIONS and _POSIX_C_SOURCE? Feel free to just send me a link to read
15:14 <jacekc> _POSIX_THREAD_SAFE_FUNCTIONS enables only a few inline functions in time.h, _POSIX_C_SOURCE enables much more (most notably mongw stdio implementation)
15:15 <jacekc> I don't know if it's documented anywhere, but it's easy to find in the sources
15:16 <dongcarl> jacekc: Thanks for your help. Very much appreciated and have a tremendous day!
15:16 <jacekc> you welcome

Here's some more general use information on _POSIX_C_SOURCE and _POSIX_THREAD_SAFE_FUNCTIONS.

@maflcko
Copy link
Member

maflcko commented Mar 17, 2020

Looks like this makes the include order of headers more resilient and fixes a bug I was running in to a year ago: https://github.com/bitcoin/bitcoin/pull/16046/files#r285601479

If you feel like, you might sort the includes in the header now:

diff --git a/src/util/time.h b/src/util/time.h
index 77de1e047d..a8c06ef350 100644
--- a/src/util/time.h
+++ b/src/util/time.h
@@ -6,9 +6,9 @@
 #ifndef BITCOIN_UTIL_TIME_H
 #define BITCOIN_UTIL_TIME_H
 
+#include <chrono>
 #include <stdint.h>
 #include <string>
-#include <chrono>
 
 void UninterruptibleSleep(const std::chrono::microseconds& n);
 

@dongcarl
Copy link
Contributor

If I'm thinking about this the right way, your diff implies that previously the #ifdef _MSC_VER would fail, and branch to the #else, which used gmtime_r, which would be fine because it was defined by default in mingw-w64.

If #ifdef _MSC_VER is how we detect if we're building for windows, and it's failing, then perhaps there are other places in the codebase where this detection needs to be fixed?

@fanquake fanquake force-pushed the mingw_w64_gmtime_s branch from e8ca7bd to 1b801e7 Compare March 18, 2020 09:21
@fanquake
Copy link
Member Author

If you feel like, you might sort the includes in the header now:

Done.

If #ifdef _MSC_VER is how we detect if we're building for windows, and it's failing, then perhaps there are other places in the codebase where this detection needs to be fixed?

I've looked into this, and there are some cases where we are using _MSC_VER where we might be running into unexpected behavior. I don't necessarily want to address that as part of this PR, but will definitely be following up.

Copy link
Member

@maflcko maflcko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hebasto
Copy link
Member

hebasto commented Mar 18, 2020

Something has changed in the mingw-w64 headers such that we
no-longer compile when using 7.0.0.

Interesting why master could be built successfully with Ubuntu's g++-mingw-w64 7.3.0?

@fanquake
Copy link
Member Author

Interesting why master could be built successfully with Ubuntu's g++-mingw-w64 7.3.0?

The version of mingw-w64 being used is 5.0.0. 7.3.0 is the version of GCC.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 1b801e7, tested on Ubuntu 20.04 with mingw-w64 7.0.0.

@dongcarl
Copy link
Contributor

From #bitcoin-builds meeting notes today: We should investigate how to detect features (i.e. check for function availability at configure time), rather than detecting platforms. This might be more robust.

@maflcko
Copy link
Member

maflcko commented Mar 18, 2020

@dongcarl Is this a general advise/issue or a blocker on this specific patch set?

@dongcarl
Copy link
Contributor

@MarcoFalke I think it's a blocker while we investigate if feature detection is the right path take going forward, no point merging this then changing right after (unless the release schedule dictates?)

@Empact
Copy link
Contributor

Empact commented Mar 18, 2020

Here's an alternative: master...Empact:2020-03-gmtime-config

Note I haven't run it on windows/mingw, and I'm not sure how to get the test to distinguish between MSC-style gmtime_s and C-standard-style gmtime_s - i.e. I doubt the existing casting-based approach.

Edit: and another more succinct alternative: master...Empact:2020-03-gmtime-check-funcs

Inspired by sqlite's handling of the same question: https://github.com/sqlite/sqlite/blob/118efd162632298bccba21b71934f666e556f594/configure.ac#L111

Although they fall back to gmtime, so it's not directly comparable.

@sipsorcery
Copy link
Contributor

ACK 1b801e7.

@laanwj
Copy link
Member

laanwj commented Mar 25, 2020

Using the windows function on windows (instead of mingw's wrapper) makes sense, imo.

@@ -78,7 +78,7 @@ int64_t GetSystemTimeInSeconds()
std::string FormatISO8601DateTime(int64_t nTime) {
struct tm ts;
time_t time_val = nTime;
#ifdef _MSC_VER
#if defined(_MSC_VER) || defined(__MINGW64_VERSION_MAJOR)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we change this to #ifdef WIN32 ?

@laanwj laanwj added this to the 0.20.0 milestone Mar 25, 2020
@dongcarl
Copy link
Contributor

dongcarl commented Apr 1, 2020

Hmmm, I think perhaps what we want to do is similar to what's being done here: https://github.com/maru/libmicrohttpd-http2/blob/316af7c06537e524c6ef02bc1d5f59b39af7d1d1/configure.ac#L1033-L1082

It's feature-based detection, and we'll have clarity between HAVE_C11_GMTIME_S, HAVE_W32_GMTIME_S, and HAVE_GMTIME_R.

@laanwj
Copy link
Member

laanwj commented Apr 1, 2020

Yeah, that's another way.
I wish this kind of basic functionality was standardized by now. This comes up so often.


If we still want this for 0.20, we need to make a decision here quickly though. I'm sorry for starting a bikeshed 😆

This improves the portability of the codebase and fixes compilation
with mingw-w64 7.0+.

Co-authored-by: fanquake <fanquake@gmail.com>
@fanquake
Copy link
Member Author

fanquake commented Apr 2, 2020

doesn't actually get printed in the gmtime_s success case.

Maybe I was just imaging this.. Have pushed up some new changes.

Tested on macOS:

checking for gmtime_r... yes

mingw-w64 cross compile:

checking for gmtime_r... no
checking for gmtime_s... yes

debian:

checking for gmtime_r... yes

MSVC ✅

@Empact
Copy link
Contributor

Empact commented Apr 2, 2020

Note one thing I have not tested with this implementation is how it operates if gmtime_r is not present, but C11-style gmtime_s is.

Copy link
Member

@hebasto hebasto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested a46484c on focal and bionic. Both produce:

$ CONFIG_SITE=$PWD/depends/x86_64-w64-mingw32/share/config.site ./configure
...
checking for gmtime_r... no
checking for gmtime_s... yes
...

To make things clear about cross compiling on systems with pre-7.0 mingw-w64 headers, e.g., on bionic:
on master the gmtime_r() is used, and with this PR the gmtime_s() is used, right?

@laanwj
Copy link
Member

laanwj commented Apr 2, 2020

ACK a46484c

@laanwj laanwj merged commit 5c1ba3a into bitcoin:master Apr 2, 2020
@theuni
Copy link
Member

theuni commented Apr 2, 2020

Post-merge ACK. Thanks for making this test-based, I think this is a nice improvement.

@fanquake fanquake deleted the mingw_w64_gmtime_s branch April 3, 2020 00:05
@fanquake
Copy link
Member Author

fanquake commented Apr 3, 2020

To make things clear about cross compiling on systems with pre-7.0 mingw-w64 headers, e.g., on bionic:

@hebasto the breakdown is:

commit Debian Buster (mingw-w64 6.0.0) Debian Bullseye (mingw-w64 7.0.0)
5c1ba3a~1 gmtime_r() fails to compile
5c1ba3a gmtime_s() gmtime_s()

random-zebra added a commit to PIVX-Project/PIVX that referenced this pull request May 5, 2021
f3f3d6c util: fix compilation with mingw-w64 7.0.0 (Fuzzbawls)
f24da7d util: Avoid potential uninitialized read in FormatISO8601DateTime (Fuzzbawls)
120a12a util: Add type safe GetTime (Fuzzbawls)
be48766 Fix for utiltime to compile with msvc. (Aaron Clauson)
2833406 Avoid std::locale/imbue in DateTimeStrFormat (Pieter Wuille)
3331c25 Format automatic wallet backup file names in ISO 8601 basic format (Fuzzbawls)
c32a208 Format timestamps using ISO 8601 formatting (e.g. "2018-02-28T12:34:56Z") (practicalswift)
f8bd173 [logging] log system time and mock time (John Newbery)

Pull request description:

  This is a backport of a collection of upstream PRs to bring our time utility more up to date. The following upstream PRs are included:

  - bitcoin#10383
  - bitcoin#12567
  - bitcoin#12973
  - bitcoin#13031
  - bitcoin#16046
  - bitcoin#18162
  - bitcoin#18358

  The two user-facing notable changes here have been documented in the release notes template. I've also connected the functionality that `-logtimemicros` was supposed to provide.

ACKs for top commit:
  furszy:
    code review ACK f3f3d6c
  random-zebra:
    utACK f3f3d6c after rebase, and merging...

Tree-SHA512: 64f9cc1f7fc65c192f3b3ab869c057a81e007cf0fae82ecd23993c6b51830e4bc80656c699ba40bb8e019115a24d1ea88a618be0c0d7112749d9ce125d62ce44
kwvg added a commit to kwvg/dash that referenced this pull request Jul 4, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 4, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 9, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 9, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 9, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 13, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 15, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 15, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Jul 16, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 1, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 5, 2021
kwvg added a commit to kwvg/dash that referenced this pull request Aug 5, 2021
5tefan pushed a commit to 5tefan/dash that referenced this pull request Aug 12, 2021
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants