build issue (with potential fix) on bigendian architectures #6105
Description
Have you checked borgbackup docs, FAQ, and open Github issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
Bug, seems already resolved in master but present in 1.1 branch (at least no sign of __BIG_ENDIAN__ in setup.py in master)
System information. For client/server mode post info for both machines.
Your borg version (borg -V).
1.1.17
Operating system (distribution) and version.
SynoCommunity, Synology DSM-6 (linux)
Hardware / network configuration, and filesystems used.
PPC qoriq arch
How much data is handled by borg?
n/a - build issue only on PPC due to endianess
Full borg commandline that lead to the problem (leave away excludes and passwords)
n/a
Describe the problem you're observing.
build log:
Collecting borgbackup==1.1.17
Using cached borgbackup-1.1.17.tar.gz (3.8 MB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: borgbackup
Building wheel for borgbackup (setup.py): started
Building wheel for borgbackup (setup.py): finished with status 'error'
ERROR: Command errored out with exit status 1:
command: /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/crossenv/cross/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-t50qub2n/borgbackup_de5df87e60224e6ca41dbf5168a72ac2/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-t50qub2n/borgbackup_de5df87e60224e6ca41dbf5168a72ac2/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-zr688xe_
cwd: /tmp/pip-wheel-t50qub2n/borgbackup_de5df87e60224e6ca41dbf5168a72ac2/
Complete output (125 lines):
Detected and preferring liblz4 over bundled LZ4
Detected and preferring libb2 over bundled BLAKE2
Detected and preferring libzstd over bundled ZSTD
/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/__init__.py:148: SetuptoolsDeprecationWarning: setup_requires is deprecated. Supply build dependencies using PEP 517 pyproject.toml build-requires.
warnings.warn(
running bdist_wheel
running build
running build_py
creating build
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/__main__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/_version.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/archive.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/archiver.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/cache.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/constants.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/fuse.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/helpers.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/locking.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/logger.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/lrucache.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/nanorst.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/patterns.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/remote.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/repository.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/selftest.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/shellpattern.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/upgrader.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/version.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/xattr.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms
copying src/borg/algorithms/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
copying src/borg/crypto/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
copying src/borg/crypto/file_integrity.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
copying src/borg/crypto/key.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
copying src/borg/crypto/keymanager.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
copying src/borg/crypto/nonces.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/crypto
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/platform
copying src/borg/platform/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/platform
copying src/borg/platform/base.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/platform
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/archive.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/archiver.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/benchmark.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/cache.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/checksums.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/chunker.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/compress.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/crypto.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/file_integrity.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/hashindex.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/helpers.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/item.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/key.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/locking.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/logger.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/lrucache.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/nanorst.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/nonces.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/patterns.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/platform.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/remote.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/repository.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/shellpattern.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/upgrader.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/version.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
copying src/borg/testsuite/xattr.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
creating build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms/msgpack
copying src/borg/algorithms/msgpack/__init__.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms/msgpack
copying src/borg/algorithms/msgpack/_version.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms/msgpack
copying src/borg/algorithms/msgpack/exceptions.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms/msgpack
copying src/borg/algorithms/msgpack/fallback.py -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/algorithms/msgpack
running egg_info
writing src/borgbackup.egg-info/PKG-INFO
writing dependency_links to src/borgbackup.egg-info/dependency_links.txt
writing entry points to src/borgbackup.egg-info/entry_points.txt
writing requirements to src/borgbackup.egg-info/requires.txt
writing top-level names to src/borgbackup.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'src/borgbackup.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching '.coafile'
warning: no previously-included files found matching '.editorconfig'
warning: no previously-included files found matching '.gitattributes'
warning: no previously-included files found matching '.gitignore'
warning: no previously-included files found matching '.mailmap'
warning: no previously-included files found matching 'Vagrantfile'
no previously-included directories found matching '.github'
adding license file 'LICENSE'
adding license file 'AUTHORS'
writing manifest file 'src/borgbackup.egg-info/SOURCES.txt'
copying src/borg/paperkey.html -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg
copying src/borg/testsuite/attic.tar.gz -> build/lib.powerpc-e500v2-linux-gnuspe-3.10/borg/testsuite
running build_ext
skipping 'src/borg/algorithms/msgpack/_packer.cpp' Cython extension (up-to-date)
skipping 'src/borg/algorithms/msgpack/_unpacker.cpp' Cython extension (up-to-date)
skipping 'src/borg/compress.c' Cython extension (up-to-date)
skipping 'src/borg/crypto/low_level.c' Cython extension (up-to-date)
skipping 'src/borg/hashindex.c' Cython extension (up-to-date)
skipping 'src/borg/item.c' Cython extension (up-to-date)
skipping 'src/borg/chunker.c' Cython extension (up-to-date)
skipping 'src/borg/algorithms/checksums.c' Cython extension (up-to-date)
skipping 'src/borg/platform/posix.c' Cython extension (up-to-date)
skipping 'src/borg/platform/linux.c' Cython extension (up-to-date)
skipping 'src/borg/platform/syncfilerange.c' Cython extension (up-to-date)
building 'borg.algorithms.msgpack._packer' extension
creating build/temp.powerpc-e500v2-linux-gnuspe-3.10
creating build/temp.powerpc-e500v2-linux-gnuspe-3.10/src
creating build/temp.powerpc-e500v2-linux-gnuspe-3.10/src/borg
creating build/temp.powerpc-e500v2-linux-gnuspe-3.10/src/borg/algorithms
creating build/temp.powerpc-e500v2-linux-gnuspe-3.10/src/borg/algorithms/msgpack
/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -I/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/lib -I /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -L /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/lib -I /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include/python3.10 -I/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install//var/packages/borgbackup/target/include -fPIC -D__LITTLE_ENDIAN__=1 -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include -Isrc/borg/algorithms/xxh64/. -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/crossenv/cross/include -I/home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include/python3.10 -c src/borg/algorithms/msgpack/_packer.cpp -o build/temp.powerpc-e500v2-linux-gnuspe-3.10/src/borg/algorithms/msgpack/_packer.o
In file included from /home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/endian.h:36:0,
from /home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/bits/waitstatus.h:64,
from /home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/stdlib.h:42,
from /home/spksrc/borgbackup-udpate/spksrc/spk/borgbackup/work-qoriq-6.1/install/var/packages/borgbackup/target/include/python3.10/Python.h:34,
from src/borg/algorithms/msgpack/_packer.cpp:6:
/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include/bits/endian.h:26:4: error: #error Both BIG_ENDIAN and LITTLE_ENDIAN defined!
# error Both BIG_ENDIAN and LITTLE_ENDIAN defined!
^
error: command '/home/spksrc/borgbackup-udpate/spksrc/toolchain/syno-qoriq-6.1/work/powerpc-e500v2-linux-gnuspe/bin/powerpc-e500v2-linux-gnuspe-gcc' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for borgbackup
Running setup.py clean for borgbackup
Failed to build borgbackup
ERROR: Failed to build one or more wheels
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Always, only fails on PPC.
Include any warning/errors/backtraces from the system logs Actual patch we use to circumvent this (a workaround upstream part of next release would be really great!)
SynoCommunity/spksrc@72866f6#diff-7b228d5ca562f5bc3b0ebfbbe98d851ed882feab3721fbb04b57ad844256eb78
Disables pseudo logic made to enforce setting of macros which are actually already defined in corresponding headers.
--- setup.py
+++ setup.py
@@ -804,20 +804,16 @@ if not on_rtd:
system_prefix=libb2_prefix, system=libb2_system,
**crypto_ext_kwargs)
- msgpack_endian = '__BIG_ENDIAN__' if (sys.byteorder == 'big') else '__LITTLE_ENDIAN__'
- msgpack_macros = [(msgpack_endian, '1')]
msgpack_packer_ext_kwargs = dict(
sources=[msgpack_packer_source],
include_dirs=include_dirs,
library_dirs=library_dirs,
- define_macros=msgpack_macros,
language='c++',
)
msgpack_unpacker_ext_kwargs = dict(
sources=[msgpack_unpacker_source],
include_dirs=include_dirs,
library_dirs=library_dirs,
- define_macros=msgpack_macros,
language='c++',
)