free(): invalid pointer crash on specific import order (on ARM64 linux) #26279
Description
What version of gRPC and what language are you using?
grpcio==1.37.1
What operating system (Linux, Windows,...) and version?
linux Docker container on a macOS Silicon (M1) laptop.
What runtime / compiler are you using (e.g. python version or version of gcc)
% docker run -ti --rm python:3.9.5-slim-buster python3 --version
Python 3.9.5What did you do?
This is a followup from a GitHub issue at googleapis/python-pubsub.
The googleapis/python-pubsub library uses grpcio as a dependency.
However, we've noticed that importing grpcio results in a static variable initialisation/destruction problem around the following line:
#15 0x0000ffff879273e0 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
To reproduce:
root@fa796b3a1f02:/# python3 -c 'import proto.message, grpc;'
free(): invalid pointer
Aborted (core dumped)
root@fa796b3a1f02:/# python3 -c 'import grpc, proto.message'
# no crashGDB
root@fa796b3a1f02:/# gdb /usr/local/bin/python3 /core
...
Reading symbols from /usr/local/bin/python3...(no debugging symbols found)...done.
warning: core file may not match specified executable file.
[New LWP 15]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
Core was generated by `python3 -c import proto.message, grpc;'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000ffff88b948e8 in __GI_abort () at abort.c:79
#2 0x0000ffff88be06f8 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0xffff88ca0bd8 "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3 0x0000ffff88be6cb4 in malloc_printerr (str=str@entry=0xffff88c9c5d8 "free(): invalid pointer") at malloc.c:5341
#4 0x0000ffff88be856c in _int_free (av=0xffff88ce0a98 <main_arena>, p=0xffff87fd2950 <(anonymous namespace)::ctype_c+560>, have_lock=0)
at malloc.c:4165
#5 0x0000ffff87e5a948 in std::locale::_Impl::_M_install_facet (this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>, __idp=<optimized out>,
__fp=0xffff87fd22e0 <(anonymous namespace)::numpunct_c>) at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale.cc:352
#6 0x0000ffff87e45df0 in std::locale::_Impl::_M_init_facet<std::numpunct<char> > (__facet=0xffff87fd22e0 <(anonymous namespace)::numpunct_c>,
this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>)
at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/bits/locale_classes.h:564
#7 std::locale::_Impl::_Impl (this=0xffff87fd2248 <(anonymous namespace)::c_locale_impl>, __refs=<optimized out>)
at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:402
#8 0x0000ffff87e46710 in std::locale::_S_initialize_once ()
at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:267
#9 0x0000ffff88dda250 in __pthread_once_slow (once_control=0xffff87fd3668 <std::locale::_S_once>,
init_routine=0xffff87e466f0 <std::locale::_S_initialize_once()>) at pthread_once.c:116
#10 0x0000ffff87e4675c in __gthread_once (__once=<optimized out>, __func=0xffff87e466f0 <std::locale::_S_initialize_once()>)
at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/aarch64-unknown-linux-gnueabi/bits/gthr-default.h:699
#11 std::locale::_S_initialize () at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:276
#12 0x0000ffff87e46794 in std::locale::locale (this=0xffff87fe63c0 <__gnu_internal::buf_cout_sync+56>)
at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/locale_init.cc:210
#13 0x0000ffff87e60510 in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::stdio_sync_filebuf (
this=0xffff87fe6388 <__gnu_internal::buf_cout_sync>, __f=0xffff87f70a30)
at /dockcross/crosstool/toolchain/.build/aarch64-unknown-linux-gnueabi/build/build-cc-gcc-final/aarch64-unknown-linux-gnueabi/libstdc++-v3/include/ext/stdio_sync_filebuf.h:77
#14 0x0000ffff87e59b24 in std::ios_base::Init::Init (this=<optimized out>)
at /dockcross/crosstool/toolchain/.build/src/gcc-4.9.4/libstdc++-v3/src/c++98/ios_init.cc:85
#15 0x0000ffff879273e0 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535)
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/xcc/aarch64-unknown-linux-gnueabi/aarch64-unknown-linux-gnueabi/include/c++/4.9.4/iostream:74
#16 _GLOBAL__sub_I_grpc_context.cc(void) () at src/core/ext/filters/census/grpc_context.cc:38
#17 0x0000ffff8920093c in call_init (l=<optimized out>, argc=argc@entry=3, argv=argv@entry=0xfffffa0a87d8, env=0xfffffa0a87f8) at dl-init.c:72
#18 0x0000ffff89200a40 in call_init (env=0xfffffa0a87f8, argv=0xfffffa0a87d8, argc=3, l=<optimized out>) at dl-init.c:30
#19 _dl_init (main_map=main_map@entry=0xaaaac9a97900, argc=3, argv=0xfffffa0a87d8, env=0xfffffa0a87f8) at dl-init.c:119
#20 0x0000ffff89204750 in dl_open_worker (a=a@entry=0xfffffa0a5140) at dl-open.c:517
#21 0x0000ffff88c7de58 in __GI__dl_catch_exception (exception=exception@entry=0xfffffa0a5128, operate=operate@entry=0xffff89204430 <dl_open_worker>,
args=args@entry=0xfffffa0a5140) at dl-error-skeleton.c:196
#22 0x0000ffff89204098 in _dl_open (file=0xffff88037c50 "/usr/local/lib/python3.9/site-packages/grpc/_cython/cygrpc.cpython-39-aarch64-linux-gnu.so",
mode=-2147483646, caller_dlopen=0xffff89071de0, nsid=-2, argc=3, argv=<optimized out>, env=<optimized out>) at dl-open.c:599
#23 0x0000ffff88db8094 in dlopen_doit (a=a@entry=0xfffffa0a53f8) at dlopen.c:66
#24 0x0000ffff88c7de58 in __GI__dl_catch_exception (exception=exception@entry=0xfffffa0a5370, operate=operate@entry=0xffff88db8030 <dlopen_doit>,
args=args@entry=0xfffffa0a53f8) at dl-error-skeleton.c:196
#25 0x0000ffff88c7df00 in __GI__dl_catch_error (objname=objname@entry=0xaaaac98db5b0, errstring=errstring@entry=0xaaaac98db5b8,
mallocedp=mallocedp@entry=0xaaaac98db5a8, operate=operate@entry=0xffff88db8030 <dlopen_doit>, args=args@entry=0xfffffa0a53f8)
at dl-error-skeleton.c:215
#26 0x0000ffff88db87e0 in _dlerror_run (operate=operate@entry=0xffff88db8030 <dlopen_doit>, args=args@entry=0xfffffa0a53f8) at dlerror.c:163
#27 0x0000ffff88db8134 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#28 0x0000ffff89071de0 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#29 0x0000ffff89060104 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#30 0x0000ffff8905fd60 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#31 0x0000ffff88f5bd54 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#32 0x0000ffff88f2a138 in PyVectorcall_Call () from /usr/local/lib/libpython3.9.so.1.0
#33 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#34 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#35 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#36 0x0000ffff88fb08d0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#37 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#38 0x0000ffff88facfac in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#39 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#40 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#41 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#42 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#43 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#44 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#45 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#46 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#47 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#48 0x0000ffff88fcd0f4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#49 0x0000ffff88fa9674 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#50 0x0000ffff88f5be88 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#51 0x0000ffff88f29ff0 in _PyObject_Call () from /usr/local/lib/libpython3.9.so.1.0
#52 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#53 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#54 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#55 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#56 0x0000ffff88fabe98 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#57 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#58 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#59 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#60 0x0000ffff88fcd1d4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#61 0x0000ffff88fb057c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#62 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#63 0x0000ffff88fab69c in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.9.so.1.0
#64 0x0000ffff88fab628 in PyEval_EvalCodeEx () from /usr/local/lib/libpython3.9.so.1.0
#65 0x0000ffff88fab5d4 in PyEval_EvalCode () from /usr/local/lib/libpython3.9.so.1.0
#66 0x0000ffff88fa9254 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#67 0x0000ffff88f5bd54 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#68 0x0000ffff88f2a138 in PyVectorcall_Call () from /usr/local/lib/libpython3.9.so.1.0
#69 0x0000ffff88fae9dc in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#70 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#71 0x0000ffff88f29224 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
--Type <RET> for more, q to quit, c to continue without paging--
#72 0x0000ffff88fb08d0 in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#73 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#74 0x0000ffff88facfac in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#75 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#76 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#77 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#78 0x0000ffff88facd7c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#79 0x0000ffff88f29168 in _PyFunction_Vectorcall () from /usr/local/lib/libpython3.9.so.1.0
#80 0x0000ffff88f28c08 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#81 0x0000ffff88f2a488 in _PyObject_CallMethodIdObjArgs () from /usr/local/lib/libpython3.9.so.1.0
#82 0x0000ffff88fcd0f4 in PyImport_ImportModuleLevelObject () from /usr/local/lib/libpython3.9.so.1.0
#83 0x0000ffff88fb057c in _PyEval_EvalFrameDefault () from /usr/local/lib/libpython3.9.so.1.0
#84 0x0000ffff88fab964 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#85 0x0000ffff88fab69c in _PyEval_EvalCodeWithName () from /usr/local/lib/libpython3.9.so.1.0
#86 0x0000ffff88fab628 in PyEval_EvalCodeEx () from /usr/local/lib/libpython3.9.so.1.0
#87 0x0000ffff88fab5d4 in PyEval_EvalCode () from /usr/local/lib/libpython3.9.so.1.0
#88 0x0000ffff8906b1c8 in ?? () from /usr/local/lib/libpython3.9.so.1.0
#89 0x0000ffff8906b10c in ?? () from /usr/local/lib/libpython3.9.so.1.0
#90 0x0000ffff8906b05c in PyRun_StringFlags () from /usr/local/lib/libpython3.9.so.1.0
#91 0x0000ffff8906af78 in PyRun_SimpleStringFlags () from /usr/local/lib/libpython3.9.so.1.0
#92 0x0000ffff89073fc4 in Py_RunMain () from /usr/local/lib/libpython3.9.so.1.0
#93 0x0000ffff89073b84 in Py_BytesMain () from /usr/local/lib/libpython3.9.so.1.0
#94 0x0000ffff88b94d24 in __libc_start_main (main=0xaaaabe7e08c0, argc=3, argv=0xfffffa0a87d8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:308
#95 0x0000aaaabe7e08f8 in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)