Support pip binary wheel manylinux installations #3060
Description
rasterio, shapely, pyproj etc. now provide pure-pip installations with binary wheels that provide manylinux [1] binaries for gdal, proj, geos etc.; can this project also provide binary wheel installations? e.g. see
Potentially Shared Libs
Any way it could work would be great, but it might be optimal if some common binary libs for gdal/ogr could be shared among various python libraries that require them. (The same could apply to proj/pyproj.). That is, not an OS installed shared lib, but a python manylinux [1] binary shared lib (in e.g. {project-venv-path}/lib/, alongside of {project-venv-path}/lib/python3.6).
[1] https://github.com/pypa/manylinux
What appears to be happening is that several related but independent pypi projects will each install their own copies of various possibly-shared libs, e.g. both rasterio and fiona both install their own copies of gdal_data into e.g.
.../lib/python3.6/site-packages/fiona/gdal_data/*.../lib/python3.6/site-packages/rasterio/gdal_data/*
The same applies to proj_data, i.e.
.../lib/python3.6/site-packages/fiona/proj_data/*.../lib/python3.6/site-packages/rasterio/proj_data/*
For rasterio alone, the binary packages are substantial, e.g.
0 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/
376744 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libopenjp2-8f6da918.so.2.3.0
2240704 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgeos--no-undefined-b94097bf.so
16792 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libsz-978a1c7f.so.2.0.1
1165464 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libsqlite3-a9c9c58e.so.0.8.6
203512 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libnghttp2-11cb20b8.so.14.17.1
44424 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libjson-c-0c137dce.so.2.0.2
87848 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libz-a147dcb0.so.1.2.3
445144 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libwebp-dc7313d0.so.7.0.5
23783432 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgdal-b29a5f73.so.20.5.4
1576712 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libnetcdf-02a36646.so.13.1.1
182056 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libhdf5_hl-308f82c1.so.100.1.2
3855616 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libhdf5-9d9b49cc.so.103.1.0
3620912 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libcurl-ea538880.so.4.4.0
175000 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libexpat-c4a93fc7.so.1.6.8
218400 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libpng16-de469eac.so.16.35.0
354216 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgeos_c-a68605fd.so.1.13.1
33624 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libaec-21547b1b.so.0.0.10
250488 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libjpeg-3b10b538.so.9.3.0
453488 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libproj-cd06b982.so.12.0.0
When combined with a few more related projects, there is some potential duplication of the libs and closer inspection of the lib-versions suggests there could be some inconsistency in the versions installed (without any explicit pip options to control those binary lib versions packaged), e.g.
0 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/
87856 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libz-fiona-a147dcb0.so.1.2.3
60800 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libjson-c-fiona-5f02f62c.so.2.0.2
21884960 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libgdal-fiona-9fe15c06.so.20.5.4
354224 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libgeos_c-fiona-a68605fd.so.1.13.1
3620920 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libcurl-fiona-ea538880.so.4.4.0
1261392 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libsqlite3-fiona-25a4bc97.so.0.8.6
175008 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libexpat-fiona-c4a93fc7.so.1.6.8
203520 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libnghttp2-fiona-11cb20b8.so.14.17.1
2240712 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libgeos--no-undefined-fiona-b94097bf.so
344704 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libjpeg-fiona-3fe7dfc0.so.9.3.0
279840 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libpng16-fiona-898afbbd.so.16.35.0
453488 2020-10-13 10:23 python/lib/python3.6/site-packages/Fiona.libs/libproj-fiona-cd06b982.so.12.0.0
0 2020-10-13 10:23 python/lib/python3.6/site-packages/pyproj/.libs/
92080 2020-10-13 10:23 python/lib/python3.6/site-packages/pyproj/.libs/libz-eb09ad1d.so.1.2.3
1271584 2020-10-13 10:23 python/lib/python3.6/site-packages/pyproj/.libs/libsqlite3-b65a32f0.so.0.8.6
8155504 2020-10-13 10:23 python/lib/python3.6/site-packages/pyproj/.libs/libproj-d352b7c6.so.15.2.1
0 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/
376744 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libopenjp2-8f6da918.so.2.3.0
2240704 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgeos--no-undefined-b94097bf.so
16792 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libsz-978a1c7f.so.2.0.1
1165464 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libsqlite3-a9c9c58e.so.0.8.6
203512 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libnghttp2-11cb20b8.so.14.17.1
44424 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libjson-c-0c137dce.so.2.0.2
87848 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libz-a147dcb0.so.1.2.3
445144 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libwebp-dc7313d0.so.7.0.5
23783432 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgdal-b29a5f73.so.20.5.4
1576712 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libnetcdf-02a36646.so.13.1.1
182056 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libhdf5_hl-308f82c1.so.100.1.2
3855616 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libhdf5-9d9b49cc.so.103.1.0
3620912 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libcurl-ea538880.so.4.4.0
175000 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libexpat-c4a93fc7.so.1.6.8
218400 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libpng16-de469eac.so.16.35.0
354216 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libgeos_c-a68605fd.so.1.13.1
33624 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libaec-21547b1b.so.0.0.10
250488 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libjpeg-3b10b538.so.9.3.0
453488 2020-10-13 10:23 python/lib/python3.6/site-packages/rasterio.libs/libproj-cd06b982.so.12.0.0
0 2020-10-13 10:23 python/lib/python3.6/site-packages/numpy.libs/
92080 2020-10-13 10:23 python/lib/python3.6/site-packages/numpy.libs/libz-eb09ad1d.so.1.2.3
30077440 2020-10-13 10:23 python/lib/python3.6/site-packages/numpy.libs/libopenblasp-r0-ae94cfde.3.9.dev.so
2260064 2020-10-13 10:23 python/lib/python3.6/site-packages/numpy.libs/libgfortran-2e0d59d6.so.5.0.0
263992 2020-10-13 10:23 python/lib/python3.6/site-packages/numpy.libs/libquadmath-2d0c479f.so.0.0.0
0 2020-10-13 10:23 python/lib/python3.6/site-packages/shapely/.libs/
2240704 2020-10-13 10:23 python/lib/python3.6/site-packages/shapely/.libs/libgeos--no-undefined-b94097bf.so
354216 2020-10-13 10:23 python/lib/python3.6/site-packages/shapely/.libs/libgeos_c-a68605fd.so.1.13.1
The request here on this project is that it is very close to the C-source used by some common python wrappers and it might provide a "source of truth" about how to package some common libs using pip/manylinux wheels. (The same request could apply to the C-source for proj perhaps.). Obviously the manylinux builds would need to provide version specific builds. (I don't have a clear idea on how they would be provided as shared-libs, only that the duplications and inconsistencies observed above might benefit from some kind of shared-libs solutions -- that are not OS package solutions, despite how much work goes into those.)