pypy: make CFFI builds find libsqlite3 #431707

MaxWipfli · 2025-08-07T10:00:42Z

This fixes the PyPy build, which has been broken since June 2025. The issue was tracked down in #419942:
Basically, SQLite stopped setting SONAME for libsqlite3.so since their switch to autosetup (in v3.50).
This breaks ctypes.util.find_library() for CPython 2.7 (and PyPy 2.7) only.
However, both PyPy 2.7 and PyPy 3.x are built using PyPy 2.7, and since this is a build-time issue, it also affects PyPy 3.

This is one approach to fix this. Others would include:

back-porting the fix from Python3 to pypy27_prebuilt and pypy27, similar to what is done for CPython 2 (https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/interpreters/python/cpython/2.7/find_library-gcc10.patch),
~~abandoning PyPy 2.7 fully and buildling PyPy 3.X using CPython 2.7 (but this apparently can take 2-3 times longer)~~ (not an option as CPython 2.7 is unsupported, insecure, and due for removal in Nixpkgs soon), or
setting the SONAME again for libsqlite3, using --soname=legacy configure flag for SQLite (as suggested in Build failure: pypy #419942).
We could also disable the sqlite3 module, but that seems like the worst idea.

I noticed that there are also some other patches to the CPython 2.7 standard library which should probably also be applied to PyPy 2.7 (since PyPy's standard library is mostly copied verbatim from CPython). In the end, it boils down to whether PyPy 2.7 should be supported

not at all (building PyPy 3.x using CPython);
well enough to just build PyPy 3; or
fully (in which case there are a number of patches that should probably be applied);

I have tested this locally using a follow-up to #430846, which uses the (now working) PyPy 3.11 in the icestorm package.

Resolves #419942.

Add a 👍 reaction to pull requests you find important.

emilazy · 2025-08-15T14:23:00Z

abandoning PyPy 2.7 fully and buildling PyPy 3.X using CPython 2.7 (but this apparently can take 2-3 times longer), or

This isn't an option - we only have our CPython 2.7 package around for resholve at this point and it is due for removal as soon as that is sorted one way or another.

PyPy considers their 2.7 implementation a going concern, right? I assume they means they're taking maintenance responsibility for the standard library too? Has this been reported upstream for them to backport the fix?

MaxWipfli · 2025-08-15T14:40:32Z

PyPy considers their 2.7 implementation a going concern, right? I assume they means they're taking maintenance responsibility for the standard library too? Has this been reported upstream for them to backport the fix?

Yes, they say: "PyPy will always support 2.7 since RPython is written for it." No, I have not reported this upstream. I will look into that.

Probably, I should also look into the other CPython 2.7 patches we have and check which of those apply to the standard library (and are thus potentially relevant for PyPy).

SuperSandroBot · 2025-08-16T20:47:23Z

`nixpkgs-review` result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 431707

Logs: https://github.com/SuperSandro2000/nixpkgs-review-gha/actions/runs/17010908506

`x86_64-linux`

❌ 2 packages failed to build:

fbpanel
tests.writers.simple.pypy3NoLibs

✅ 5 packages built:

cvs2svn
cvs2svn.dist
pypy (pypy2, pypy27)
pypy3 (pypy311)
pypy310

Error logs: `x86_64-linux`

fbpanel

plugin.h:32:38: warning: 'struct _plugin_instance' declared inside parameter list will not be visible outside of this definition or declaration
   32 |     GtkWidget *(*edit_config)(struct _plugin_instance *this);
      |                                      ^~~~~~~~~~~~~~~~
plugin.c: In function 'plugin_start':
plugin.c:188:35: error: passing argument 1 of 'this->class->constructor' from incompatible pointer type [8;;https://gcc.gnu.org/onlinedocs/gcc-14.3.0/gcc/Warning-Options.html#index-Wincompatible-pointer-types�-Wincompatible-pointer-types8;;�]
  188 |     if (!this->class->constructor(this)) {
      |                                   ^~~~
      |                                   |
      |                                   plugin_instance * {aka struct _plugin_instance *}
plugin.c:188:35: note: expected 'struct _plugin_instance *' but argument is of type 'plugin_instance *' {aka 'struct _plugin_instance *'}
plugin.c: In function 'plugin_stop':
plugin.c:202:29: error: passing argument 1 of 'this->class->destructor' from incompatible pointer type [8;;https://gcc.gnu.org/onlinedocs/gcc-14.3.0/gcc/Warning-Options.html#index-Wincompatible-pointer-types�-Wincompatible-pointer-types8;;�]
  202 |     this->class->destructor(this);
      |                             ^~~~
      |                             |
      |                             plugin_instance * {aka struct _plugin_instance *}
plugin.c:202:29: note: expected 'struct _plugin_instance *' but argument is of type 'plugin_instance *' {aka 'struct _plugin_instance *'}
make[1]: *** [../.config/rules.mk:276: plugin.o] Error 1
make[1]: Leaving directory '/build/source/panel'
make: *** [.config/rules.mk:46: panel] Error 2

`aarch64-linux`

❌ 2 packages failed to build:

fbpanel
tests.writers.simple.pypy3NoLibs

✅ 5 packages built:

cvs2svn
cvs2svn.dist
pypy (pypy2, pypy27)
pypy3 (pypy311)
pypy310

Error logs: `aarch64-linux`

fbpanel

plugin.h:32:38: warning: 'struct _plugin_instance' declared inside parameter list will not be visible outside of this definition or declaration
   32 |     GtkWidget *(*edit_config)(struct _plugin_instance *this);
      |                                      ^~~~~~~~~~~~~~~~
plugin.c: In function 'plugin_start':
plugin.c:188:35: error: passing argument 1 of 'this->class->constructor' from incompatible pointer type [8;;https://gcc.gnu.org/onlinedocs/gcc-14.3.0/gcc/Warning-Options.html#index-Wincompatible-pointer-types�-Wincompatible-pointer-types8;;�]
  188 |     if (!this->class->constructor(this)) {
      |                                   ^~~~
      |                                   |
      |                                   plugin_instance * {aka struct _plugin_instance *}
plugin.c:188:35: note: expected 'struct _plugin_instance *' but argument is of type 'plugin_instance *' {aka 'struct _plugin_instance *'}
plugin.c: In function 'plugin_stop':
plugin.c:202:29: error: passing argument 1 of 'this->class->destructor' from incompatible pointer type [8;;https://gcc.gnu.org/onlinedocs/gcc-14.3.0/gcc/Warning-Options.html#index-Wincompatible-pointer-types�-Wincompatible-pointer-types8;;�]
  202 |     this->class->destructor(this);
      |                             ^~~~
      |                             |
      |                             plugin_instance * {aka struct _plugin_instance *}
plugin.c:202:29: note: expected 'struct _plugin_instance *' but argument is of type 'plugin_instance *' {aka 'struct _plugin_instance *'}
make[1]: *** [../.config/rules.mk:276: plugin.o] Error 1
make[1]: Leaving directory '/build/source/panel'
make: *** [.config/rules.mk:46: panel] Error 2

`x86_64-darwin` (sandbox = true)

❌ 1 package failed to build:

tests.writers.simple.pypy3NoLibs

✅ 5 packages built:

cvs2svn
cvs2svn.dist
pypy (pypy2, pypy27)
pypy3 (pypy311)
pypy310

`aarch64-darwin` (sandbox = true)

❌ 1 package failed to build:

tests.writers.simple.pypy3NoLibs

✅ 5 packages built:

cvs2svn
cvs2svn.dist
pypy (pypy2, pypy27)
pypy3 (pypy311)
pypy310

test_misencodedFileUTF16 fails with PyPy3 as it outputs a different error message than CPython.

MaxWipfli · 2025-08-17T13:54:07Z

I have addressed the nixpkgs-review build failures (at least on x86_64-linux).

fbpanel: just required an additional -Wno-error=incompatible-pointer-types (and correctly setting NIX_CFLAGS_COMPILE to a string).
tests.writers.simple.pypy3NoLibs: just had to disable a failing test for pypy3Packages.pyflakes (see also pypy3Packages.pyflakes: disable another broken test #412923, but I included this trivial change here)

7c6f434c · 2025-08-17T15:20:51Z

Strictly speaking the expectation of DT_SONAME in general is also an ongoing concern for CPython3 ctypes.util in general, although they seem to avoid the question as much as possible python/cpython#65821 python/cpython#18380

SuperSandro2000 · 2025-08-17T23:13:17Z

So we can merge this as is now?

MaxWipfli · 2025-08-18T05:08:12Z

So we can merge this as is now?

Yes. I have a follow-up to make more progress on pypy3.passthru.tests, but that will need to go into staging.

pypy{27,310,311}: make CFFI builds find libsqlite3

e33181b

nix-owners bot requested review from natsukium, mweinelt and andersk August 7, 2025 10:02

nixpkgs-ci bot added 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 12.first-time contribution This PR is the author's first one; please be gentle! labels Aug 7, 2025

nix-owners bot requested a review from FliegendeWurst August 7, 2025 10:07

nixpkgs-ci bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Aug 7, 2025

mweinelt requested a review from SuperSandro2000 August 7, 2025 12:16

nixpkgs-ci bot removed the 12.first-time contribution This PR is the author's first one; please be gentle! label Aug 15, 2025

SuperSandro2000 approved these changes Aug 16, 2025

View reviewed changes

nixpkgs-ci bot added the 12.approvals: 1 This PR was reviewed and approved by one person. label Aug 16, 2025

MaxWipfli added 2 commits August 17, 2025 15:01

fbpanel: correctly apply -Wno-error

f6b62e0

python3Packages.pyflakes: disable failing test for PyPy3

20d8a65

test_misencodedFileUTF16 fails with PyPy3 as it outputs a different error message than CPython.

nix-owners bot requested a review from 7c6f434c August 17, 2025 13:12

SuperSandro2000 approved these changes Aug 17, 2025

View reviewed changes

SuperSandro2000 mentioned this pull request Aug 17, 2025

pypy3Packages.pyflakes: disable another broken test #412923

Closed

13 tasks

7c6f434c merged commit dc2ad08 into NixOS:master Aug 18, 2025
27 of 29 checks passed

MaxWipfli deleted the pypy_sqlite3_fix branch August 19, 2025 14:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

pypy: make CFFI builds find libsqlite3 #431707

pypy: make CFFI builds find libsqlite3 #431707

Uh oh!

MaxWipfli commented Aug 7, 2025 •

edited

Loading

Uh oh!

emilazy commented Aug 15, 2025

Uh oh!

MaxWipfli commented Aug 15, 2025 •

edited

Loading

Uh oh!

SuperSandroBot commented Aug 16, 2025

Uh oh!

MaxWipfli commented Aug 17, 2025

Uh oh!

7c6f434c commented Aug 17, 2025

Uh oh!

SuperSandro2000 commented Aug 17, 2025 •

edited

Loading

Uh oh!

MaxWipfli commented Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pypy: make CFFI builds find libsqlite3 #431707

pypy: make CFFI builds find libsqlite3 #431707

Uh oh!

Conversation

MaxWipfli commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilazy commented Aug 15, 2025

Uh oh!

MaxWipfli commented Aug 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SuperSandroBot commented Aug 16, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin (sandbox = true)

aarch64-darwin (sandbox = true)

Uh oh!

MaxWipfli commented Aug 17, 2025

Uh oh!

7c6f434c commented Aug 17, 2025

Uh oh!

SuperSandro2000 commented Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxWipfli commented Aug 18, 2025

Uh oh!

Uh oh!

Uh oh!

MaxWipfli commented Aug 7, 2025 •

edited

Loading

MaxWipfli commented Aug 15, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin` (sandbox = true)

`aarch64-darwin` (sandbox = true)

SuperSandro2000 commented Aug 17, 2025 •

edited

Loading