Skip to content

Add *python-call-mode* in config: allow using a separate thread to call python #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 33 commits into from
Jun 28, 2025
Merged

Add *python-call-mode* in config: allow using a separate thread to call python #23

merged 33 commits into from
Jun 28, 2025

Conversation

digikar99
Copy link
Owner

Fixes #11

This PR adds the variables py4cl2-cffi/config:*python-call-mode* and py4cl2-cffi:+python-call-mode+, along with appropriate changes to ecase on these variables.

The central idea is that when python call mode is :single-threaded, py4cl2-cffi starts the python interpreter in its own separate thread. All calls to python via pyvalue pycall and friends occur through this thread. This allows libraries like matplotlib to assume that there is a single thread.

To use this functionality -

(asdf:load-system "py4cl2-cffi/config")
(setf py4cl2-cffi/config:*python-call-mode* :single-threaded) ; or :multi-threaded - this is the default
(asdf:load-system "py4cl2-cffi" :force t)
(py4cl2-cffi:pystart)

Note that the decision to use a separate thread or otherwise needs to be made before python initialization aka pystart.

This also makes the single-threaded/ directory unnecessary.

Two main concerns with this PR include:

  1. Naming: :dedicated and :standard instead of :single-threaded and :multi-threaded sound more informative. I'd be happy to hear any opinions from repository watchers or anyone passing by.
  2. :single-threaded mode is about 10 times slower than :multi-threaded. I'm also yet to benchmark :multi-threaded against the existing code. Ideally, this should not incur a penalty, since we are trying to push the decisions to compile time instead of runtime.

@digikar99 digikar99 mentioned this pull request May 1, 2025
Open
@digikar99
Copy link
Owner Author

Running the scripts in ./perf-compare/ indicates a minimal loss of performance with the :multi-threaded mode above. Though, the :single-threaded mode is unimaginably slower. Comparing against (funcall/single-threaded #'identity i) in a loop reveals :single-threaded can be upto 10 times faster without modifying the current thread synchronization primitives.

@digikar99
Copy link
Owner Author

Proof of concept: Wrapping the body of pycall inside with-dedicated-python-thread committed in 0c69d4d boosts the performance in :single-threaded mode by about 5 times(!)

@digikar99 digikar99 force-pushed the python-call-mode branch 3 times, most recently from dd3a17f to e18dd82 Compare May 23, 2025 09:20
@atzmueller
Copy link
Contributor

I think :dedicated and :standard are OK.

digikar99 added 23 commits June 7, 2025 15:28
@digikar99
[add] *python-call-mode* and +python-call-mode+
848f893
@digikar99
[int] add dedicated-thread-utils
144e1b3
@digikar99
[fix] differentiate pystart on +python-call-mode+
69ff94a
@digikar99
[add] verbose option to pystart and children
ac7eae9
@digikar99
[fix] Don't release gil when starting in dedicated thread mode
bd441ec
@digikar99
[int] push and pop to global vars instead of structure
40284be
@digikar99
[fix] mask float traps inside the pymain-thread
cf084b6
@digikar99
[fix] make pyforeign-funcall work in both modes
6809cee
@digikar99
[fix] rewrite pyimport-importmodule to case on +python-call-mode+
9a6b7ff
@digikar99
[add] *pystart-verbosity*
73c7086
@digikar99
[fix] improve error handling in dedicated-thread mode
e42285f
@digikar99
[fix] handle multiple return values in dedicated-thread mode
54528db
@digikar99
[fix] get pystop to work preliminarily in dedicated-thread mode
ce7b5c2
@digikar99
[fix] pyforeign-funcall: better handle dedicated-thread mode
df14d6c
@digikar99
[fix] with-python-gil: case on +python-call-mode+
f6735b9
@digikar99
[fix] define-capi-function: case on +python-call-mode+
9fa07b5
@digikar99
[fix] inform +python-call-mode+ during pystart
622db35
@digikar99
[fix] ecase finalization-lambda on +python-call-mode+
e951d8f
@digikar99
[fix] add and use *pymain-call-lock*: fixes mysterious CCL errors
55fdbad
@digikar99
[int] slightly cleanup pyforeign-funcall
1a036f6
@digikar99
[fix] add the missing py-decref
6d777b0
@digikar99
[ci] add ci-test-dedicated.lisp
87ec693
@digikar99
[fix] pystop: typo and commit remnants
cd2e6ea
@digikar99
Copy link
Owner Author

With the way calls are made to the dedicated thread via dedicated-thread-utils.lisp, there's not much performance to be gained beyond what 5ce8660 enables.

PY4CL2-CFFI> (time
              (loop for i below 10000 do (funcall/dedicated-thread #'pycall "str" i)))
  0.253 seconds of real time
NIL
PY4CL2-CFFI> (time
              (loop for i below 10000 do (funcall/dedicated-thread #'null i)))
  0.232 seconds of real time
NIL
PY4CL2-CFFI> (time
              (loop for i below 10000 do (funcall/dedicated-thread #'null-pointer)))
  0.189 seconds of real time
NIL

This is 10+ time slower than when *python-call-mode* is :standard.

I think this PR is ready to be merged. Perhaps in the future, if someone discovers a better method of sync-ing or interthread calls, this might be optimizable.

@digikar99 digikar99 merged commit 1e85e17 into master Jun 28, 2025
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

plt.plot hangs
2 participants
@digikar99 @atzmueller