Skip to content

Conversation

licy183
Copy link
Member

@licy183 licy183 commented Feb 9, 2022

Bionic libc doesn't support sem_open, sem_close and sem_unlink. This fix contains a port of posix named semaphore and shared memory from musl libc and is used for python multiprocessing module. Note that Android has no /dev/shm, so the mapped file will be created at the temp directory of termux.

Related issues: #570, #8938

@xtkoba
Copy link
Contributor

xtkoba commented Feb 10, 2022

Nice work!

It would be more convenient if sem_* implementations were separated in a shared library so that they can be used from other packages.

@licy183
Copy link
Member Author

licy183 commented Feb 10, 2022

It would be more convenient if sem_* implementations were separated in a shared library so that they can be used from other packages.

OK. I will try to separate the sem_* functions into a package.

Should I fold it into the libandroid-support package or add a new package called libandroid-posix-semaphore then remove <semaphore.h> from NDK header? The <semaphore.h> seems to be vital, the latter choice may cause compilation failure of other packages.

I will not seperate the shm_* functions. It may cause security issues because lots of packages use /dev/shm to save password, et al.

@licy183 licy183 marked this pull request as draft February 10, 2022 07:02
@xtkoba
Copy link
Contributor

xtkoba commented Feb 10, 2022

OK. I will try to separate the sem_* functions into a package.

Great!

Should I fold it into the libandroid-support package or add a new package called libandroid-posix-semaphore then remove <semaphore.h> from NDK header? The <semaphore.h> seems to be vital, the latter choice may cause compilation failure of other packages.

A new package will be preferable.

I suppose we can safely remove <semaphore.h> from NDK. If a package requires semaphore.h then it should be sufficient to add the semaphore package as a (build-time only) dependency.

I will not seperate the shm_* functions. It may cause security issues because lots of packages use /dev/shm to save password, et al.

I see.

@licy183 licy183 marked this pull request as ready for review February 11, 2022 11:01
@licy183
Copy link
Member Author

licy183 commented Feb 11, 2022

#8993 has been merged. This PR is now ready for a review.

Copy link
Contributor

@truboxl truboxl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

+ errno = ENAMETOOLONG;
+ return 0;
+ }
+ memcpy(buf, "@TERMUX_PREFIX@/tmp/shm.", 40);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there any reason why it end with a . ? Musl ends with /
Same goes in libandroid-posix-semaphore

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The filename is appended to the path after, so we either have $PREFIX/tmp/shm.something or $PREFIX/tmp/shm/something. With $PREFIX/tmp/shm/ we would have to check if folder exists (and create it if it doesn't), so I suppose this is just a way to save a couple of lines of code

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not hardcode the string length. The 40 here should be replaced with strlen("@TERMUX_PREFIX@/tmp/shm.") or something. (See also #9007.)

+ return 0;
+ }
+ memcpy(buf, "@TERMUX_PREFIX@/tmp/shm.", 40);
+ memcpy(buf+40, name, p-name+1);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same as #8990 (comment).

+
+int shm_open(const char *name, int flag, mode_t mode)
+{
+ char buf[NAME_MAX+41];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost the same as #8990 (comment). The 41 here should be something like strlen("@TERMUX_PREFIX@/tmp/shm.")+1.

+
+int shm_unlink(const char *name)
+{
+ char buf[NAME_MAX+41];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same here.

Copy link
Member

@Grimler91 Grimler91 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with one python package that normally fails (papis). Everything now seem to work!

@xtkoba xtkoba merged commit 258f5fa into termux:master Feb 13, 2022
@licy183 licy183 deleted the fix-python-multiprocessing branch February 13, 2022 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants