-
-
Notifications
You must be signed in to change notification settings - Fork 722
Description
In that past, using Eventlet with Watchdog would cause your program to freeze indefinitely (#679). This was fixed with eventlet/eventlet#975 (though they haven't published a new release yet).
However, there is a new problem. If you stop watchdog (using observer.stop()
), the program will not exit because Watchdog fails to clean up its threads. The relevant code is in inotify_c.py
.
The close
method:
watchdog/src/watchdog/observers/inotify_c.py
Lines 250 to 258 in 9f23b59
if self._path in self._wd_for_path: | |
wd = self._wd_for_path[self._path] | |
inotify_rm_watch(self._inotify_fd, wd) | |
try: | |
os.close(self._inotify_fd) | |
except OSError: | |
# descriptor may be invalid because file was deleted | |
pass |
The thread blocks while reading the file descriptor:
watchdog/src/watchdog/observers/inotify_c.py
Lines 300 to 303 in 9f23b59
while True: | |
try: | |
event_buffer = os.read(self._inotify_fd, event_buffer_size) | |
except OSError as e: |
Without Eventlet, if you call the close
method, the call to inotify_rm_watch
coincidentally causes the Inotify API to write output to the file descriptor. Specifically, it sends the event <InotifyEvent: src_path=b'...', wd=1, mask=IN_IGNORED, ...>
. The thread which is blocking on os.read
sees this output and wakes up. All of this happens before the file descriptor is closed with os.close(self._inotify_fd)
.
However, when you monkey patch with Evenlet, it converts all blocking os.read
calls into non-blocking reads by first polling on the file descriptor and then reading when data is available. Now, when you call the close
method, the call to inotify_rm_watch
will still write data to the file descriptor, but the file descriptor is immediately closed before the reading thread is able to wake up and read the data. Consequently, the reading thread continue blocking and isn't cleaned up.
There are 2 fixes to this race condition:
- If you are certain that calling the
close
method will always write data to the file descriptor, you can defer closing the file descriptor until after the thread blocking onread
has woken up. - Instead of doing a blocking read, the thread can block on a
select
call that monitors both the file descriptor and another channel. Whenclose
is called, it writes to the secondary channel, guaranteeing that the thread is woken up.
I'd be happy to make a PR