Skip to content

Threads not cleaning up when closing Observer if using Eventlet. #1045

@ethan-vanderheijden

Description

@ethan-vanderheijden

In that past, using Eventlet with Watchdog would cause your program to freeze indefinitely (#679). This was fixed with eventlet/eventlet#975 (though they haven't published a new release yet).

However, there is a new problem. If you stop watchdog (using observer.stop()), the program will not exit because Watchdog fails to clean up its threads. The relevant code is in inotify_c.py.

The close method:

if self._path in self._wd_for_path:
wd = self._wd_for_path[self._path]
inotify_rm_watch(self._inotify_fd, wd)
try:
os.close(self._inotify_fd)
except OSError:
# descriptor may be invalid because file was deleted
pass

The thread blocks while reading the file descriptor:

while True:
try:
event_buffer = os.read(self._inotify_fd, event_buffer_size)
except OSError as e:

Without Eventlet, if you call the close method, the call to inotify_rm_watch coincidentally causes the Inotify API to write output to the file descriptor. Specifically, it sends the event <InotifyEvent: src_path=b'...', wd=1, mask=IN_IGNORED, ...>. The thread which is blocking on os.read sees this output and wakes up. All of this happens before the file descriptor is closed with os.close(self._inotify_fd).

However, when you monkey patch with Evenlet, it converts all blocking os.read calls into non-blocking reads by first polling on the file descriptor and then reading when data is available. Now, when you call the close method, the call to inotify_rm_watch will still write data to the file descriptor, but the file descriptor is immediately closed before the reading thread is able to wake up and read the data. Consequently, the reading thread continue blocking and isn't cleaned up.

There are 2 fixes to this race condition:

  1. If you are certain that calling the close method will always write data to the file descriptor, you can defer closing the file descriptor until after the thread blocking on read has woken up.
  2. Instead of doing a blocking read, the thread can block on a select call that monitors both the file descriptor and another channel. When close is called, it writes to the secondary channel, guaranteeing that the thread is woken up.

I'd be happy to make a PR

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions