Cluster duplicate handler dying from RuntimeError due to dict change while iterating it #160
Description
The following traceback has been seen during shutdown at least once with v2.0:
peekaboo[984]: peekaboo.queuing - (Worker-9) - INFO - Worker 9: Stopped
peekaboo[984]: peekaboo.queuing - (MainThread) - DEBUG - 1: 32 workers still running
peekaboo[984]: peekaboo.db - (MainThread) - DEBUG - Clearing database of all in-flight samples of instance 5010.
peekaboo[984]: peekaboo.db - (MainThread) - DEBUG - Clearing database of all stale in-flight samples (900 seconds)
peekaboo[984]: Exception in thread ClusterDuplicateHandler:
peekaboo[984]: Traceback (most recent call last):
peekaboo[984]: File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
peekaboo[984]: self.run()
peekaboo[984]: File "/opt/peekaboo/local/lib/python3.6/site-packages/peekaboo/queuing.py", line 327, in run
peekaboo[984]: self.job_queue.submit_cluster_duplicates()
peekaboo[984]: File "/opt/peekaboo/local/lib/python3.6/site-packages/peekaboo/queuing.py", line 168, in submit_cluster_duplicates
peekaboo[984]: for sample_hash, sample_duplicates in self.cluster_duplicates.items():
peekaboo[984]: RuntimeError: dictionary changed size during iteration
peekaboo[984]: peekaboo.daemon - (MainThread) - DEBUG - Removing PID file /var/run/peekaboo/peekaboo.pid
systemd[1]: Stopped Peekaboo Extended Email Attachment Behavior Observation Owl.
There seems to be some kind of a race in Queue.shut_down() between cluster duplicate handler and queue shutdown which is odd because duplicate handler shutdown is the very first thing triggered, so it should not do another cleanup run while the queue is shutting down workers.