Deadlock in python queue module - Python Help - Discussions on Python.org
Deadlock in python queue module Python Help zdenek (Zdenek Radouch) September 9, 2021, 4:40am 1 I have a program that deadlocks from time to time and it very much looks like it is deadlocking in the standard python queue, which should not be possible given its multi-producer, multi-consumer properties. The code is approximately like this: [the hacky 3-thread solution is because libusb does not support select] thread 1 while True: q.put(socket.recv()) thread 2 while True: q.put(usbdev.read()) thread 3 while True: process(q.get()) It is pretty much impossible to reproduce the problem on demand, but it does usually happen within several days of continuous running.
When hung, it looks like all three threads are deadlocked on what I guess is the queue mutex: pi@raspberrypi:~ $ sudo strace -p 554 strace: Process 554 attached futex(0x75c005b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 554 detached <detached …> pi@raspberrypi:~ $ sudo strace -p 562 strace: Process 562 attached futex(0x75c22510, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 562 detached <detached …> pi@raspberrypi:~ $ sudo strace -p 568 strace: Process 568 attached futex(0x752005d0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 568 detached <detached …> (gdb) i th Id Target Id Frame 1 Thread 0x76ff2ad0 (LWP 554) “python3” 0x76f5e088 in futex_abstimed_wait_cancelable ( private=0, abstime=0x0, expected=1, futex_word=0x75c005b8) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 2 Thread 0x765c8460 (LWP 562) “python3” 0x76f5e088 in futex_abstimed_wait_cancelable ( private=0, abstime=0x0, expected=1, futex_word=0x75c22510) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 3 Thread 0x75bff460 (LWP 568) “python3” 0x76f5e088 in futex_abstimed_wait_cancelable ( private=0, abstime=0x0, expected=1, futex_word=0x752005d0) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 Note that this is a Pi distribution: pi@raspberrypi:~ $ python3 --version Python 3.7.3 I am looking for ideas.
Thanks cameron (Cameron Simpson) September 9, 2021, 7:17am 2 I have a program that deadlocks from time to time and it very much looks like it is deadlocking in the standard python queue, which should not be possible given its multi-producer, multi-consumer properties. The code is approximately like this: [the hacky 3-thread solution is because libusb does not support select] thread 1 while True: q.put(socket.recv()) thread 2 while True: q.put(usbdev.read()) thread 3 while True: process(q.get()) Indeed that should never deadlock.
It is pretty much impossible to reproduce the problem on demand, but it does usually happen within several days of continuous running. When hung, it looks like all three threads are deadlocked on what I guess is the queue mutex: pi@raspberrypi:~ sudo strace -p 554 strace: Process 554 attached futex(0x75c005b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 554 detached <detached ...> pi@raspberrypi:~ sudo strace -p 562 strace: Process 562 attached futex(0x75c22510, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 562 detached <detached …> pi@raspberrypi:~ $ sudo strace -p 568 strace: Process 568 attached futex(0x752005d0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, FUTEX_BITSET_MATCH_ANY ^Cstrace: Process 568 detached <detached …> That looks naively like 3 distinct locks. (Of course a u eue will have more than one internal lock.) Are you sure it is locked on the queue?
If you separate out the q.put and its arguments? while True: print("recv") data = socket.recv() print("got", len(data), "bytes") q.put(data) print("put completed") and similarly for the other loops. Is it definitely blocked in the queue operation? Have you any opportunity to try this against the latest release Python? Cheers, Cameron Simpson cs@cskk.id.au zdenek (Zdenek Radouch) September 9, 2021, 12:53pm 3 Well, no I don’t know how to exactly interpret the hung condition, other than guessing.
All three threads are deadlocked the same way in the same piece of code: (gdb) thr 1 [Switching to thread 1 (Thread 0x76ff2ad0 (LWP 554))] #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x75c005b8) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 205 …/sysdeps/unix/sysv/linux/futex-internal.h: No such file or directory. (gdb) bt #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x75c005b8) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x75c005b8, abstime=0x0) at sem_waitcommon.c:115 #2 0x76f5e1f4 in __new_sem_wait_slow (sem=0x75c005b8, abstime=0x0) at sem_waitcommon.c:282 #3 0x000f9028 in PyThread_acquire_lock_timed () #4 0x000cbc44 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thr 2 [Switching to thread 2 (Thread 0x765c8460 (LWP 562))] #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x75c22510) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 205 in …/sysdeps/unix/sysv/linux/futex-internal.h (gdb) bt #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x75c22510) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x75c22510, abstime=0x0) at sem_waitcommon.c:115 #2 0x76f5e1f4 in __new_sem_wait_slow (sem=0x75c22510, abstime=0x0) at sem_waitcommon.c:282 #3 0x000f9028 in PyThread_acquire_lock_timed () #4 0x000cbc44 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb) thr 3 [Switching to thread 3 (Thread 0x75bff460 (LWP 568))] #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x752005d0) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 205 in …/sysdeps/unix/sysv/linux/futex-internal.h (gdb) bt #0 0x76f5e088 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=1, futex_word=0x752005d0) at …/sysdeps/unix/sysv/linux/futex-internal.h:205 #1 do_futex_wait (sem=sem@entry=0x752005d0, abstime=0x0) at sem_waitcommon.c:115 #2 0x76f5e1f4 in __new_sem_wait_slow (sem=0x752005d0, abstime=0x0) at sem_waitcommon.c:282 #3 0x000f9028 in PyThread_acquire_lock_timed () #4 0x000cbc44 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) I am not familiar with the implementation so all I am considering is the user-visible stuff – the two producer threads should be waiting in the respective syscalls (which they are until the thing deadlocks).
I tried instrumenting the code, too, but all I get is the expected output until the deadlock, then all is hung. Milton_Mobley (Milton Mobley) September 24, 2021, 12:09am 4 I hope this may help: In your original post you give only a simplified version of the code’s action. The threads might appear to be blocked if socket.recv() and usbdev.read() are not actually producing messages. What data do you expect socket.recv() to produce?
Same for usbdev.read(). I mean not contents, but size and frequency of messages. You could write simple programs to measure these, and verify the data you expect is being produced. I looked at the Python source for the queue module. It looks pretty straightforward for queue code, and is probably not the source of the problem. Otherwise, you could set some gdb breakpoints in the C backend code. I see that PyThread_acquire_lock_timed() is implemented in the source in the file Python/thread_pthread.h.
Perhaps you could find some place around there to set breakpoints, in an error path that you do not expect to be taken, not the main (success) path. The semaphore get operations are supposed to time out. Alternately, you could try to use the queues in the multiprocessing module
The rest of this page is on the live site.