Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

GIL Release in copy_from?

#140

Submitted by: Walt Woods

Hey, we have an issue with our software where an entire Python interpreter locks up. Every thread is locked on PyEval_RestoreThread except one, which is in psycopg2:

Thread 24 (Thread 0x7fce8aeef700 (LWP 8655)):
[#0](/projects/62710/tickets/0 "Ticket #0")  0x00007fce9a49eb23 in poll () from /lib/x86_64-linux-gnu/libc.so.6
[#1](/projects/62710/tickets/1 "Ticket #1")  0x00007fce95fcb5ed in ?? () from /usr/lib/libpq.so.5
[#2](/projects/62710/tickets/2 "Ticket #2")  0x00007fce95fcb670 in ?? () from /usr/lib/libpq.so.5
[#3](/projects/62710/tickets/3 "Ticket #3")  0x00007fce95fc9841 in PQgetResult () from /usr/lib/libpq.so.5
[#4](/projects/62710/tickets/4 "Ticket #4")  0x00007fce961f0f20 in _pq_copy_in_v3 (curs=0x7fcddc58ced8)
    at psycopg/pqpath.c:1322
[#5](/projects/62710/tickets/5 "Ticket #5")  pq_fetch (curs=0x7fcddc58ced8) at psycopg/pqpath.c:1462
[#6](/projects/62710/tickets/6 "Ticket #6")  0x00007fce961f12f1 in pq_execute (curs=0x7fcddc58ced8, query=
    0x7fcddc74c880 "COPY myQuery"..., async=0)
    at psycopg/pqpath.c:945
[#7](/projects/62710/tickets/7 "Ticket #7")  0x00007fce961f5dbc in psyco_curs_copy_from (self=0x7fcddc58ced8, 
    args=<optimized out>, kwargs=<optimized out>) at psycopg/cursor_type.c:1336
[#8](/projects/62710/tickets/8 "Ticket #8")  0x000000000049d585 in PyEval_EvalFrameEx ()

So... While the query stalling is happening on our side, it's still bothersome that the GIL isn't released during a COPY_FROM, especially since my understanding is that COPY_FROM is an IO intensive operation (waiting on network) that doesn't need to hold the GIL.

It'd be nice if the GIL wasn't locked for cursor copies.

Reported by Psycopg website · November 23rd, 2012 @ 06:33 PM

State: resolved
Milestone: none
Assigned to: nobody

Activity

  1. Daniele Varrazzo
    Daniele Varrazzo

    Hello Walt,

    it is surprising PQgetResult blocks during copy: the IO intensive part of the copy should be performed by PQputCopyData, which releases the gil. However the libpq doc say PQgetResult is blocking so generally it should be a good idea to release the gil while calling it.

    Can you test with the attached patch if your application starts working correctly?

    Thank you very much.

    November 23rd, 2012 @ 07:04 PM

  2. Daniele Varrazzo
    Daniele Varrazzo

    Looking at the traceback again, the library is probably not blocked in IO but just locked in the poll waiting for the result to arrive. I think we can just wrap each PGgetResult call into a GIL release (provided we have the connection lock, which should be true but I'll double check) as a tentative way to fix this issue. Walt should let us know in the next days if the patch is good for them anyway.

    November 23rd, 2012 @ 09:34 PM

  3. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from new to resolved

    I've committed a patch for PQgetResult after copy. The other uses of the function shouldn't need it (the most important one is in async code, and by that we should have received the message that it won't block anyway)

    I'll mark the issue closed but let me know if it fails to fix the problem.

    November 24th, 2012 @ 10:57 PM