This project is archived and is in readonly mode.

#140 ✓resolved
Psycopg website

GIL Release in copy_from?

Reported by Psycopg website | November 23rd, 2012 @ 06:33 PM

Submitted by: Walt Woods

Hey, we have an issue with our software where an entire Python interpreter locks up. Every thread is locked on PyEval_RestoreThread except one, which is in psycopg2:

Thread 24 (Thread 0x7fce8aeef700 (LWP 8655)):
[#0](/projects/62710/tickets/0 "Ticket #0")  0x00007fce9a49eb23 in poll () from /lib/x86_64-linux-gnu/libc.so.6
[#1](/projects/62710/tickets/1 "Ticket #1")  0x00007fce95fcb5ed in ?? () from /usr/lib/libpq.so.5
[#2](/projects/62710/tickets/2 "Ticket #2")  0x00007fce95fcb670 in ?? () from /usr/lib/libpq.so.5
[#3](/projects/62710/tickets/3 "Ticket #3")  0x00007fce95fc9841 in PQgetResult () from /usr/lib/libpq.so.5
[#4](/projects/62710/tickets/4 "Ticket #4")  0x00007fce961f0f20 in _pq_copy_in_v3 (curs=0x7fcddc58ced8)
    at psycopg/pqpath.c:1322
[#5](/projects/62710/tickets/5 "Ticket #5")  pq_fetch (curs=0x7fcddc58ced8) at psycopg/pqpath.c:1462
[#6](/projects/62710/tickets/6 "Ticket #6")  0x00007fce961f12f1 in pq_execute (curs=0x7fcddc58ced8, query=
    0x7fcddc74c880 "COPY myQuery"..., async=0)
    at psycopg/pqpath.c:945
[#7](/projects/62710/tickets/7 "Ticket #7")  0x00007fce961f5dbc in psyco_curs_copy_from (self=0x7fcddc58ced8, 
    args=<optimized out>, kwargs=<optimized out>) at psycopg/cursor_type.c:1336
[#8](/projects/62710/tickets/8 "Ticket #8")  0x000000000049d585 in PyEval_EvalFrameEx ()

So... While the query stalling is happening on our side, it's still bothersome that the GIL isn't released during a COPY_FROM, especially since my understanding is that COPY_FROM is an IO intensive operation (waiting on network) that doesn't need to hold the GIL.

It'd be nice if the GIL wasn't locked for cursor copies.

Comments and changes to this ticket

  • Daniele Varrazzo

    Daniele Varrazzo November 23rd, 2012 @ 07:04 PM

    Hello Walt,

    it is surprising PQgetResult blocks during copy: the IO intensive part of the copy should be performed by PQputCopyData, which releases the gil. However the libpq doc say PQgetResult is blocking so generally it should be a good idea to release the gil while calling it.

    Can you test with the attached patch if your application starts working correctly?

    Thank you very much.

  • Daniele Varrazzo

    Daniele Varrazzo November 23rd, 2012 @ 09:34 PM

    Looking at the traceback again, the library is probably not blocked in IO but just locked in the poll waiting for the result to arrive. I think we can just wrap each PGgetResult call into a GIL release (provided we have the connection lock, which should be true but I'll double check) as a tentative way to fix this issue. Walt should let us know in the next days if the patch is good for them anyway.

  • Daniele Varrazzo

    Daniele Varrazzo November 24th, 2012 @ 10:57 PM

    • State changed from “new” to “resolved”

    I've committed a patch for PQgetResult after copy. The other uses of the function shouldn't need it (the most important one is in async code, and by that we should have received the message that it won't block anyway)

    I'll mark the issue closed but let me know if it fails to fix the problem.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

People watching this ticket

Attachments

Tags

Pages