This project is archived and is in readonly mode.

#90 ✓resolved
Faheem Mitha

psycopg exceptions cannot be pickled

Reported by Faheem Mitha | January 11th, 2012 @ 11:39 AM

import psycopg2
conn = psycopg2.connect("dbname='somedb'")
cur = conn.cursor() 
try:
    cur.execute("SELECT * FROM barf")
except Exception, e:
    print e
    pass

import cPickle
a = cPickle.dumps(e)
l = cPickle.loads(a)

which gives:

relation "barf" does not exist
LINE 1: SELECT * FROM barf
                      ^
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle cursor objects

Exception pickling becomes an issue for example when one is using the multiprocessing module. See http://stackoverflow.com/questions/8785899/hang-in-python-script-us... , http://bugs.python.org/issue13751 , http://bugs.python.org/issue1692335 and http://www.sqlalchemy.org/trac/ticket/2371

See also http://stackoverflow.com/questions/2246384/multiprocessing-pool-han...

If you want I can produce a slightly more complicated example that uses multiprocessing, but I hope the given example is convincing. Thanks.

Comments and changes to this ticket

  • Daniele Varrazzo

    Daniele Varrazzo January 11th, 2012 @ 01:08 PM

    • State changed from “new” to “invalid”

    It's not possible to pickle a connection nor a cursor because the connection wraps an open socket, which has a state that can't be pickled. It's not just a problem of constructor signature as in the links provided.

    Furthermore, don't try to share a connection across processes: it won't work (see the docs for relevant info). If you want to use psycopg with processing, make sure to create a connection in every process and return only the results from the cursor (well, you should do this with threads too, as cursors are not thread-safe whereas connections are).

  • Daniele Varrazzo

    Daniele Varrazzo January 11th, 2012 @ 01:33 PM

    • State changed from “invalid” to “open”

    Uhm... reading back, I see what you mean. Sorry, I see now you just want to pickle the exception.

    I think it can be done if the cursor is dropped from it. Will think about that.

    I didn't even know cursor were attached to exceptions...

  • Federico Di Gregorio

    Federico Di Gregorio January 11th, 2012 @ 01:39 PM

    They are because when the exception propagates up the call stack you may want to issue a ROLLBACK from a piece of code that doesn't have an easy way to obtain the current connection/cursor.

  • Daniele Varrazzo

    Daniele Varrazzo January 11th, 2012 @ 03:38 PM

    Yeah, makes entirely sense.

    I think it would be fine if we set cursor to None on pickle: because of the exception, we can say nobody has ever used pickle + exception.cursor together.

  • Federico Di Gregorio
  • Daniele Varrazzo

    Daniele Varrazzo January 14th, 2012 @ 06:11 PM

    • State changed from “open” to “resolved”

    Fixed in my branch.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

Pages