Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

psycopg exceptions cannot be pickled

#90
import psycopg2
conn = psycopg2.connect("dbname='somedb'")
cur = conn.cursor() 
try:
    cur.execute("SELECT * FROM barf")
except Exception, e:
    print e
    pass

import cPickle
a = cPickle.dumps(e)
l = cPickle.loads(a)

which gives:

relation "barf" does not exist
LINE 1: SELECT * FROM barf
                      ^
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle cursor objects

Exception pickling becomes an issue for example when one is using the multiprocessing module. See http://stackoverflow.com/questions/8785899/hang-in-python-script-us... , http://bugs.python.org/issue13751 , http://bugs.python.org/issue1692335 and http://www.sqlalchemy.org/trac/ticket/2371

See also http://stackoverflow.com/questions/2246384/multiprocessing-pool-han...

If you want I can produce a slightly more complicated example that uses multiprocessing, but I hope the given example is convincing. Thanks.

Reported by Faheem Mitha · January 11th, 2012 @ 11:39 AM

State: resolved
Milestone: none
Assigned to: nobody

Activity

  1. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from new to invalid

    It's not possible to pickle a connection nor a cursor because the connection wraps an open socket, which has a state that can't be pickled. It's not just a problem of constructor signature as in the links provided.

    Furthermore, don't try to share a connection across processes: it won't work (see the docs for relevant info). If you want to use psycopg with processing, make sure to create a connection in every process and return only the results from the cursor (well, you should do this with threads too, as cursors are not thread-safe whereas connections are).

    January 11th, 2012 @ 01:08 PM

  2. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from invalid to open

    Uhm... reading back, I see what you mean. Sorry, I see now you just want to pickle the exception.

    I think it can be done if the cursor is dropped from it. Will think about that.

    I didn't even know cursor were attached to exceptions...

    January 11th, 2012 @ 01:33 PM

  3. Federico Di Gregorio
    Federico Di Gregorio

    They are because when the exception propagates up the call stack you may want to issue a ROLLBACK from a piece of code that doesn't have an easy way to obtain the current connection/cursor.

    January 11th, 2012 @ 01:39 PM

  4. Daniele Varrazzo
    Daniele Varrazzo

    Yeah, makes entirely sense.

    I think it would be fine if we set cursor to None on pickle: because of the exception, we can say nobody has ever used pickle + exception.cursor together.

    January 11th, 2012 @ 03:38 PM

  5. Federico Di Gregorio
  6. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from open to resolved

    Fixed in my branch.

    January 14th, 2012 @ 06:11 PM