Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

There was a problem

You must be a member of this account.

This project is archived and is in readonly mode.

psycopg exceptions cannot be pickled

#90
import psycopg2
conn = psycopg2.connect("dbname='somedb'")
cur = conn.cursor() 
try:
    cur.execute("SELECT * FROM barf")
except Exception, e:
    print e
    pass

import cPickle
a = cPickle.dumps(e)
l = cPickle.loads(a)

which gives:

relation "barf" does not exist
LINE 1: SELECT * FROM barf
                      ^
Traceback (most recent call last):
  File "<stdin>", line 11, in <module>
  File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle cursor objects

Exception pickling becomes an issue for example when one is using the multiprocessing module. See http://stackoverflow.com/questions/8785899/hang-in-python-script-us... , http://bugs.python.org/issue13751 , http://bugs.python.org/issue1692335 and http://www.sqlalchemy.org/trac/ticket/2371

See also http://stackoverflow.com/questions/2246384/multiprocessing-pool-han...

If you want I can produce a slightly more complicated example that uses multiprocessing, but I hope the given example is convincing. Thanks.

Reported by Faheem Mitha · January 11th, 2012 @ 11:39 AM

State: resolved
Milestone: none
Assigned to: nobody

Activity

  1. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from new to invalid

    It's not possible to pickle a connection nor a cursor because the connection wraps an open socket, which has a state that can't be pickled. It's not just a problem of constructor signature as in the links provided.

    Furthermore, don't try to share a connection across processes: it won't work (see the docs for relevant info). If you want to use psycopg with processing, make sure to create a connection in every process and return only the results from the cursor (well, you should do this with threads too, as cursors are not thread-safe whereas connections are).

    January 11th, 2012 @ 01:08 PM

  2. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from invalid to open

    Uhm... reading back, I see what you mean. Sorry, I see now you just want to pickle the exception.

    I think it can be done if the cursor is dropped from it. Will think about that.

    I didn't even know cursor were attached to exceptions...

    January 11th, 2012 @ 01:33 PM

  3. Federico Di Gregorio
    Federico Di Gregorio

    They are because when the exception propagates up the call stack you may want to issue a ROLLBACK from a piece of code that doesn't have an easy way to obtain the current connection/cursor.

    January 11th, 2012 @ 01:39 PM

  4. Daniele Varrazzo
    Daniele Varrazzo

    Yeah, makes entirely sense.

    I think it would be fine if we set cursor to None on pickle: because of the exception, we can say nobody has ever used pickle + exception.cursor together.

    January 11th, 2012 @ 03:38 PM

  5. Federico Di Gregorio
  6. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from open to resolved

    Fixed in my branch.

    January 14th, 2012 @ 06:11 PM