psycopg exceptions cannot be pickled
Reported by Faheem Mitha | January 11th, 2012 @ 11:39 AM
import psycopg2 conn = psycopg2.connect("dbname='somedb'") cur = conn.cursor() try: cur.execute("SELECT * FROM barf") except Exception, e: print e pass import cPickle a = cPickle.dumps(e) l = cPickle.loads(a)
relation "barf" does not exist LINE 1: SELECT * FROM barf ^ Traceback (most recent call last): File "<stdin>", line 11, in <module> File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex raise TypeError, "can't pickle %s objects" % base.__name__ TypeError: can't pickle cursor objects
Exception pickling becomes an issue for example when one is using the multiprocessing module. See http://stackoverflow.com/questions/8785899/hang-in-python-script-us... , http://bugs.python.org/issue13751 , http://bugs.python.org/issue1692335 and http://www.sqlalchemy.org/trac/ticket/2371
If you want I can produce a slightly more complicated example that uses multiprocessing, but I hope the given example is convincing. Thanks.
Comments and changes to this ticket
- State changed from new to invalid
It's not possible to pickle a connection nor a cursor because the connection wraps an open socket, which has a state that can't be pickled. It's not just a problem of constructor signature as in the links provided.
Furthermore, don't try to share a connection across processes: it won't work (see the docs for relevant info). If you want to use psycopg with processing, make sure to create a connection in every process and return only the results from the cursor (well, you should do this with threads too, as cursors are not thread-safe whereas connections are).
- State changed from invalid to open
Uhm... reading back, I see what you mean. Sorry, I see now you just want to pickle the exception.
I think it can be done if the cursor is dropped from it. Will think about that.
I didn't even know cursor were attached to exceptions...
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile »
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.