This project is archived and is in readonly mode.
psycopg exceptions cannot be pickled
Reported by Faheem Mitha | January 11th, 2012 @ 11:39 AM
import psycopg2
conn = psycopg2.connect("dbname='somedb'")
cur = conn.cursor()
try:
cur.execute("SELECT * FROM barf")
except Exception, e:
print e
pass
import cPickle
a = cPickle.dumps(e)
l = cPickle.loads(a)
which gives:
relation "barf" does not exist
LINE 1: SELECT * FROM barf
^
Traceback (most recent call last):
File "<stdin>", line 11, in <module>
File "/usr/lib/python2.6/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle cursor objects
Exception pickling becomes an issue for example when one is using the multiprocessing module. See http://stackoverflow.com/questions/8785899/hang-in-python-script-us... , http://bugs.python.org/issue13751 , http://bugs.python.org/issue1692335 and http://www.sqlalchemy.org/trac/ticket/2371
See also http://stackoverflow.com/questions/2246384/multiprocessing-pool-han...
If you want I can produce a slightly more complicated example that uses multiprocessing, but I hope the given example is convincing. Thanks.
Comments and changes to this ticket
-
Daniele Varrazzo January 11th, 2012 @ 01:08 PM
- State changed from “new” to “invalid”
It's not possible to pickle a connection nor a cursor because the connection wraps an open socket, which has a state that can't be pickled. It's not just a problem of constructor signature as in the links provided.
Furthermore, don't try to share a connection across processes: it won't work (see the docs for relevant info). If you want to use psycopg with processing, make sure to create a connection in every process and return only the results from the cursor (well, you should do this with threads too, as cursors are not thread-safe whereas connections are).
-
Daniele Varrazzo January 11th, 2012 @ 01:33 PM
- State changed from “invalid” to “open”
Uhm... reading back, I see what you mean. Sorry, I see now you just want to pickle the exception.
I think it can be done if the cursor is dropped from it. Will think about that.
I didn't even know cursor were attached to exceptions...
-
Federico Di Gregorio January 11th, 2012 @ 01:39 PM
They are because when the exception propagates up the call stack you may want to issue a ROLLBACK from a piece of code that doesn't have an easy way to obtain the current connection/cursor.
-
Daniele Varrazzo January 11th, 2012 @ 03:38 PM
Yeah, makes entirely sense.
I think it would be fine if we set cursor to None on pickle: because of the exception, we can say nobody has ever used pickle + exception.cursor together.
-
-
Daniele Varrazzo January 14th, 2012 @ 06:11 PM
- State changed from “open” to “resolved”
Fixed in my branch.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.
Daniele Varrazzo
Faheem Mitha
Federico Di Gregorio