This project is archived and is in readonly mode.

Pickling OperationalError causes segfault with Python 2.7.3

#170

When attempting to pickle an OperationalError, I get an immediate segfault. I understand that OperationalError might not be pickle-able, but I would expect a separate Python exception as a result, not a segfault.

Host: Ubuntu Linux 3.5.0-34-generic SMP i686
Python: 2.7.3
psycopg2: 2.5

Minimal repro script:

import pickle
from psycopg2 import connect, OperationalError

print 'Start.'
try:
  conn = connect("host='doesnotexist' dbname='foo'")
  print 'Connected.'
except OperationalError as e:
  print 'Exception.'
  pickled = pickle.dumps(e)
  print 'Pickled.'

print 'Fin.'

Output:

Start.
Exception.
Segmentation fault (core dumped)

When debugging with gdb:

Program received signal SIGSEGV, Segmentation fault.
dict_set_item_by_hash_or_entry (value=0x0, ep=0x0, hash=-2121493956, key='pgerror', op={})
    at ../Objects/dictobject.c:762
762 ../Objects/dictobject.c: No such file or directory.

(gdb) bt
#0  dict_set_item_by_hash_or_entry (value=0x0, ep=0x0, hash=-2121493956, key='pgerror', op={})
    at ../Objects/dictobject.c:762
#1  PyDict_SetItem (op=op@entry={}, key='pgerror', value=value@entry=0x0) at ../Objects/dictobject.c:818
#2  0x080f482f in PyDict_SetItemString (v={}, key=key@entry=0xb7ac6b9b "pgerror", item=0x0)
    at ../Objects/dictobject.c:2438
#3  0xb7ab9990 in psyco_error_reduce (self=0x838ef7c) at psycopg/error_type.c:166
#4  0x080e4975 in PyObject_Call (kw=0x0, arg=(), func=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>) at ../Objects/abstract.c:2529
#5  PyEval_CallObjectWithKeywords (func=func@entry=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>, arg=(), arg@entry=0x0, kw=kw@entry=
    0x0) at ../Python/ceval.c:3890
#6  0x0814869f in PyObject_CallObject (o=o@entry=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>, a=a@entry=0x0)
    at ../Objects/abstract.c:2517
#7  0x08115568 in object_reduce_ex.25465 (self=<OperationalError at remote 0x838ef7c>, args=(0,))
    at ../Objects/typeobject.c:3412
#8  0x080a517b in call_function (oparg=<optimized out>, pp_stack=0xbfffeb9c) at ../Python/ceval.c:4021
#9  PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83ec5a4, for file /usr/lib/python2.7/pickle.py, line 306, in save (self=<Pickler(write=<built-in method write of cStringIO.StringO object at remote 0xb7d02540>, bin=False, memo={}, fast=0, proto=0) at remote 0x839646c>, obj=<OperationalError at remote 0x838ef7c>, pid=None, x=None, t=<type at remote 0x83702bc>, f=None, reduce=<built-in method __reduce_ex__ of OperationalError object at remote 0x838ef7c>, issc=False), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#10 0x080a5728 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0xbfffec8c, func=
    <function at remote 0xb7cfe924>) at ../Python/ceval.c:4107
#11 call_function (oparg=<optimized out>, pp_stack=0xbfffec8c) at ../Python/ceval.c:4042
#12 PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83c39a4, for file /usr/lib/python2.7/pickle.py, line 224, in dump (self=<Pickler(write=<built-in method write of cStringIO.StringO object at remote 0xb7d02540>, bin=False, memo={}, fast=0, proto=0) at remote 0x839646c>, obj=<OperationalError at remote 0x838ef7c>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#13 0x080a5728 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0xbfffed7c, func=
    <function at remote 0xb7cfe844>) at ../Python/ceval.c:4107
#14 call_function (oparg=<optimized out>, pp_stack=0xbfffed7c) at ../Python/ceval.c:4042
#15 PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83fa114, for file /usr/lib/python2.7/pickle.py, line 1374, in dumps (obj=<OperationalError at remote 0x838ef7c>, protocol=None, file=<cStringIO.StringO at remote 0xb7d02540>), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#16 0x080abac9 in PyEval_EvalCodeEx (co=0xb7cfca88, globals=globals@entry=
    {'EMPTY_DICT': '}', 'NEWTRUE': '\x88', 'TypeType': <type at remote 0x8281c80>, 'Unpickler': <classobj at remote 0xb7cedecc>, 'whichmodule': <function at remote 0xb7cfed4c>, 'BuiltinFunctionType': <type at remote 0x8281740>, 'CodeType': <type at remote 0x82814a0>, 'TRUE': 'I01\n', 'SETITEMS': 'u', 'struct': <module at remote 0xb7cf93ec>, 'LONG': 'L', 'DictProxyType': <type at remote 0x827be00>, '_EmptyClass': <classobj at remote 0xb7cedefc>, 'GET': 'g', 'ObjectType': <type at remote 0x827e3a0>, 'DictType': <type at remote 0x8281ac0>, 'EXT4': '\x84', 'EXT2': '\x83', 'EXT1': '\x82', '__file__': '/usr/lib/python2.7/pickle.pyc', 'BUILD': 'b', 'ListType': <type at remote 0x8281ba0>, 'MethodType': <type at remote 0x827b700>, 'mloads': <built-in function loads>, 'ModuleType': <type at remote 0x8281580>, 'TracebackType': <type at remote 0x8281380>, 'dumps': <function at remote 0xb7d01bc4>, 'loads': <function at remote 0xb7d01c34>, 'LambdaType': <type at remote 0x8281820>, 'XRangeType': <type at remote 0x827d220>, 'for...(truncated), locals=locals@entry=0x0, 
    args=args@entry=0x82d9320, argcount=argcount@entry=1, kws=0x82d9324, kwcount=0, defs=0xb7d022b8, defcount=1, 
    closure=0x0) at ../Python/ceval.c:3253
#17 0x080a57cd in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0xbfffeeec, func=
    <function at remote 0xb7d01bc4>) at ../Python/ceval.c:4117
#18 call_function (oparg=<optimized out>, pp_stack=0xbfffeeec) at ../Python/ceval.c:4042
#19 PyEval_EvalFrameEx (f=f@entry=Frame 0x82d91e4, for file p.py, line 10, in <module> (), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#20 0x080abac9 in PyEval_EvalCodeEx (co=co@entry=0xb7d50cc8, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, args=args@entry=0x0, 
    argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=closure@entry=0x0) at ../Python/ceval.c:3253
#21 0x08113b57 in PyEval_EvalCode (co=co@entry=0xb7d50cc8, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}) at ../Python/ceval.c:667
#22 0x0814ca7d in run_mod.42766 (mod=mod@entry=0x8354410, filename=filename@entry=0xbffff3c4 "p.py", 
    globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, flags=flags@entry=0xbffff0ec, 
    arena=arena@entry=0x82e2f88) at ../Python/pythonrun.c:1365
#23 0x0808e4d2 in PyRun_FileExFlags (fp=fp@entry=0x82d91d8, filename=filename@entry=0xbffff3c4 "p.py", 
    start=start@entry=257, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, closeit=closeit@entry=1, flags=flags@entry=
    0xbffff0ec) at ../Python/pythonrun.c:1351
#24 0x0808ea80 in PyRun_SimpleFileExFlags (fp=0x82d91d8, filename=<optimized out>, closeit=1, flags=0xbffff0ec)
    at ../Python/pythonrun.c:943
#25 0x0808eb42 in PyRun_AnyFileExFlags (fp=fp@entry=0x82d91d8, filename=filename@entry=0xbffff3c4 "p.py", 
    closeit=closeit@entry=1, flags=flags@entry=0xbffff0ec) at ../Python/pythonrun.c:747
#26 0x0808f917 in Py_Main (argc=argc@entry=2, argv=argv@entry=0xbffff224) at ../Modules/main.c:639
#27 0x0808f9e9 in main (argc=2, argv=0xbffff224) at ../Modules/python.c:23

(gdb) py-list
 301                    return
 302    
 303                # Check for a __reduce_ex__ method, fall back to __reduce__
 304                reduce = getattr(obj, "__reduce_ex__", None)
 305                if reduce:
>306                    rv = reduce(self.proto)
 307                else:
 308                    reduce = getattr(obj, "__reduce__", None)
 309                    if reduce:
 310                        rv = reduce()
 311                    else:

For a little more context on this issue, I am using psycopg2 connections in some worker code running under Celery. When these errors occur, Celery attempts to pickle the exception, which causes a segfault that Python then handles and (for some reason) ignores, causing an infinite loop of segfault > trap > resume > segfault > ... and 100% CPU usage.

Thanks & regards,
Chris

P.S. While I completely respect your decision in how you manage your software, I must put in my vote for you to use GitHub's issue tracker. I say this because I am far more inclined to raise issues and support open source with a single, centralized account rather than having to create a new account for each piece of software I wish to give feedback on. Also, running into Lighthouse problems while even trying to post this issue was pretty discouraging. With that said, though, thanks for your work on this project!

Reported by Chris Schmich · July 12th, 2013 @ 09:12 PM

Activity

Chris Schmich
More details: this appears to be an issue with the base error class, psycopg2.Error. I hit a segfault when trying to pickle any of the exceptions inheriting from this (OperationalError, InterfaceError, DatabaseError, ... ).

I was able to successfully pickle psycopg2.Warning instances.

July 12th, 2013 @ 11:40 PM
Daniele Varrazzo
- State changed from new to open
- Tag set to rel-2.5.1
Chris,

thank you for your report. Yes, bug confirmed. The problem is probably with exceptions containing no result (e.g. generated on connect), because for the ones generated after db operations pickling is fine (and unit-tested):
```
print 'Start.'
try:
  conn = connect("")
  cur = conn.cursor()
  cur.execute("select * from nowhere")
  print 'Connected.'
except Error as e:
  print 'Exception.'
  pickled = pickle.dumps(e)
  print 'Pickled.'
```
I'll add your case to the test suite too.

About this issue tracker: yes, I was thinking about the problem yesterday after answering to #169 Cannot submit tickets through the Lighthouse website. I would love to have Lighthouse history exported to github though.
Please check #171 Move out of Lighhouse for my thoughts (and feel free to give a hand).
July 13th, 2013 @ 09:40 AM
Chris Schmich

Hi Daniele,

Thanks for the quick response, and sorry for being so delayed in mine.

Your explanation makes sense, and it's consistent with what I'm seeing.

I'd be happy to help with the migration to GitHub (I started watching issue #171 Move out of Lighhouse). I don't want to overlap with any work you guys are doing, so if there's a specific task you'd like me to do (e.g. writing an automation script to copy tickets from Lighthouse to GitHub), let me know.

Thanks again!
Chris

July 18th, 2013 @ 11:57 PM
Daniele Varrazzo
- State changed from open to resolved
Issue fixed in my branch: https://github.com/dvarrazzo/psycopg/commit/0e08fbb20b9f5bf004d8ab7...

To be released in 2.5.2.

July 19th, 2013 @ 03:12 PM