Psycopg psycopg

When attempting to pickle an OperationalError, I get an immediate segfault. I understand that OperationalError might not be pickle-able, but I would expect a separate Python exception as a result, not a segfault.

• Host: Ubuntu Linux 3.5.0-34-generic SMP i686
• Python: 2.7.3
• psycopg2: 2.5

Minimal repro script:

import pickle
from psycopg2 import connect, OperationalError

print 'Start.'
try:
  conn = connect("host='doesnotexist' dbname='foo'")
  print 'Connected.'
except OperationalError as e:
  print 'Exception.'
  pickled = pickle.dumps(e)
  print 'Pickled.'

print 'Fin.'

Output:

Start.
Exception.
Segmentation fault (core dumped)

When debugging with gdb:

Program received signal SIGSEGV, Segmentation fault.
dict_set_item_by_hash_or_entry (value=0x0, ep=0x0, hash=-2121493956, key='pgerror', op={})
    at ../Objects/dictobject.c:762
762 ../Objects/dictobject.c: No such file or directory.

(gdb) bt
#0  dict_set_item_by_hash_or_entry (value=0x0, ep=0x0, hash=-2121493956, key='pgerror', op={})
    at ../Objects/dictobject.c:762
#1  PyDict_SetItem (op=op@entry={}, key='pgerror', value=value@entry=0x0) at ../Objects/dictobject.c:818
#2  0x080f482f in PyDict_SetItemString (v={}, key=key@entry=0xb7ac6b9b "pgerror", item=0x0)
    at ../Objects/dictobject.c:2438
#3  0xb7ab9990 in psyco_error_reduce (self=0x838ef7c) at psycopg/error_type.c:166
#4  0x080e4975 in PyObject_Call (kw=0x0, arg=(), func=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>) at ../Objects/abstract.c:2529
#5  PyEval_CallObjectWithKeywords (func=func@entry=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>, arg=(), arg@entry=0x0, kw=kw@entry=
    0x0) at ../Python/ceval.c:3890
#6  0x0814869f in PyObject_CallObject (o=o@entry=
    <built-in method __reduce__ of OperationalError object at remote 0x838ef7c>, a=a@entry=0x0)
    at ../Objects/abstract.c:2517
#7  0x08115568 in object_reduce_ex.25465 (self=<OperationalError at remote 0x838ef7c>, args=(0,))
    at ../Objects/typeobject.c:3412
#8  0x080a517b in call_function (oparg=<optimized out>, pp_stack=0xbfffeb9c) at ../Python/ceval.c:4021
#9  PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83ec5a4, for file /usr/lib/python2.7/pickle.py, line 306, in save (self=<Pickler(write=<built-in method write of cStringIO.StringO object at remote 0xb7d02540>, bin=False, memo={}, fast=0, proto=0) at remote 0x839646c>, obj=<OperationalError at remote 0x838ef7c>, pid=None, x=None, t=<type at remote 0x83702bc>, f=None, reduce=<built-in method __reduce_ex__ of OperationalError object at remote 0x838ef7c>, issc=False), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#10 0x080a5728 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0xbfffec8c, func=
    <function at remote 0xb7cfe924>) at ../Python/ceval.c:4107
#11 call_function (oparg=<optimized out>, pp_stack=0xbfffec8c) at ../Python/ceval.c:4042
#12 PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83c39a4, for file /usr/lib/python2.7/pickle.py, line 224, in dump (self=<Pickler(write=<built-in method write of cStringIO.StringO object at remote 0xb7d02540>, bin=False, memo={}, fast=0, proto=0) at remote 0x839646c>, obj=<OperationalError at remote 0x838ef7c>), throwflag=throwflag@entry=0) at ../Python/ceval.c:2666
#13 0x080a5728 in fast_function (nk=<optimized out>, na=<optimized out>, n=2, pp_stack=0xbfffed7c, func=
    <function at remote 0xb7cfe844>) at ../Python/ceval.c:4107
#14 call_function (oparg=<optimized out>, pp_stack=0xbfffed7c) at ../Python/ceval.c:4042
#15 PyEval_EvalFrameEx (f=f@entry=
    Frame 0x83fa114, for file /usr/lib/python2.7/pickle.py, line 1374, in dumps (obj=<OperationalError at remote 0x838ef7c>, protocol=None, file=<cStringIO.StringO at remote 0xb7d02540>), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#16 0x080abac9 in PyEval_EvalCodeEx (co=0xb7cfca88, globals=globals@entry=
    {'EMPTY_DICT': '}', 'NEWTRUE': '\x88', 'TypeType': <type at remote 0x8281c80>, 'Unpickler': <classobj at remote 0xb7cedecc>, 'whichmodule': <function at remote 0xb7cfed4c>, 'BuiltinFunctionType': <type at remote 0x8281740>, 'CodeType': <type at remote 0x82814a0>, 'TRUE': 'I01\n', 'SETITEMS': 'u', 'struct': <module at remote 0xb7cf93ec>, 'LONG': 'L', 'DictProxyType': <type at remote 0x827be00>, '_EmptyClass': <classobj at remote 0xb7cedefc>, 'GET': 'g', 'ObjectType': <type at remote 0x827e3a0>, 'DictType': <type at remote 0x8281ac0>, 'EXT4': '\x84', 'EXT2': '\x83', 'EXT1': '\x82', '__file__': '/usr/lib/python2.7/pickle.pyc', 'BUILD': 'b', 'ListType': <type at remote 0x8281ba0>, 'MethodType': <type at remote 0x827b700>, 'mloads': <built-in function loads>, 'ModuleType': <type at remote 0x8281580>, 'TracebackType': <type at remote 0x8281380>, 'dumps': <function at remote 0xb7d01bc4>, 'loads': <function at remote 0xb7d01c34>, 'LambdaType': <type at remote 0x8281820>, 'XRangeType': <type at remote 0x827d220>, 'for...(truncated), locals=locals@entry=0x0, 
    args=args@entry=0x82d9320, argcount=argcount@entry=1, kws=0x82d9324, kwcount=0, defs=0xb7d022b8, defcount=1, 
    closure=0x0) at ../Python/ceval.c:3253
#17 0x080a57cd in fast_function (nk=<optimized out>, na=1, n=<optimized out>, pp_stack=0xbfffeeec, func=
    <function at remote 0xb7d01bc4>) at ../Python/ceval.c:4117
#18 call_function (oparg=<optimized out>, pp_stack=0xbfffeeec) at ../Python/ceval.c:4042
#19 PyEval_EvalFrameEx (f=f@entry=Frame 0x82d91e4, for file p.py, line 10, in <module> (), throwflag=throwflag@entry=0)
    at ../Python/ceval.c:2666
#20 0x080abac9 in PyEval_EvalCodeEx (co=co@entry=0xb7d50cc8, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, args=args@entry=0x0, 
    argcount=argcount@entry=0, kws=kws@entry=0x0, kwcount=kwcount@entry=0, defs=defs@entry=0x0, 
    defcount=defcount@entry=0, closure=closure@entry=0x0) at ../Python/ceval.c:3253
#21 0x08113b57 in PyEval_EvalCode (co=co@entry=0xb7d50cc8, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}) at ../Python/ceval.c:667
#22 0x0814ca7d in run_mod.42766 (mod=mod@entry=0x8354410, filename=filename@entry=0xbffff3c4 "p.py", 
    globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, flags=flags@entry=0xbffff0ec, 
    arena=arena@entry=0x82e2f88) at ../Python/pythonrun.c:1365
#23 0x0808e4d2 in PyRun_FileExFlags (fp=fp@entry=0x82d91d8, filename=filename@entry=0xbffff3c4 "p.py", 
    start=start@entry=257, globals=globals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, locals=locals@entry=
    {'e': <OperationalError at remote 0x838ef7c>, '__builtins__': <module at remote 0xb7d5611c>, '__file__': 'p.py', '__package__': None, 'connect': <function at remote 0x838ef0c>, '__name__': '__main__', 'pickle': <module at remote 0xb7cf93bc>, '__doc__': None, 'OperationalError': <type at remote 0x83702bc>}, closeit=closeit@entry=1, flags=flags@entry=
    0xbffff0ec) at ../Python/pythonrun.c:1351
#24 0x0808ea80 in PyRun_SimpleFileExFlags (fp=0x82d91d8, filename=<optimized out>, closeit=1, flags=0xbffff0ec)
    at ../Python/pythonrun.c:943
#25 0x0808eb42 in PyRun_AnyFileExFlags (fp=fp@entry=0x82d91d8, filename=filename@entry=0xbffff3c4 "p.py", 
    closeit=closeit@entry=1, flags=flags@entry=0xbffff0ec) at ../Python/pythonrun.c:747
#26 0x0808f917 in Py_Main (argc=argc@entry=2, argv=argv@entry=0xbffff224) at ../Modules/main.c:639
#27 0x0808f9e9 in main (argc=2, argv=0xbffff224) at ../Modules/python.c:23

(gdb) py-list
 301                    return
 302    
 303                # Check for a __reduce_ex__ method, fall back to __reduce__
 304                reduce = getattr(obj, "__reduce_ex__", None)
 305                if reduce:
>306                    rv = reduce(self.proto)
 307                else:
 308                    reduce = getattr(obj, "__reduce__", None)
 309                    if reduce:
 310                        rv = reduce()
 311                    else:

For a little more context on this issue, I am using psycopg2 connections in some worker code running under Celery. When these errors occur, Celery attempts to pickle the exception, which causes a segfault that Python then handles and (for some reason) ignores, causing an infinite loop of segfault > trap > resume > segfault > ... and 100% CPU usage.

Thanks & regards,
Chris

P.S. While I completely respect your decision in how you manage your software, I must put in my vote for you to use GitHub's issue tracker. I say this because I am far more inclined to raise issues and support open source with a single, centralized account rather than having to create a new account for each piece of software I wish to give feedback on. Also, running into Lighthouse problems while even trying to post this issue was pretty discouraging. With that said, though, thanks for your work on this project!