#192 ✓ invalid
Psycopg website

enhancement: connection.closed attribute to actually become a property that dynamically probes the current cxn state

Reported by Psycopg website | February 6th, 2014 @ 01:06 PM | in psycopg2 2.5.3 (closed)

Submitted by: matt.bradshaw@gmail.com

At present, it appears as though .closed can lag behind the truth of things. If one explicitly .close()'d the connection, it's accurate. Similarly, if one uses the connection and the library encounters trouble it flips the boolean to False. But what I want is an active probe to see if the connection is currently in existence. As it stands, one can stop the underlying database and the .closed attributed says that things are connected, in effect. Can we instead change .closed to be a property that probes the connection state at call time?

Comments and changes to this ticket

  • Daniele Varrazzo

    Daniele Varrazzo February 6th, 2014 @ 02:05 PM

    • State changed from “new” to “open”

    I would say no: users of that property don't expect an extra query to be performed to check the connection is still up. For one, it would become a blocking operation where people don't expect to be blocked.

    You can subclass the connection and execute a "select 1" on cnn.closed to perform your own probe. Or call cnn.poll() and catch any error.

    Note: looking into that I've found a couple of buglets:

    1) poll() on a broken connection doesn't mark the connection closed (so, if you want to play with poll, you better just trust the exception, or check cnn.get_transaction_status().

    In [4]: cnn.poll()
    ---------------------------------------------------------------------------
    OperationalError                          Traceback (most recent call last)
    /home/piro/<ipython-input-4-2d99764f7d0a> in <module>()
    ----> 1 cnn.poll()
    
    OperationalError: SSL SYSCALL error: EOF detected
    
    
    In [5]: cnn.closed
    Out[5]: 0
    

    2) the connection.closed docstring is wrong: it claims to be a boolean whereas it assumes values 0, 1, 2 where 2 means connection broken. Also, 2 is not mentioned in the docs.

    Leaving this bug open to fix these two points.

    Note: in an ideal world we shouldn't have cnn.closed and cnn.status attributes on the connection structure but only rely on PQtransactionStatus (the property exposed to python as get_transaction_status). I'm tempted to give a go at dropping the C attributes, reimplementing these properties reading PQtransactionStatus and see if anything breaks in the test suite.

  • Daniele Varrazzo

    Daniele Varrazzo March 6th, 2014 @ 06:19 PM

    • State changed from “open” to “resolved”

    Pushed a change for this bug in a github branch.

    https://github.com/dvarrazzo/psycopg/commits/connection-closed

    Will merge to master and release in 2.5.3 but first I want to fix poll too. Zaar, if you can test it it would be welcome.

  • Zaar Hai

    Zaar Hai March 10th, 2014 @ 07:03 AM

    Master does not compile:

    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.6.dev0 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x09010C -DPG_VERSION_HEX=0x09010C -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.1/server -c psycopg/connection_type.c -o build/temp.linux-x86_64-2.7/psycopg/connection_type.o -Wdeclaration-after-statement -Wdeclaration-after-statement
    
    psycopg/connection_type.c:1037:13: error: ‘psyco_conn_get_exception’ undeclared here (not in a function)
    
         { #exc, psyco_conn_get_exception, NULL, exc ## _doc, &exc }
    
                 ^
    
    psycopg/connection_type.c:1041:5: note: in expansion of macro ‘EXCEPTION_GETTER’
    
         EXCEPTION_GETTER(Error),
    
         ^
    
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
  • Daniele Varrazzo

    Daniele Varrazzo March 10th, 2014 @ 11:19 AM

    It does for me. psyco_conn_get_exception is declared in the same file at line 835 in current master.

  • Zaar Hai

    Zaar Hai March 10th, 2014 @ 02:36 PM

    I knew that :)

    Don't know what is the problem. Just run

    pip install -e git+https://github.com/dvarrazzo/psycopg.git#egg=psycopg2
    

    Simple pip install psycopg2 works flawlessly.

    I guess I'll wait for official 2.5.3.

    Thanks for fixing this issue!

    Zaar

  • Daniele Varrazzo

    Daniele Varrazzo March 10th, 2014 @ 02:44 PM

    Uhm... I was going to reply you that you can pip-install a project from a branch: you could have tested my branch with:

    pip install -e git+https://github.com/dvarrazzo/psycopg.git@connection-closed#egg=psycopg2
    

    but actually I tried that... and triggered your bug. So I'll check how to fix it.

    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DHAVE_MXDATETIME=1 -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.5.3.dev0 (dt dec mx pq3 ext)" -DPG_VERSION_HEX=0x090303 -DPG_VERSION_HEX=0x090303 -I/usr/include/python2.7/mx -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.3/server -c psycopg/connection_type.c -o build/temp.linux-x86_64-2.7/psycopg/connection_type.o -Wdeclaration-after-statement -Wdeclaration-after-statement
    
    psycopg/connection_type.c:1039:5: error: ‘psyco_conn_get_exception’ undeclared here (not in a function)
    
    error: command 'gcc' failed with exit status 1
    

    first attempt to reproduce the issue failed: "make" and "python setup.py build" in the checkout dir work. mmm...

  • Daniele Varrazzo

    Daniele Varrazzo March 10th, 2014 @ 03:01 PM

    The difference is that the failing version is building without:

    -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1
    

    and we haven't really tested using no PSYCOPG_EXTENSIONS in this century. I wonder why the setup.py/.cfg is not doing what expected: that's new.

  • Daniele Varrazzo

    Daniele Varrazzo March 10th, 2014 @ 04:40 PM

    For what I can see it's a pip bug which doesn't take in consideration the setup.cfg. I've reported the issue at https://github.com/pypa/pip/issues/1630

    Zaar, if you can help me testing, just check out psycopg from the branch in github and install it with a python setup.py install. That should just work.

  • Daniele Varrazzo

    Daniele Varrazzo March 10th, 2014 @ 05:28 PM

    • State changed from “resolved” to “open”
    • Milestone set to psycopg2 2.5.3

    This bug was closed by mistake: the one to be closed should have been the #196. To be reviewed for 2.5.3

  • Dave

    Dave March 25th, 2014 @ 01:22 AM

    Daniele,
    Don't know if this is related but when opening a connection I do the following.

    class Conn(psycopg2.extensions.connection):
        def __init__(self, dsnStr):
            try:
                super(Conn, self).__init__(dsnStr)
                print "Connection using < %s > is successful" % self.dsn
            except:
                self.close() # explicit close otherwise 'connection.closed' reads as open still
                print "Connection using < %s > failed" % self.dsn
        
        # bunch of db utility methods here
        
    conn = Conn('user=dpostgres password=fght98 host=localhost')
    if conn.closed == 0:
        # Do something with the db
    else:
        # Ask user to correct dsn parameters and try again
    

    If the connection attempt fails, the exception is caught.... but the closed attribute
    has already been set to 0 which should be 1 at this point. That's why I do a close().

    Maybe I'm doing it wrong and should not attempt to do anything with the class if
    the connection fails and just return False; but having access to the dsn attribute is a good thing.

    Cheers
    :)

  • Daniele Varrazzo

    Daniele Varrazzo March 25th, 2014 @ 08:39 AM

    Yes, I'd say it's related. It's a sort of corner case but I'll try to iron out that too for next release.

  • Dave

    Dave March 26th, 2014 @ 10:40 PM

    On the same subject,

    connection.dsn
    
    will normally return a hidden password, however if the connection class instantiation is partial due to a connection error as per my previous example,
    connection.dsn
    
    will now expose the password used in the attempted connection.
  • Daniele Varrazzo

    Daniele Varrazzo April 3rd, 2014 @ 12:43 AM

    Fixed the problems with dsn and closed in connection subclasses. Please open a different issue if you have something else to report :)

  • Daniele Varrazzo

    Daniele Varrazzo April 3rd, 2014 @ 10:35 AM

    Open ticket #204 to deal with the pip -e problem

  • Daniele Varrazzo

    Daniele Varrazzo April 5th, 2014 @ 02:41 PM

    • State changed from “open” to “invalid”

    Fixed the poll() and docs reworded. However poll() doesn't notice immediately either that the connection got closed: it takes another roundtrip. Example:

    connect and restart the server

    In [2]: import psycopg2    
    In [3]: cnn = psycopg2.connect('')
    
    In [4]: !sudo /etc/init.d/postgresql stop 9.2
     * Stopping PostgreSQL 9.2 database server   ...  [ OK ]
    

    the first poll succeeds, although a notice is sent that the server restarted

    In [5]: cnn.poll()
    Out[5]: 0
    
    In [6]: cnn.notices
    Out[6]: ['FATAL:  terminating connection due to administrator command\n']
    

    the libpq still thinks the connection is good

    In [7]: cnn.get_transaction_status()
    Out[7]: 0
    

    the following poll finds the connection broken

    In [8]: cnn.poll()
    ---------------------------------------------------------------------------
    OperationalError                          Traceback (most recent call last)
    /home/piro/dev/psycopg2/build/lib.2.7/<ipython-input-8-2d99764f7d0a> in <module>()
    ----> 1 cnn.poll()
    
    OperationalError: 
    
    In [9]: cnn.get_transaction_status()
    Out[9]: 4
    

    but at least now (psycopg 2.5.3 the connection is marked closed)

    In [10]: cnn.closed
    Out[10]: 2
    

    Closing this bug as I've cleaned up all the glitches that cropped out of it but closed will stay a client-side only operation. Cheers!

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.

New-ticket Create new ticket

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

Referenced by

Pages