Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

enhancement: connection.closed attribute to actually become a property that dynamically probes the current cxn state

#192

Submitted by: matt.bradshaw@gmail.com

At present, it appears as though .closed can lag behind the truth of things. If one explicitly .close()'d the connection, it's accurate. Similarly, if one uses the connection and the library encounters trouble it flips the boolean to False. But what I want is an active probe to see if the connection is currently in existence. As it stands, one can stop the underlying database and the .closed attributed says that things are connected, in effect. Can we instead change .closed to be a property that probes the connection state at call time?

Reported by Psycopg website · February 6th, 2014 @ 01:06 PM

State: invalid
Milestone: psycopg2 2.5.3
Assigned to: nobody

Activity

  1. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from new to open

    I would say no: users of that property don't expect an extra query to be performed to check the connection is still up. For one, it would become a blocking operation where people don't expect to be blocked.

    You can subclass the connection and execute a "select 1" on cnn.closed to perform your own probe. Or call cnn.poll() and catch any error.

    Note: looking into that I've found a couple of buglets:

    1) poll() on a broken connection doesn't mark the connection closed (so, if you want to play with poll, you better just trust the exception, or check cnn.get_transaction_status().

    In [4]: cnn.poll()
    ---------------------------------------------------------------------------
    OperationalError                          Traceback (most recent call last)
    /home/piro/<ipython-input-4-2d99764f7d0a> in <module>()
    ----> 1 cnn.poll()
    
    OperationalError: SSL SYSCALL error: EOF detected
    
    
    In [5]: cnn.closed
    Out[5]: 0
    

    2) the connection.closed docstring is wrong: it claims to be a boolean whereas it assumes values 0, 1, 2 where 2 means connection broken. Also, 2 is not mentioned in the docs.

    Leaving this bug open to fix these two points.

    Note: in an ideal world we shouldn't have cnn.closed and cnn.status attributes on the connection structure but only rely on PQtransactionStatus (the property exposed to python as get_transaction_status). I'm tempted to give a go at dropping the C attributes, reimplementing these properties reading PQtransactionStatus and see if anything breaks in the test suite.

    February 6th, 2014 @ 02:05 PM

  2. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from open to resolved

    Pushed a change for this bug in a github branch.

    https://github.com/dvarrazzo/psycopg/commits/connection-closed

    Will merge to master and release in 2.5.3 but first I want to fix poll too. Zaar, if you can test it it would be welcome.

    March 6th, 2014 @ 06:19 PM

  3. Zaar Hai
    Zaar Hai

    Master does not compile:

    x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.6.dev0 (dt dec pq3 ext)" -DPG_VERSION_HEX=0x09010C -DPG_VERSION_HEX=0x09010C -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.1/server -c psycopg/connection_type.c -o build/temp.linux-x86_64-2.7/psycopg/connection_type.o -Wdeclaration-after-statement -Wdeclaration-after-statement
    
    psycopg/connection_type.c:1037:13: error: ‘psyco_conn_get_exception’ undeclared here (not in a function)
    
         { #exc, psyco_conn_get_exception, NULL, exc ## _doc, &exc }
    
                 ^
    
    psycopg/connection_type.c:1041:5: note: in expansion of macro ‘EXCEPTION_GETTER’
    
         EXCEPTION_GETTER(Error),
    
         ^
    
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    

    March 10th, 2014 @ 07:03 AM

  4. Daniele Varrazzo
    Daniele Varrazzo

    It does for me. psyco_conn_get_exception is declared in the same file at line 835 in current master.

    March 10th, 2014 @ 11:19 AM

  5. Zaar Hai
    Zaar Hai

    I knew that :)

    Don't know what is the problem. Just run

    pip install -e git+https://github.com/dvarrazzo/psycopg.git#egg=psycopg2
    

    Simple pip install psycopg2 works flawlessly.

    I guess I'll wait for official 2.5.3.

    Thanks for fixing this issue!

    Zaar

    March 10th, 2014 @ 02:36 PM

  6. Daniele Varrazzo
    Daniele Varrazzo

    Uhm... I was going to reply you that you can pip-install a project from a branch: you could have tested my branch with:

    pip install -e git+https://github.com/dvarrazzo/psycopg.git@connection-closed#egg=psycopg2
    

    but actually I tried that... and triggered your bug. So I'll check how to fix it.

    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -DHAVE_MXDATETIME=1 -DPSYCOPG_DEFAULT_PYDATETIME=1 -DPSYCOPG_VERSION="2.5.3.dev0 (dt dec mx pq3 ext)" -DPG_VERSION_HEX=0x090303 -DPG_VERSION_HEX=0x090303 -I/usr/include/python2.7/mx -I/usr/include/python2.7 -I. -I/usr/include/postgresql -I/usr/include/postgresql/9.3/server -c psycopg/connection_type.c -o build/temp.linux-x86_64-2.7/psycopg/connection_type.o -Wdeclaration-after-statement -Wdeclaration-after-statement
    
    psycopg/connection_type.c:1039:5: error: ‘psyco_conn_get_exception’ undeclared here (not in a function)
    
    error: command 'gcc' failed with exit status 1
    

    first attempt to reproduce the issue failed: "make" and "python setup.py build" in the checkout dir work. mmm...

    March 10th, 2014 @ 02:44 PM

  7. Daniele Varrazzo
    Daniele Varrazzo

    The difference is that the failing version is building without:

    -DPSYCOPG_EXTENSIONS=1 -DPSYCOPG_NEW_BOOLEAN=1 -DHAVE_PQFREEMEM=1
    

    and we haven't really tested using no PSYCOPG_EXTENSIONS in this century. I wonder why the setup.py/.cfg is not doing what expected: that's new.

    March 10th, 2014 @ 03:01 PM

  8. Daniele Varrazzo
    Daniele Varrazzo

    For what I can see it's a pip bug which doesn't take in consideration the setup.cfg. I've reported the issue at https://github.com/pypa/pip/issues/1630

    Zaar, if you can help me testing, just check out psycopg from the branch in github and install it with a python setup.py install. That should just work.

    March 10th, 2014 @ 04:40 PM

  9. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from resolved to open
    • Milestone set to psycopg2 2.5.3

    This bug was closed by mistake: the one to be closed should have been the #196 connection.closed attribute is not set properly on localconnections. To be reviewed for 2.5.3

    March 10th, 2014 @ 05:28 PM

  10. Dave
    Dave

    Daniele,
    Don't know if this is related but when opening a connection I do the following.

    class Conn(psycopg2.extensions.connection):
        def __init__(self, dsnStr):
            try:
                super(Conn, self).__init__(dsnStr)
                print "Connection using < %s > is successful" % self.dsn
            except:
                self.close() # explicit close otherwise 'connection.closed' reads as open still
                print "Connection using < %s > failed" % self.dsn
        
        # bunch of db utility methods here
        
    conn = Conn('user=dpostgres password=fght98 host=localhost')
    if conn.closed == 0:
        # Do something with the db
    else:
        # Ask user to correct dsn parameters and try again
    

    If the connection attempt fails, the exception is caught.... but the closed attribute
    has already been set to 0 which should be 1 at this point. That's why I do a close().

    Maybe I'm doing it wrong and should not attempt to do anything with the class if
    the connection fails and just return False; but having access to the dsn attribute is a good thing.

    Cheers
    :)

    March 25th, 2014 @ 01:22 AM

  11. Daniele Varrazzo
    Daniele Varrazzo

    Yes, I'd say it's related. It's a sort of corner case but I'll try to iron out that too for next release.

    March 25th, 2014 @ 08:39 AM

  12. Dave
    Dave

    On the same subject,

    connection.dsn
    
    will normally return a hidden password, however if the connection class instantiation is partial due to a connection error as per my previous example,
    connection.dsn
    
    will now expose the password used in the attempted connection.

    March 26th, 2014 @ 10:40 PM

  13. Daniele Varrazzo
    Daniele Varrazzo

    Fixed the problems with dsn and closed in connection subclasses. Please open a different issue if you have something else to report :)

    April 3rd, 2014 @ 12:43 AM

  14. Daniele Varrazzo
  15. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from open to invalid

    Fixed the poll() and docs reworded. However poll() doesn't notice immediately either that the connection got closed: it takes another roundtrip. Example:

    connect and restart the server

    In [2]: import psycopg2    
    In [3]: cnn = psycopg2.connect('')
    
    In [4]: !sudo /etc/init.d/postgresql stop 9.2
     * Stopping PostgreSQL 9.2 database server   ...  [ OK ]
    

    the first poll succeeds, although a notice is sent that the server restarted

    In [5]: cnn.poll()
    Out[5]: 0
    
    In [6]: cnn.notices
    Out[6]: ['FATAL:  terminating connection due to administrator command\n']
    

    the libpq still thinks the connection is good

    In [7]: cnn.get_transaction_status()
    Out[7]: 0
    

    the following poll finds the connection broken

    In [8]: cnn.poll()
    ---------------------------------------------------------------------------
    OperationalError                          Traceback (most recent call last)
    /home/piro/dev/psycopg2/build/lib.2.7/<ipython-input-8-2d99764f7d0a> in <module>()
    ----> 1 cnn.poll()
    
    OperationalError: 
    
    In [9]: cnn.get_transaction_status()
    Out[9]: 4
    

    but at least now (psycopg 2.5.3 the connection is marked closed)

    In [10]: cnn.closed
    Out[10]: 2
    

    Closing this bug as I've cleaned up all the glitches that cropped out of it but closed will stay a client-side only operation. Cheers!

    April 5th, 2014 @ 02:41 PM