Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

Need a way to remove 'bad' connections from a connection pool

#62

In the following code:

c = pool.getconn()
try:
    f(c)
finally:
    pool.putconn()

If the database is restarted while f is executing then an OperationalError will be raised. The connection will then be put back into the pool in a bad state; when the code is run subsequently, f will receive the bad connection.

I'd like to be able to catch the OperationalError and tell the pool to remove the bad connection, but rummaging around inside used, rused and _pool seems impolite.

Reported by Sam Morris · June 30th, 2011 @ 10:45 AM

State: resolved
Milestone: none
Assigned to: nobody

Activity

  1. Daniele Varrazzo
    Daniele Varrazzo
    • Tag set to pool
    • State changed from new to open

    I see other problems in the pool too: if a connection is returned to the pool while a transaction is open, the connection is left idle in transaction.

    A robust way to handle all the cases I think would be to check the transaction status, so cases such as lost connection or connection in transaction/error can be handled properly.

    June 30th, 2011 @ 02:05 PM

  2. Daniele Varrazzo
    Daniele Varrazzo

    Looking at the pool code, there is a way to discard the connection: use putconn(cnn, close=True), but it is not documented: I will fix the documentation.

    However I think the pool should be more careful on the connection it receives: I will fix it to detect broken/intrans/error connections and deal with them.

    June 30th, 2011 @ 03:23 PM

  3. Sam Morris
    Sam Morris

    Ah, thanks for that information.

    Does this look like the right thing to do in order to get a connection from the pool? Sorry if this sounds like a support request, but it will be a while until updated versions of psycopg2 filter down to end-users, so some example code might be helpful:

    conn = None
    while conn is None:
        c = pool.getconn()
        try:
            c.cursor().execute('SELECT 0')
        except:
            pool.putconn(c,close=True)
        else:
            c.rollback()
            conn = c
    try:
        f(conn)
    except:
        conn.rollback()
        raise
    finally:
        pool.putconn(conn)
    

    June 30th, 2011 @ 05:07 PM

  4. Daniele Varrazzo
    Daniele Varrazzo

    Yes, what you propose is a robust way to get a connection from the pool guarding from a server disconnection. I don't want to add a probe query into the pool code: the code will have a guard on putconn() discarding a broken connection but the f() could still receive a connection we don't know if broken. If your f() can't cope with it, you'd better keep your probing pattern even with future psycopg releases.

    July 1st, 2011 @ 10:52 AM

  5. Daniele Varrazzo
    Daniele Varrazzo
    • State changed from open to resolved

    The pool now is able to watch for the connection returned, caring to rolling back an eventual open connection or to dispose broken ones.

    August 9th, 2011 @ 10:39 AM