Process Hangs

Reposting the code for formatting

  def _execute(self, query, params=None, fetch=False):
    self.logger.debug("starting _execute...")
    with psycopg2.connect(database = self.database,
                          user = self.user,
                          host = self.host,
                          password = self.password) as conn:
      with conn.cursor() as cur:
        cur.execute("SET TIME ZONE 'PDT8PST';") 
        if params:
          cur.execute(query,tuple(params))
        else:
          cur.execute(query)
        if fetch:
          return cur.fetchall()
    self.logger.debug("_execute complete")

[4420] pint_setup: init pint object at 0x7f31c51558b8, refcnt = 1
[4420] pint_setup: good pint object at 0x7f31c51558b8, refcnt = 1
[4420] microprotocol_getquoted: adapted to psycopg2._psycopg.Int
[4420] pint_dealloc: deleted pint object at 0x7f31c51558b8, refcnt = 0
[4420] microprotocols_adapt: trying to adapt int
[4420] pint_setup: init pint object at 0x7f31c51558b8, refcnt = 1
[4420] pint_setup: good pint object at 0x7f31c51558b8, refcnt = 1
[4420] microprotocol_getquoted: adapted to psycopg2._psycopg.Int
[4420] pint_dealloc: deleted pint object at 0x7f31c51558b8, refcnt = 0
[4420] curs_execute: pg connection at 0x294cf80 OK
[4420] pq_begin_locked: pgconn = 0x294cf80, autocommit = 0, status = 2
[4420] pq_begin_locked: transaction in progress
[4420] pq_execute: executing SYNC query: pgconn = 0x294cf80
[4420]     UPDATE etl_mongo_oplog SET time = 1405625377, ordinal = 576 WHERE name = 'tracker'
[4420] pq_execute: entering synchronous DBAPI compatibility mode
[4420] pq_fetch: pgstatus = PGRES_COMMAND_OK
[4420] pq_fetch: command returned OK (no tuples)
[4420] _read_rowcount: PQcmdTuples = 1
[4420] pq_fetch: fetching done; check for critical errors
[4420] psyco_curs_execute: res = 1, pgres = (nil)
[4420] psyco_curs_close: cursor at 0x7f31c6c656d8 closed
[4420] pq_commit: pgconn = 0x294cf80, autocommit = 0, status = 2
[4420] pq_execute_command_locked: pgconn = 0x294cf80, query = COMMIT
[4420] conn_close: PQfinish called
[4420] connection_dealloc: deleted connection object at 0x7f31c6987048, refcnt = 0
[4420] cursor_dealloc: deleted cursor object at 0x7f31c6c656d8, refcnt = 0
[4420] psyco_connect: dsn = '<%SENSITIVE DATA%>', async = 0
[4420] connection_setup: init connection object at 0x7f31c6987048, async 0, refcnt = 1
[4420] con_connect: connecting in SYNC mode

I am not sure how you can claim the server is causing this error. The process hangs infinitely. I have since implemented connection pooling which seems to help, however not sure why not having it would cause this bug. Even when I include a connect timeout, which means the driver should disconnect after a timeout, the process still hangs indefinitely. There have been other reports of this @112. Not sure what other evidence I can provide. I did notice in the debug output this line:

[4420] connection_dealloc: deleted connection object at 0x7f31c6987048, refcnt = 0

Then 3 lines later:

[4420] connection_setup: init connection object at 0x7f31c6987048, async 0, refcnt = 1

Am I misunderstanding or is the driver deallocating the connection object from memory then re-referencing it later?

Ticket #112 you reference is about queries, not about connections.

The driver is not referencing a deallocated object: it destroyed one and allocated a new one (when the conn name is re-bound in the loop) which happened to be created at the previous location, by chance or by optimization: the Python memory manager knows why.

As for being AWS or psycopg: try repeating your test connecting to a regular postgres server instead of AWS and see if you can repeat the problem. If you can and provide a test showing the bug I'll be happy to fix it. Note that the code you posted above is just a middle-man function that creates a connection and runs whatever query you pass it, not a complete test.

Thank you very much.

Daniele-
Why would the process still hang with a connect_timeout set? The process definitely hangs!

Thanks!

With AWS or with regular Postgres?

You may be bumping in some quirkiness of AWS pooling or throttling that the libpq is not prepared to handle. Getting a C stack trace of a stuck process could be helpful.

With aws. How do I get the c stack trace? I have a strace of the issue.

You can attach gcc to the stuck process, or kill it generating a core.

You can also try to make the process responsive again (killable at python level) switching to green mode as discussed in the last couple of messages in the ML (registering wait_select with set_wait_callback).

You can attach gcc to the stuck process

sorry, "gdb"