Interface error: decoding empty BYTEA

Hello,

the check should be correct: an encoded empty string correctly decoded should have input s="\x" and output str="", so the check str[0] == 'x' should fail and the error shouldn't be raised.

This check also proves an empty bytea only represented with \x is correctly handled:

In [1]: import psycopg2

In [2]: cnn = psycopg2.connect('')

In [3]: cur = cnn.cursor()

In [4]: cur.execute(r"select E'\\x'::bytea")

In [5]: str(cur.fetchone()[0])
Out[5]: ''

Are you sure the libpq psycopg is linked to is the correct version? Please take a look using:

ldd path/to/psycopg2/_psycopg.so | grep libpq

What is decode_fn? Can you provide a self-contained failing test case?

I think we really need to have our own hex format parser instead of relying on the libpq one anyway...

Yes, linked libpq is ok

ldd /usr/lib/python2.4/site-packages/psycopg2/_psycopg.so
libpq.so.5 => /usr/pgsql-9.0/lib/libpq.so.5 (0xb7824000)

Unfortunately I can't provide any complete test case - decode_fn is custom function written in C that was used before with PostgreSQL 8.4 and psycopg2 2.0.

However I found another test case when SELECT of BYTEA fails:

In [1]: import psycopg2

In [2]: conn = psycopg2.connect("")

In [3]: cur = conn.cursor()

In [4]: cur.execute("SELECT ''::BYTEA")

In [5]: cur.fetchall()
---------------------------------------------------------------------------
InterfaceError                            Traceback (most recent call last)

/root/<ipython console> 

InterfaceError: can't receive bytea data from server >= 9.0 with the current libpq client library: please update the libpq to at least 9.0 or set bytea_output to 'escape' in the server config or with a query

In [6]: cur.execute(r"SELECT E'\\x'::bytea")

In [7]: cur.fetchall()
---------------------------------------------------------------------------
InterfaceError                            Traceback (most recent call last)

/root/<ipython console> 

InterfaceError: can't receive bytea data from server >= 9.0 with the current libpq client library: please update the libpq to at least 9.0 or set bytea_output to 'escape' in the server config or with a query

After closing ipython and repeating those commands again, everything was OK.
My colleague looked at PQunescapeBytea function and found out that when unescaping empty data, it allocates 1 byte and returns it unchanged - so there actually could be some bad data from last allocation. Could this be the problem?

We're using PGRPMS on CentOS - so wen can run pg 8.4 and 9.0. But on this server there's only v9.0.
Like I said - once it raised those errors and for the second time, everything was ok. So we tried to apply this patch

diff -Naur psycopg2-2.4/psycopg/typecast_binary.c psycopg2-2.4-fix/psycopg/typecast_binary.c
--- psycopg2-2.4/psycopg/typecast_binary.c  2011-02-24 11:38:28.000000000 +0100
+++ psycopg2-2.4-fix/psycopg/typecast_binary.c  2011-03-16 16:44:25.701055330 +0100
@@ -168,7 +168,7 @@

/* Check the escaping was successful */
if (s[0] == '\\' && s[1] == 'x' /* input encoded in hex format */
- && str[0] == 'x' /* output resulted in an 'x' */
+ && len > 0 && str[0] == 'x' /* output resulted in an 'x' */
&& s[2] != '7' && s[3] != '8') /* input wasn't really an x (0x78) */
{
PyErr_SetString(InterfaceError,

and it seem ok, no errors - it skips escape check if PQunescapeBytea returns empty string (len == 0, but str is 1 byte long with uninitialized data).
Please consider this as a patch and if it could cause any problems. Thanks

Thank you for the patch, but I think a safer approach would be to ditch entirely the libpq unescape and deal with the hex format ourselves - it is an easy to parse format after all.

I should have done it for the 2.4 release: it is my only regret for an otherwise perfect release. Will fix it in 2.4.1: will try to write the patch this weekend.

By the way, your analysis was correct: the check str[0] is wrong when len == 0.

The libpq allocates at least one byte to avoid a non portable malloc(0). I was in the mindset of a null-terminated string, but str[0] was reading an uninitialized block of memory.

Your patch would have been to be applied, but now it's superseded by having our own parser. Thank you very much anyway.