This project is archived and is in readonly mode.

#51 ✓invalid

TypeError: can't escape unicode to binary: writing unicode into bytea

Reported by tamagotchi | April 23rd, 2011 @ 01:51 AM


I have been using psycopg2 on PostgreSQL 8.3, 8.4 on Linux and Windows without any issues.
Now I started working on a Mac OSX machine and installed the latest PostgreSQL 9 + psycopgs2 2.4 (dt dec pq3 ext) + Python 2.7.1
And I noticed that my application (which is using SQLAlchemy) can't write into bytea fields anymore.
I'm getting:
TypeError: can't escape unicode to binary

I boiled it down to this simple code block:

b = psycopg2.Binary(u"XYZABCTEst")
c = psycopg2.connect("dbname=acttest")
cur = c.cursor()
cur.execute("INSERT INTO users (userid, preferences) VALUES ('userid', %(prefs)s) RETURNING", {'prefs': b})

If I use a regular string
b = psycopg2.Binary("XYZABCTEst")
everything works fine.

Binary is supposed to work with unicode objects, isn't it?



Comments and changes to this ticket

  • tamagotchi

    tamagotchi April 25th, 2011 @ 07:30 PM

    After poking around some more, I figure out that the above code doesn't work with other versions of Postgres and psycopg as well.
    The reason why my application broke was the upgrade from cherrypy 3.1.x to 3.2 where the data in the request is now passed as unicode objects rather than strings.
    The question for me remains still, is it reasonable to expect that byteas/Binaries should work with unicode?
    If not, why not? (in which case this bug report would be invalid)



  • Daniele Varrazzo

    Daniele Varrazzo May 3rd, 2011 @ 11:02 PM

    • Tag set to binary, unicode
    • State changed from “new” to “invalid”

    No, it's not reasonable for unicode to be interpreted as a binary string. This is the entire point for which there are separate unicode and string object (or str and bytes in Py3).

    Internal unicode representation could be 2 or 4 bytes per codepoint. It can be converted to binary in different ways according to different encoding.

    If you want to pass unicode to a binary you will have to convert it to byte string specifying your own encoding. For me it's a bug in cherrypy 3.2.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="">the new tracker</a>.
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

People watching this ticket