Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

This project is archived and is in readonly mode.

TypeError: can't escape unicode to binary: writing unicode into bytea

#51

Hi,

I have been using psycopg2 on PostgreSQL 8.3, 8.4 on Linux and Windows without any issues.
Now I started working on a Mac OSX machine and installed the latest PostgreSQL 9 + psycopgs2 2.4 (dt dec pq3 ext) + Python 2.7.1
And I noticed that my application (which is using SQLAlchemy) can't write into bytea fields anymore.
I'm getting:
TypeError: can't escape unicode to binary

I boiled it down to this simple code block:

b = psycopg2.Binary(u"XYZABCTEst")
c = psycopg2.connect("dbname=acttest")
cur = c.cursor()
cur.execute("INSERT INTO users (userid, preferences) VALUES ('userid', %(prefs)s) RETURNING users.id", {'prefs': b})

If I use a regular string
b = psycopg2.Binary("XYZABCTEst")
everything works fine.

Binary is supposed to work with unicode objects, isn't it?

Thanks,

-T

Reported by tamagotchi · April 23rd, 2011 @ 01:51 AM

State: invalid
Milestone: none
Assigned to: nobody

Activity

  1. tamagotchi
    tamagotchi

    After poking around some more, I figure out that the above code doesn't work with other versions of Postgres and psycopg as well.
    The reason why my application broke was the upgrade from cherrypy 3.1.x to 3.2 where the data in the request is now passed as unicode objects rather than strings.
    The question for me remains still, is it reasonable to expect that byteas/Binaries should work with unicode?
    If not, why not? (in which case this bug report would be invalid)

    Thanks,

    -T

    April 25th, 2011 @ 07:30 PM

  2. Daniele Varrazzo
    Daniele Varrazzo
    • Tag set to binary, unicode
    • State changed from new to invalid

    No, it's not reasonable for unicode to be interpreted as a binary string. This is the entire point for which there are separate unicode and string object (or str and bytes in Py3).

    Internal unicode representation could be 2 or 4 bytes per codepoint. It can be converted to binary in different ways according to different encoding.

    If you want to pass unicode to a binary you will have to convert it to byte string specifying your own encoding. For me it's a bug in cherrypy 3.2.

    May 3rd, 2011 @ 11:02 PM