Psycopg psycopg

Comments and changes to this ticket

• 

  Daniele Varrazzo March 25th, 2014 @ 09:29 AM

  It doesn't seem a psycopg bug to me: it seems a freeze problem. We just provide a setup.py for the Python toolchain to build. Or is there anything we have to change in setup.py to support new features?

• 

  Daniele Varrazzo March 25th, 2014 @ 09:34 AM

  • State changed from “new” to “invalid”

  Taken a look at the above bug report. Closing as not a psycopg bug.

• You flagged this item as spam.
  

  Christian Bachmaier May 14th, 2014 @ 06:40 AM

  Adding

  from pkgutil import extend_path
  __path__ = extend_path(__path__, __name__)
  

  to the top of psycopg2/__init__.py resolves the problem and the frozen binary works as expected. I had to learn this in a painful and time wasting way without any help from here or the Python bug databse http://bugs.python.org/issue16047 :(

  I am not quite sure if this is due to a flaw of Python/freeze as running in interpreted mode does not need it. At least the behaviours in interpreted and frozen mode should not be different. Or it is a bug of psycopg2. However, the above works in both modes. Other libraries like janitor or gi do already contain these lines.

• 

  Daniele Varrazzo May 16th, 2014 @ 09:29 PM

  • State changed from “invalid” to “open”

  from extend_path docs:

  "This is useful if one wants to distribute different parts of a single logical package as multiple directories"

  This doesn't sound what psycopg wants to do. Could there be a way to adjust psycopg imports to make it compatible with freeze? This comment from M.-A. L. seems suggesting it is possible.

  If you can fix psycopg by adjusting its imports but remaining in the realm of the canonical ways to import internal modules I'll be happy to apply the patch. If you want to see the patch applied in psycopg 2.5.x it must be compatible with everything between Python 2.5 and 3.4.

  I'll leave the bug open for you but I won't be working on this problem: the ball is yours.

• You flagged this item as spam.
  

  Christian Bachmaier May 17th, 2014 @ 05:18 AM

  As far as I can tell, this is exactly as needed. Using debug outputs like

  print(__path__)
  
  from pkgutil import extend_path
  __path__ = extend_path(__path__, __name__)
  
  print(__path__)
  

  show that running in interpreted mode does not change the search path path of the package:

  ['/usr/lib/python3/dist-packages/psycopg2']
  ['/usr/lib/python3/dist-packages/psycopg2']
  

  So there is not any change for that at all. Everything stays as before.

  In compiled mode path seems to be empty. See the link in my previous postings. The guys there told me that this is by design. Don't know why, this may be a bug in freeze if you are asking me. Hower running in compiled mode, the above code only adds the psycopg2 directory which contains the native so-library and which is there in interpreted mode. Then the native library can be found at runtime by a frozen binary.

  []
  ['/usr/lib/python3/dist-packages/psycopg2']
  

  Other (similar built up) libraries use this code. Look at /usr/lib/python3/dist-packages/gi or .../janitor.

  Fazit: The 2 lines fix the problem enitrely. There is nothing to work on, but to test with older python version and to put into the official code. Unfortunately, I can only test with 3.4 at the moment. Theoretically this should work also with older Python versions. But maybe the maintainer of psycopg have some testing installations?

  Remember to use the fixed version of freeze (e.g. hg clone http://hg.python.org/cpython) as prior versions in Python 3.3 and 3.4 were broken.
  Then it is easy:

  $ /$HOME/cpython-xxx/Tools/freeze/freeze.py hello.py
  $ make
  $ ./hello
  

  For your convenience I have also attached an archive with an operating version of freeze. So at least for 3.3 and 3.4 you need not to download it.
  hello.py contains some code which uses psycopg2. Consiting only of a line

  import psycopg2
  

  already suffices.

  • freeze.tar.gz 20.6 KB
• 

  Daniele Varrazzo May 19th, 2014 @ 01:59 PM

  Christian, extend_path is not designed to fix freeze but for a different use case that doesn't apply to psycopg, so I'm very reluctant to add it. In the past I've been pyinstaller maintainer and I remember having no problem with psycopg. For what I (don't) know your solution may end up breaking other freeze solutions that currently work.

  Please talk with freeze authors and have the issue fixed there. If we are doing our internal imports wrong I'm ready to change every single one of them, but I'm not happy to add a call to a function that I don't know what it does and the docs say it's not for our use; the fact two packages I don't know anything about do the same is not enough for me to accept it as a quality solution.

• 

  Daniele Varrazzo August 25th, 2014 @ 10:43 PM

  Christian, I've taken a look at your problem.

  Freeze cannot freeze psycopg just because it cannot work with C extension. This should be enough to rule out what you want to do. I've taken a look at why adding the unrelated hack of extend_path() work: it works just because it finds an externally available _psycopg.so, however we don't want to distribute psycopg "as multiple directories": that's not for us.

  Note that using absolute or relative imports doesn't change anything. However we will likely move to relative imports as they are the right thing and we don't target Python 2.4 anymore.

  In order to solve your problem you can add an import hack into your frozen application:

  sys.path.insert(0, '/path/to/psycopg2')
  import _psycopg
  sys.modules['psycopg2._psycopg'] = _psycopg
  sys.path.pop(0)
  
  # this will now work
  import psycopg2
  

  so you can ship _psycopg.so in a separate directory together with your frozen application and make it available before psycopg2 itself.

  The above hack only works with a psycopg version patched with this changeset; without it importing _psycopg would fail because it would try to import psycopg2.tz.

  This is the best I can do for your issue. Please acknowledge that you are trying to use a tool designed to freeze pure Python modules: if it doesn't work for psycopg2 it isn't our fault and we cannot do much. Please test the workaround above: if it works I'll try and ship the patch in the next 2.5.4 release (it still needs some testing).

• You flagged this item as spam.
  

  Christian Bachmaier August 26th, 2014 @ 03:31 PM

  Daniele, thanks.

  Your code sys.path Manipulation works with the version from

  cd /scratch
  git clone https://github.com/dvarazzo/psycopg -b freeze
  cd psycopg
  python3 setup.py build

  and using

  import sys
  sys.path.insert(0, '/scratch/psycopg2/build/lib.linux-x86_64-3.4/psycopg2')
  ...

  Please let me know from which release version on the changes will be contained.
  I assume that in init.py

  from pkgutil import extend_path
  path = extend_path(__path__, name)

  does something similar, as many other libraries (see above) use that. The advantage is that the library does the trick then and not the user programm.

  However, there should be a clean way. Either this is a bug in freeze or in psycopg2. I tend to say it is in freeze as the interpreted and compiled beavior should be identical. However, the freeze guys are not very amused about my opionion (http://bugs.python.org/issue16047). Maybe you could open a ticket?

  Please acknowledge that you are trying to use a tool designed to freeze pure Python modules

  The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).

• 

  Daniele Varrazzo August 26th, 2014 @ 04:05 PM

  • State changed from “open” to “resolved”

  Hi Christian,

  I consider the refactoring I've made to _psycopg.so generally useful and I've tested it more yesterday. If there is no surprise from the test grid (with python versions I've not tested yet) I'll release soon, in the next 2.5.4.

  Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.

  The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).

  Freeze doesn't support non-pure modules at all: read its docstring:

  The script should not use modules provided only as shared libraries;
  if it does, the resulting binary is not self-contained.
  

  so, that's what happens. Freeze doesn't freeze the .so, but the resulting executable can still use them if it finds them, i.e. with the regular python importing machinery. It's up to you to make the .so available to the exe. I've fixed psycopg2 so that the .so can be imported outside the package and that's about what I can do to help whatever program needs pythonpath hacking. But you are really asking too much to freeze: other programs are better suited to create executables out of non-pure modules (e.g. pyinstaller).

  Closing the bug. Have a nice day.

• You flagged this item as spam.
  

  Christian Bachmaier August 26th, 2014 @ 04:15 PM

  Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.

  Unfortunately, this does not work, even not if _psycopg.so is in a local subdirectory psycopg. The latter worked with python 2.7 and its freeze.