This project is archived and is in readonly mode.

#201 ✓resolved
Christian Bachmaier

Python freezed binary cannot use psycopg.so

Reported by Christian Bachmaier | March 25th, 2014 @ 09:13 AM

I use psycopg2 2.4.5-1buidl5 under Ubuntu 14.04 LTS x86_64. After pachting Python 3.4.0 final as described under http://bugs.python.org/issue16047 to work again since Python 3.2 under Python 3.3 and 3.4 I can sucessfully freeze my Python cgi project to a binary with the in Python included Tool freeze.py command.

However, a freezed script hello.py containing only the one line

import psycopg2

does not operate, since it does not find the psycopg.so library. It delivers after execution:

Traceback (most recent call last):
  File "hello.py", line 18, in <module>
    import psycopg2
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2203, in _find_and_load_unlocked
    module = _SpecMethods(spec)._load_unlocked()
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1200, in _load_unlocked
    self._exec(module)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1129, in _exec
    self.spec.loader.exec_module(module)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1336, in exec_module
    exec(code, module.__dict__)
  File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 67, in <module>
    from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
    return _find_and_load_unlocked(name, import_)
  File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'psycopg2._psycopg'

In Python 3.2 it helped to create a link
_psycopg.so -> /usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so in a subdir psycopg2 of the current directory of the execution of the freezed binary.

The same issue is on Python3.4r2 under Debian Wheezy/Sid.

Comments and changes to this ticket

  • Daniele Varrazzo

    Daniele Varrazzo March 25th, 2014 @ 09:29 AM

    It doesn't seem a psycopg bug to me: it seems a freeze problem. We just provide a setup.py for the Python toolchain to build. Or is there anything we have to change in setup.py to support new features?

  • Daniele Varrazzo

    Daniele Varrazzo March 25th, 2014 @ 09:34 AM

    • State changed from “new” to “invalid”

    Taken a look at the above bug report. Closing as not a psycopg bug.

  • Christian Bachmaier

    Christian Bachmaier May 14th, 2014 @ 06:40 AM

    Adding

    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)
    

    to the top of psycopg2/__init__.py resolves the problem and the frozen binary works as expected. I had to learn this in a painful and time wasting way without any help from here or the Python bug databse http://bugs.python.org/issue16047 :(

    I am not quite sure if this is due to a flaw of Python/freeze as running in interpreted mode does not need it. At least the behaviours in interpreted and frozen mode should not be different. Or it is a bug of psycopg2. However, the above works in both modes. Other libraries like janitor or gi do already contain these lines.

  • Daniele Varrazzo

    Daniele Varrazzo May 16th, 2014 @ 09:29 PM

    • State changed from “invalid” to “open”

    from extend_path docs:

    "This is useful if one wants to distribute different parts of a single logical package as multiple directories"

    This doesn't sound what psycopg wants to do. Could there be a way to adjust psycopg imports to make it compatible with freeze? This comment from M.-A. L. seems suggesting it is possible.

    If you can fix psycopg by adjusting its imports but remaining in the realm of the canonical ways to import internal modules I'll be happy to apply the patch. If you want to see the patch applied in psycopg 2.5.x it must be compatible with everything between Python 2.5 and 3.4.

    I'll leave the bug open for you but I won't be working on this problem: the ball is yours.

  • Christian Bachmaier

    Christian Bachmaier May 17th, 2014 @ 05:18 AM

    As far as I can tell, this is exactly as needed. Using debug outputs like

    print(__path__)
    
    from pkgutil import extend_path
    __path__ = extend_path(__path__, __name__)
    
    print(__path__)
    

    show that running in interpreted mode does not change the search path path of the package:

    ['/usr/lib/python3/dist-packages/psycopg2']
    ['/usr/lib/python3/dist-packages/psycopg2']
    

    So there is not any change for that at all. Everything stays as before.

    In compiled mode path seems to be empty. See the link in my previous postings. The guys there told me that this is by design. Don't know why, this may be a bug in freeze if you are asking me. Hower running in compiled mode, the above code only adds the psycopg2 directory which contains the native so-library and which is there in interpreted mode. Then the native library can be found at runtime by a frozen binary.

    []
    ['/usr/lib/python3/dist-packages/psycopg2']
    

    Other (similar built up) libraries use this code. Look at /usr/lib/python3/dist-packages/gi or .../janitor.

    Fazit: The 2 lines fix the problem enitrely. There is nothing to work on, but to test with older python version and to put into the official code. Unfortunately, I can only test with 3.4 at the moment. Theoretically this should work also with older Python versions. But maybe the maintainer of psycopg have some testing installations?

    Remember to use the fixed version of freeze (e.g. hg clone http://hg.python.org/cpython) as prior versions in Python 3.3 and 3.4 were broken.
    Then it is easy:

    $ /$HOME/cpython-xxx/Tools/freeze/freeze.py hello.py
    $ make
    $ ./hello
    

    For your convenience I have also attached an archive with an operating version of freeze. So at least for 3.3 and 3.4 you need not to download it.
    hello.py contains some code which uses psycopg2. Consiting only of a line

    import psycopg2
    

    already suffices.

  • Daniele Varrazzo

    Daniele Varrazzo May 19th, 2014 @ 01:59 PM

    Christian, extend_path is not designed to fix freeze but for a different use case that doesn't apply to psycopg, so I'm very reluctant to add it. In the past I've been pyinstaller maintainer and I remember having no problem with psycopg. For what I (don't) know your solution may end up breaking other freeze solutions that currently work.

    Please talk with freeze authors and have the issue fixed there. If we are doing our internal imports wrong I'm ready to change every single one of them, but I'm not happy to add a call to a function that I don't know what it does and the docs say it's not for our use; the fact two packages I don't know anything about do the same is not enough for me to accept it as a quality solution.

  • Daniele Varrazzo

    Daniele Varrazzo August 25th, 2014 @ 10:43 PM

    Christian, I've taken a look at your problem.

    Freeze cannot freeze psycopg just because it cannot work with C extension. This should be enough to rule out what you want to do. I've taken a look at why adding the unrelated hack of extend_path() work: it works just because it finds an externally available _psycopg.so, however we don't want to distribute psycopg "as multiple directories": that's not for us.

    Note that using absolute or relative imports doesn't change anything. However we will likely move to relative imports as they are the right thing and we don't target Python 2.4 anymore.

    In order to solve your problem you can add an import hack into your frozen application:

    sys.path.insert(0, '/path/to/psycopg2')
    import _psycopg
    sys.modules['psycopg2._psycopg'] = _psycopg
    sys.path.pop(0)
    
    # this will now work
    import psycopg2
    

    so you can ship _psycopg.so in a separate directory together with your frozen application and make it available before psycopg2 itself.

    The above hack only works with a psycopg version patched with this changeset; without it importing _psycopg would fail because it would try to import psycopg2.tz.

    This is the best I can do for your issue. Please acknowledge that you are trying to use a tool designed to freeze pure Python modules: if it doesn't work for psycopg2 it isn't our fault and we cannot do much. Please test the workaround above: if it works I'll try and ship the patch in the next 2.5.4 release (it still needs some testing).

  • Christian Bachmaier

    Christian Bachmaier August 26th, 2014 @ 03:31 PM

    Daniele, thanks.

    Your code sys.path Manipulation works with the version from

    cd /scratch
    git clone https://github.com/dvarazzo/psycopg -b freeze
    cd psycopg
    python3 setup.py build

    and using

    import sys
    sys.path.insert(0, '/scratch/psycopg2/build/lib.linux-x86_64-3.4/psycopg2')
    ...

    Please let me know from which release version on the changes will be contained.
    I assume that in init.py

    from pkgutil import extend_path
    path = extend_path(__path__, name)

    does something similar, as many other libraries (see above) use that. The advantage is that the library does the trick then and not the user programm.

    However, there should be a clean way. Either this is a bug in freeze or in psycopg2. I tend to say it is in freeze as the interpreted and compiled beavior should be identical. However, the freeze guys are not very amused about my opionion (http://bugs.python.org/issue16047). Maybe you could open a ticket?

    Please acknowledge that you are trying to use a tool designed to freeze pure Python modules

    The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).

  • Daniele Varrazzo

    Daniele Varrazzo August 26th, 2014 @ 04:05 PM

    • State changed from “open” to “resolved”

    Hi Christian,

    I consider the refactoring I've made to _psycopg.so generally useful and I've tested it more yesterday. If there is no surprise from the test grid (with python versions I've not tested yet) I'll release soon, in the next 2.5.4.

    Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.

    The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).

    Freeze doesn't support non-pure modules at all: read its docstring:

    The script should not use modules provided only as shared libraries;
    if it does, the resulting binary is not self-contained.
    

    so, that's what happens. Freeze doesn't freeze the .so, but the resulting executable can still use them if it finds them, i.e. with the regular python importing machinery. It's up to you to make the .so available to the exe. I've fixed psycopg2 so that the .so can be imported outside the package and that's about what I can do to help whatever program needs pythonpath hacking. But you are really asking too much to freeze: other programs are better suited to create executables out of non-pure modules (e.g. pyinstaller).

    Closing the bug. Have a nice day.

  • Christian Bachmaier

    Christian Bachmaier August 26th, 2014 @ 04:15 PM

    Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.

    Unfortunately, this does not work, even not if _psycopg.so is in a local subdirectory psycopg. The latter worked with python 2.7 and its freeze.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile »

<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

Attachments

Referenced by

Pages