This project is archived and is in readonly mode.
Python freezed binary cannot use psycopg.so
Reported by Christian Bachmaier | March 25th, 2014 @ 09:13 AM
I use psycopg2 2.4.5-1buidl5 under Ubuntu 14.04 LTS x86_64. After pachting Python 3.4.0 final as described under http://bugs.python.org/issue16047 to work again since Python 3.2 under Python 3.3 and 3.4 I can sucessfully freeze my Python cgi project to a binary with the in Python included Tool freeze.py command.
However, a freezed script hello.py containing only the one line
import psycopg2
does not operate, since it does not find the psycopg.so library. It delivers after execution:
Traceback (most recent call last):
File "hello.py", line 18, in <module>
import psycopg2
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2203, in _find_and_load_unlocked
module = _SpecMethods(spec)._load_unlocked()
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1200, in _load_unlocked
self._exec(module)
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1129, in _exec
self.spec.loader.exec_module(module)
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 1336, in exec_module
exec(code, module.__dict__)
File "/usr/lib/python3/dist-packages/psycopg2/__init__.py", line 67, in <module>
from psycopg2._psycopg import BINARY, NUMBER, STRING, DATETIME, ROWID
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2214, in _find_and_load
return _find_and_load_unlocked(name, import_)
File "/usr/lib/python3.4/importlib/_bootstrap.py", line 2201, in _find_and_load_unlocked
raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'psycopg2._psycopg'
In Python 3.2 it helped to create a link
_psycopg.so ->
/usr/lib/python3/dist-packages/psycopg2/_psycopg.cpython-32mu.so in
a subdir psycopg2 of the current directory of the execution of the
freezed binary.
The same issue is on Python3.4r2 under Debian Wheezy/Sid.
Comments and changes to this ticket
-
Daniele Varrazzo March 25th, 2014 @ 09:29 AM
It doesn't seem a psycopg bug to me: it seems a freeze problem. We just provide a
setup.py
for the Python toolchain to build. Or is there anything we have to change insetup.py
to support new features? -
Daniele Varrazzo March 25th, 2014 @ 09:34 AM
- State changed from new to invalid
Taken a look at the above bug report. Closing as not a psycopg bug.
-
Christian Bachmaier May 14th, 2014 @ 06:40 AM
Adding
from pkgutil import extend_path __path__ = extend_path(__path__, __name__)
to the top of psycopg2/__init__.py resolves the problem and the frozen binary works as expected. I had to learn this in a painful and time wasting way without any help from here or the Python bug databse http://bugs.python.org/issue16047 :(
I am not quite sure if this is due to a flaw of Python/freeze as running in interpreted mode does not need it. At least the behaviours in interpreted and frozen mode should not be different. Or it is a bug of psycopg2. However, the above works in both modes. Other libraries like janitor or gi do already contain these lines.
-
Daniele Varrazzo May 16th, 2014 @ 09:29 PM
- State changed from invalid to open
from
extend_path
docs:"This is useful if one wants to distribute different parts of a single logical package as multiple directories"
This doesn't sound what psycopg wants to do. Could there be a way to adjust psycopg imports to make it compatible with freeze? This comment from M.-A. L. seems suggesting it is possible.
If you can fix psycopg by adjusting its imports but remaining in the realm of the canonical ways to import internal modules I'll be happy to apply the patch. If you want to see the patch applied in psycopg 2.5.x it must be compatible with everything between Python 2.5 and 3.4.
I'll leave the bug open for you but I won't be working on this problem: the ball is yours.
-
Christian Bachmaier May 17th, 2014 @ 05:18 AM
As far as I can tell, this is exactly as needed. Using debug outputs like
print(__path__) from pkgutil import extend_path __path__ = extend_path(__path__, __name__) print(__path__)
show that running in interpreted mode does not change the search path path of the package:
['/usr/lib/python3/dist-packages/psycopg2'] ['/usr/lib/python3/dist-packages/psycopg2']
So there is not any change for that at all. Everything stays as before.
In compiled mode path seems to be empty. See the link in my previous postings. The guys there told me that this is by design. Don't know why, this may be a bug in freeze if you are asking me. Hower running in compiled mode, the above code only adds the psycopg2 directory which contains the native so-library and which is there in interpreted mode. Then the native library can be found at runtime by a frozen binary.
[] ['/usr/lib/python3/dist-packages/psycopg2']
Other (similar built up) libraries use this code. Look at /usr/lib/python3/dist-packages/gi or .../janitor.
Fazit: The 2 lines fix the problem enitrely. There is nothing to work on, but to test with older python version and to put into the official code. Unfortunately, I can only test with 3.4 at the moment. Theoretically this should work also with older Python versions. But maybe the maintainer of psycopg have some testing installations?
Remember to use the fixed version of freeze (e.g. hg clone http://hg.python.org/cpython) as prior versions in Python 3.3 and 3.4 were broken.
Then it is easy:$ /$HOME/cpython-xxx/Tools/freeze/freeze.py hello.py $ make $ ./hello
For your convenience I have also attached an archive with an operating version of freeze. So at least for 3.3 and 3.4 you need not to download it.
hello.py contains some code which uses psycopg2. Consiting only of a lineimport psycopg2
already suffices.
-
Daniele Varrazzo May 19th, 2014 @ 01:59 PM
Christian, extend_path is not designed to fix freeze but for a different use case that doesn't apply to psycopg, so I'm very reluctant to add it. In the past I've been pyinstaller maintainer and I remember having no problem with psycopg. For what I (don't) know your solution may end up breaking other freeze solutions that currently work.
Please talk with freeze authors and have the issue fixed there. If we are doing our internal imports wrong I'm ready to change every single one of them, but I'm not happy to add a call to a function that I don't know what it does and the docs say it's not for our use; the fact two packages I don't know anything about do the same is not enough for me to accept it as a quality solution.
-
Daniele Varrazzo August 25th, 2014 @ 10:43 PM
Christian, I've taken a look at your problem.
Freeze cannot freeze psycopg just because it cannot work with C extension. This should be enough to rule out what you want to do. I've taken a look at why adding the unrelated hack of
extend_path()
work: it works just because it finds an externally available_psycopg.so
, however we don't want to distribute psycopg "as multiple directories": that's not for us.Note that using absolute or relative imports doesn't change anything. However we will likely move to relative imports as they are the right thing and we don't target Python 2.4 anymore.
In order to solve your problem you can add an import hack into your frozen application:
sys.path.insert(0, '/path/to/psycopg2') import _psycopg sys.modules['psycopg2._psycopg'] = _psycopg sys.path.pop(0) # this will now work import psycopg2
so you can ship
_psycopg.so
in a separate directory together with your frozen application and make it available beforepsycopg2
itself.The above hack only works with a psycopg version patched with this changeset; without it importing
_psycopg
would fail because it would try to importpsycopg2.tz
.This is the best I can do for your issue. Please acknowledge that you are trying to use a tool designed to freeze pure Python modules: if it doesn't work for psycopg2 it isn't our fault and we cannot do much. Please test the workaround above: if it works I'll try and ship the patch in the next 2.5.4 release (it still needs some testing).
-
Christian Bachmaier August 26th, 2014 @ 03:31 PM
Daniele, thanks.
Your code sys.path Manipulation works with the version from
cd /scratch
git clone https://github.com/dvarazzo/psycopg -b freeze
cd psycopg
python3 setup.py buildand using
import sys
sys.path.insert(0, '/scratch/psycopg2/build/lib.linux-x86_64-3.4/psycopg2')
...Please let me know from which release version on the changes will be contained.
I assume that in init.pyfrom pkgutil import extend_path
path = extend_path(__path__, name)does something similar, as many other libraries (see above) use that. The advantage is that the library does the trick then and not the user programm.
However, there should be a clean way. Either this is a bug in freeze or in psycopg2. I tend to say it is in freeze as the interpreted and compiled beavior should be identical. However, the freeze guys are not very amused about my opionion (http://bugs.python.org/issue16047). Maybe you could open a ticket?
Please acknowledge that you are trying to use a tool designed to freeze pure Python modules
The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).
-
Daniele Varrazzo August 26th, 2014 @ 04:05 PM
- State changed from open to resolved
Hi Christian,
I consider the refactoring I've made to _psycopg.so generally useful and I've tested it more yesterday. If there is no surprise from the test grid (with python versions I've not tested yet) I'll release soon, in the next 2.5.4.
Have another test: I suspect if
_psycopg.so
is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.The docu (I have found) says nothing about not supporting .so-files, nor the maintainer of freeze do, if I interpret them right (see above link).
Freeze doesn't support non-pure modules at all: read its docstring:
The script should not use modules provided only as shared libraries; if it does, the resulting binary is not self-contained.
so, that's what happens. Freeze doesn't freeze the .so, but the resulting executable can still use them if it finds them, i.e. with the regular python importing machinery. It's up to you to make the .so available to the exe. I've fixed psycopg2 so that the .so can be imported outside the package and that's about what I can do to help whatever program needs pythonpath hacking. But you are really asking too much to freeze: other programs are better suited to create executables out of non-pure modules (e.g. pyinstaller).
Closing the bug. Have a nice day.
-
Christian Bachmaier August 26th, 2014 @ 04:15 PM
Have another test: I suspect if _psycopg.so is in the same directory of the executable you don't even need the sys.path hack, but I may be wrong.
Unfortunately, this does not work, even not if _psycopg.so is in a local subdirectory psycopg. The latter worked with python 2.7 and its freeze.
Create your profile
Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป
<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.
People watching this ticket
Attachments
Referenced by
- 214 Python freeze does not like psycopg2 I have added a comment with an solution to the (premature...
- 214 Python freeze does not like psycopg2 I have added a comment with an solution to the (premature...