This project is archived and is in readonly mode.

#20 ✓resolved
Psycopg website

Building psycopg2 for Python 2.7 on Windows XP

Reported by Psycopg website | November 26th, 2010 @ 06:43 AM

Submitted by: mirco

Install psycopg2 from binary package psycopg2-2.2.2.win32-py2.7-pg9.0.1-release.exe (Windows download from official site http://initd.org/psycopg/)
In some computer the psycopg2 works well with django test server but with Apache Server httpd fails to load some dll, perhaps because compiled against msvcrt.dll instead msvc90.dll with entries in error.log (for example “TemplateSyntaxError: Caught ImproperlyConfigured while rendering: 'django.db.backends.postgresql_psycopg2' isn't an available database backend.”)
The solution that I’ve found is to re-build psycopg2. These are the steps:
• Download and install MinGW. (Check for gcc.exe file in C:\MinGW\bin)
• Add C:\MinGW\bin to your PATH. (Right click My Computer | Advanced tab | Environment Variables | Select ‘Path’ from the ‘System variables’ list | Edit | Append ‘;C:\MinGW\bin’ to the end of the string in ‘Variable value’). Probably you need to reboot your computer.
• Download and execute the Postgre. SQL one-click installer.
• Download and extract psycopg2 (tar.gz archive with source codes)
• From within the extracted psycopg2 directory, execute the following command:
python setup.py build_ext --compiler=mingw32 build (Note: In command output check if there is msvcr90.dll instead of msvcrt.dll)
Your goodies will be in the local build directory (build\lib.win32-2.7\psycopg2). You can rename the installed psycopg2 directory C:\Python27\Lib\site-packages\psycopg2 with the local build one.
Someone suggests python setup.py build_ext –compiler=mingw32 install but I think it works if you haven’t already installed psycopg2.

Comments and changes to this ticket

  • Daniele Varrazzo

    Daniele Varrazzo November 26th, 2010 @ 11:38 AM

    • Tag changed from msvc90.dll rel-2.2.2, apache, mingw, msvcrt.dll to apache, mingw, msvc90.dll, msvcrt.dll, rel-2.2.2

    As a reference, here is a thread discussing about the issue on the modwsgi group.

    I've informed Jason about the issue: let's see if he knows how to come out the DLL hell.

  • Daniele Varrazzo

    Daniele Varrazzo December 23rd, 2010 @ 12:56 AM

    In ticket #31 it is reported this is not a 2.7 only problem.

    psycopg 2.0.14 works fine. 2.2.1 and 2.3.2 both have this problem.

    Running Windows, Python 2.6.5, Postgre 8.4.4, mod_wsgi 3.3, Django 1.2.3, Apache 2.2.17. I have a friend running my code with a similar setup and they have the same error.

  • bright_pan

    bright_pan January 9th, 2011 @ 07:03 AM

    i recompiled the package successful on windows sp3 using mingw32, but i found output file _psycopg.pyd only 127KB, and that file in the official package is 1MB, whats wrong with me?

    in apache+wsgi, the error (ImportError: DLL load failed) is always exist.
    but in dev server, it works fine.

  • JamesD

    JamesD January 11th, 2011 @ 11:16 PM

    I just wanted to comment that the instructions given above worked perfectly for myself.

    @Bright_pan. It sounds like you still need to run "build" or "install". I uninstalled the bad version and ran this to get it to work:

    python setup.py build_ext --compiler=mingw32 install

  • Jason Erickson

    Jason Erickson January 14th, 2011 @ 05:58 PM

    Note: This issue applies to Python 2.6 and Python 2.7 builds built with MSVC

    Quick Overview


    The problem is that the psycopg.pyd does not have a ‘link’ to the VC 2008 runtime in its manifest file when built with MSVC 2008 (the standard compiler for Python 2.6 and Python 2.7 on windows) and Python versions >= 2.6.4 and 2.7 (Note that I build against the latest Pythons, so with 2.6 I am using the most recent). Changes were made at these versions that purposely strip out the VC runtime reference when building C extensions. Attached is a patch that modifies psycopg’s setup.py to reinsert the reference into the build process after it has been stripped out, fixing the condition when python is embedded and the originating exe does not point to the VC 2008 runtimes (aka apache/mod_wsgi).
    Drawback for this fix is that _psycopg will break if ALL of the following conditions are met:
    • Python is used in an embedded situation (a C program linked to python)
    • Python/embedded app is NOT installed for all users (ie, Install just for me)
    • MSVC 2008 Runtime libraries are not installed in the system folder (C:\Windows\WinSxS)
    Note that if these all these conditions are met, the pywin32 extension would also have the same problem.
    Also, a solution to the above is to install the MSVC 2008 runtime ( http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9b2da53... )

    Long Overview


    The Windows builds for Python 2.6 and Python 2.7 use the MSVC 2008 compiler, which uses (forces?) Microsoft SxS (Side-by-Side) for the MS VC++ runtime library (MSVC 2005 also uses SxS for the runtime library). The advantage of SxS was to allow multiple versions of DLLs to be installed on a system, yet have the application use the DLL that it was designed for. MS stored this DLL linkage in a manifest embedded in the DLL/exe so that the OS would be able to load the DLLs at runtime. Basically, the 10,000ft overview of this idea is similar to UNIX’s shared library versioning, just more complicated ;)

    Like many things, in theory this sounds good, but in has cases in reality where it falls apart. One such area seems to be if the user does not have administrative rights to install shared libraries into the common folder, C:\Windows\WinSxS. SxS does allow for this situation by having the shared DLLs be local (called ‘private assemblies’ or ‘xcopy deployments’) to the executable, but gets muddled when we are talking about DLLs loading SxS DLLs (which is what C complied python extension do).

    Originally, Python’s distutils did not touch the automatically generated manifest file of extension libraries, but changes were made in Python versions >=2.6.4 and the 2.7 line that ended up stripping the VC runtime information out of the manifest for the python extension DLLs (see http://bugs.python.org/issue4120 ). Ironically, it was people who were embedding python into other application that wanted the VC runtimes stripped out, since their applications were being deployed in situations where users did not have permission to write to SxS the system directories. They were relying on the fact that if the originating EXE loaded the SxS DLL, it would get mapped into the extensions space automatically.

    So then why doesn’t the Apache->mod_wsgi->python->psycopg situation work? Apache is not built with a version of MSVC that uses SxS, so Apache has no manifest embedded in its executable. And having a SxS manifest on other DLLs in the chain (ie. which python26.dll does) is ignored unless it is in the executable or the originating DLL, so the only two places that we can point to the runtime DLL is either in apache or psycopg. And if neither point to the runtime, it results in the error that we see.

    The ways to resolve the situation is to embed a manifest into apache or embed it into psycopg (Note, once could try and force linking against the static version of the VC runtime library, but MS strongly discourages this practice for DLLs http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.90%29.aspx ).

    Since we have no control over embedding the manifest into apache (and why should they ‘link’ to a runtime they don’t use), that leaves the solution of embedding it into psycopg. Trying to find ‘precedence’ for other precompiled Python 2.7 C extensions for windows (2.6 versions might have been built with a version before 2.6.4), I found that pywin32, has the manifest in it. So I don’t feel that wrong reinserting the manifest back into it.

    What is the implication of this fix? Based upon why the change was made in the first place, psycopg breaks if ALL of the following conditions are met:
    • Python is used in an embedded situation (a C program linked to python)
    • Python/embedded app is NOT installed for all users (aka, Install just for me)
    • MSVC 2008 Runtime libraries are not installed in the system folder (C:\Windows\WinSxS)
    Note that this applies to any C extension that has the VC runtime in its manifest, which includes pywin32, and any C extensions built using Python <2.6.4.

  • jerrys

    jerrys February 27th, 2011 @ 05:59 PM

    I just used the roadmap in the initial posting, ending in
    python setup.py build_ext --compiler=mingw32 install

    . Python 2.6 Windows 7 Professional mod_wsgi-win32-apppy26-3.3.so Apache 2.2 All is well first time out of the box.
    Thanks!!

  • Daniele Varrazzo

    Daniele Varrazzo March 1st, 2011 @ 10:43 AM

    Psycopg 2.4 has been released: Jason has changed the way it is built to try to cope with the problem reported.

    We got no feedback about the 2.4 beta, so we have released it anyway. Please test 2.4 with your deploy and report failure/success, so we can close the bug or take different measures.

    Windows packages are available as usual from http://www.stickpeople.com/projects/python/win-psycopg/

    Thank you.

  • jerrys

    jerrys March 2nd, 2011 @ 02:39 PM

    After my positive comment above, I did find a somewhat anomalous situation where my hand built version failed. I tried this new build in a variety of settings, particularly hard in the one just referred to, and it works great, everywhere!

  • Daniele Varrazzo

    Daniele Varrazzo March 2nd, 2011 @ 03:19 PM

    • State changed from “new” to “resolved”

    Thank you very much jerrys. And thank you Jason: great job as usual.

  • Alec Munro

    Alec Munro May 13th, 2011 @ 06:29 PM

    This is still occurring, at least on 2.7.

    Both building my own using Visual C++ 2008 Express, and downloading the built version.

Create your profile

Help contribute to this project by taking a few moments to create your personal profile. Create your profile ยป

<b>WARNING:</b> the informations in this tracker are archived. Please submit new tickets or comments to <a href="https://github.com/psycopg/psycopg2/issues">the new tracker</a>.
<br/>
Psycopg is the most used PostgreSQL adapter for the Python programming language. At the core it fully implements the Python DB API 2.0 specifications. Several extensions allow access to many of the features offered by PostgreSQL.

Shared Ticket Bins

Attachments

Referenced by

Pages