Psycopg psycopg

Comments and changes to this ticket

• 

  Daniele Varrazzo November 26th, 2010 @ 11:38 AM

  • Tag changed from “msvc90.dll rel-2.2.2, apache, mingw, msvcrt.dll” to “apache, mingw, msvc90.dll, msvcrt.dll, rel-2.2.2”

  As a reference, here is a thread discussing about the issue on the modwsgi group.

  I've informed Jason about the issue: let's see if he knows how to come out the DLL hell.

• 

  Daniele Varrazzo December 23rd, 2010 @ 12:56 AM

  In ticket #31 it is reported this is not a 2.7 only problem.

  psycopg 2.0.14 works fine. 2.2.1 and 2.3.2 both have this problem.

  Running Windows, Python 2.6.5, Postgre 8.4.4, mod_wsgi 3.3, Django 1.2.3, Apache 2.2.17. I have a friend running my code with a similar setup and they have the same error.

• You flagged this item as spam.
  

  bright_pan January 9th, 2011 @ 07:03 AM

  i recompiled the package successful on windows sp3 using mingw32, but i found output file _psycopg.pyd only 127KB, and that file in the official package is 1MB, whats wrong with me?

  in apache+wsgi, the error (ImportError: DLL load failed) is always exist.
  but in dev server, it works fine.

• You flagged this item as spam.
  

  JamesD January 11th, 2011 @ 11:16 PM

  I just wanted to comment that the instructions given above worked perfectly for myself.

  @Bright_pan. It sounds like you still need to run "build" or "install". I uninstalled the bad version and ran this to get it to work:

  python setup.py build_ext --compiler=mingw32 install

• You flagged this item as spam.
  

  Jason Erickson January 14th, 2011 @ 05:58 PM

  Note: This issue applies to Python 2.6 and Python 2.7 builds built with MSVC
  

  Quick Overview

  ---

  The problem is that the psycopg.pyd does not have a ‘link’ to the VC 2008 runtime in its manifest file when built with MSVC 2008 (the standard compiler for Python 2.6 and Python 2.7 on windows) and Python versions >= 2.6.4 and 2.7 (Note that I build against the latest Pythons, so with 2.6 I am using the most recent). Changes were made at these versions that purposely strip out the VC runtime reference when building C extensions. Attached is a patch that modifies psycopg’s setup.py to reinsert the reference into the build process after it has been stripped out, fixing the condition when python is embedded and the originating exe does not point to the VC 2008 runtimes (aka apache/mod_wsgi).
  Drawback for this fix is that _psycopg will break if ALL of the following conditions are met:
  • Python is used in an embedded situation (a C program linked to python)
  • Python/embedded app is NOT installed for all users (ie, Install just for me)
  • MSVC 2008 Runtime libraries are not installed in the system folder (C:\Windows\WinSxS)
  Note that if these all these conditions are met, the pywin32 extension would also have the same problem.
  Also, a solution to the above is to install the MSVC 2008 runtime ( http://www.microsoft.com/downloads/en/details.aspx?FamilyID=9b2da53... )

  Long Overview

  ---

  The Windows builds for Python 2.6 and Python 2.7 use the MSVC 2008 compiler, which uses (forces?) Microsoft SxS (Side-by-Side) for the MS VC++ runtime library (MSVC 2005 also uses SxS for the runtime library). The advantage of SxS was to allow multiple versions of DLLs to be installed on a system, yet have the application use the DLL that it was designed for. MS stored this DLL linkage in a manifest embedded in the DLL/exe so that the OS would be able to load the DLLs at runtime. Basically, the 10,000ft overview of this idea is similar to UNIX’s shared library versioning, just more complicated ;)

  Like many things, in theory this sounds good, but in has cases in reality where it falls apart. One such area seems to be if the user does not have administrative rights to install shared libraries into the common folder, C:\Windows\WinSxS. SxS does allow for this situation by having the shared DLLs be local (called ‘private assemblies’ or ‘xcopy deployments’) to the executable, but gets muddled when we are talking about DLLs loading SxS DLLs (which is what C complied python extension do).

  Originally, Python’s distutils did not touch the automatically generated manifest file of extension libraries, but changes were made in Python versions >=2.6.4 and the 2.7 line that ended up stripping the VC runtime information out of the manifest for the python extension DLLs (see http://bugs.python.org/issue4120 ). Ironically, it was people who were embedding python into other application that wanted the VC runtimes stripped out, since their applications were being deployed in situations where users did not have permission to write to SxS the system directories. They were relying on the fact that if the originating EXE loaded the SxS DLL, it would get mapped into the extensions space automatically.

  So then why doesn’t the Apache->mod_wsgi->python->psycopg situation work? Apache is not built with a version of MSVC that uses SxS, so Apache has no manifest embedded in its executable. And having a SxS manifest on other DLLs in the chain (ie. which python26.dll does) is ignored unless it is in the executable or the originating DLL, so the only two places that we can point to the runtime DLL is either in apache or psycopg. And if neither point to the runtime, it results in the error that we see.

  The ways to resolve the situation is to embed a manifest into apache or embed it into psycopg (Note, once could try and force linking against the static version of the VC runtime library, but MS strongly discourages this practice for DLLs http://msdn.microsoft.com/en-us/library/abx4dbyh%28v=VS.90%29.aspx ).

  Since we have no control over embedding the manifest into apache (and why should they ‘link’ to a runtime they don’t use), that leaves the solution of embedding it into psycopg. Trying to find ‘precedence’ for other precompiled Python 2.7 C extensions for windows (2.6 versions might have been built with a version before 2.6.4), I found that pywin32, has the manifest in it. So I don’t feel that wrong reinserting the manifest back into it.

  What is the implication of this fix? Based upon why the change was made in the first place, psycopg breaks if ALL of the following conditions are met:
  • Python is used in an embedded situation (a C program linked to python)
  • Python/embedded app is NOT installed for all users (aka, Install just for me)
  • MSVC 2008 Runtime libraries are not installed in the system folder (C:\Windows\WinSxS)
  Note that this applies to any C extension that has the VC runtime in its manifest, which includes pywin32, and any C extensions built using Python <2.6.4.

  • 0001-change-win32-build-to-reinsert-vc-library-manifest.patch 3.1 KB

• You flagged this item as spam.
  

  jerrys February 27th, 2011 @ 05:59 PM

  I just used the roadmap in the initial posting, ending in
  python setup.py build_ext --compiler=mingw32 install

  . Python 2.6 Windows 7 Professional mod_wsgi-win32-apppy26-3.3.so Apache 2.2 All is well first time out of the box.
  Thanks!!

• 

  Daniele Varrazzo March 1st, 2011 @ 10:43 AM

  Psycopg 2.4 has been released: Jason has changed the way it is built to try to cope with the problem reported.

  We got no feedback about the 2.4 beta, so we have released it anyway. Please test 2.4 with your deploy and report failure/success, so we can close the bug or take different measures.

  Windows packages are available as usual from http://www.stickpeople.com/projects/python/win-psycopg/

  Thank you.

• You flagged this item as spam.
  

  jerrys March 2nd, 2011 @ 02:39 PM

  After my positive comment above, I did find a somewhat anomalous situation where my hand built version failed. I tried this new build in a variety of settings, particularly hard in the one just referred to, and it works great, everywhere!

• 

  Daniele Varrazzo March 2nd, 2011 @ 03:19 PM

  • State changed from “new” to “resolved”

  Thank you very much jerrys. And thank you Jason: great job as usual.

• You flagged this item as spam.
  

  Alec Munro May 13th, 2011 @ 06:29 PM

  This is still occurring, at least on 2.7.

  Both building my own using Visual C++ 2008 Express, and downloading the built version.