Building Python 2.7.10 with Visual Studio 2010 or 2015

7th revision, August 7, 2015.
A document history can be found at the very end of this page

Downloads

For the impatient:

Note: these builds use updated versions of OpenSSL (1.0.2d) and SQLite (3.8.11), plus tons of other new goodies.

Why bother?

Python on Windows is built with Visual Studio 2008 Professional. If you have an application built with Visual Studio 2010 (or any other Visual Studio version), and you have Python embedded in it, you have two problems:

1. It is a very bad idea to mix different Visual C++ runtimes. See http://siomsystems.com/mixing-visual-studio-versions/ for an explanation. This means if you use the official libraries to link in Python, you are in for some nasty surprises.
2. The official distribution does not contain debug libraries. For some reason I don’t fully understand the python libraries have different names for debug and release builds (rather than just be located in a different directory, as I would normally approach this). This means that you cannot easily build a working debug build of your project.

So, you need to rebuild Python with the Visual Studio version you are using. In the following post I am going to explain what you need to do.

Build Instructions

Get the sources from http://python.org/download/. At the time of this writing, Python 2.7.10 is the official release, so I will usePython-2.7.10.tgz in the following.

Extract the files. You’ll get a folder Python-2.7.10 with several subfolders for sources, headers and so on.

Go to the PCBuild folder and open readme.txt in an editor capable of showing unix-style newlines, i.e. ANYTHING BUT NOTEPAD. I personally am a hardcore fan of scite, which is available at http://www.scintilla.org/SciTE.html.

Read readme.txt. No, I mean seriously: read it. In particular, let’s recap the section named Python-controlled subprojects that wrap external projects and follow slowly.

We’ll start at the top, with SQLite:

_sqlite3

Wraps SQLite 3.6.21 which is currently built by sqlite3.vcproj (see below).

Well, 3.6.21 is old – it dates back from 2009. So since we’re going to rebuild everything anyway, why not upgrade sqlite as we go along? (You might not care so much, but me being a heavy SQLite user do). If you open the project sqlite3.vcproj in an editor (SciTE), you’ll see that the sources are supposed to be in a directory with the variable $(sqlite3Dir), which in turn is kept inpyproject.vsprops. Once you open that in an editor, you’ll notice it defines all the dependent directories as well. Take a look at this:

    ...

	<UserMacro

		Name="sqlite3Dir"

		Value="$(externalsDir)\sqlite-3.6.21"

	/>

	<UserMacro

		Name="bz2Dir"

		Value="$(externalsDir)\bzip2-1.0.6"

	/>

	<UserMacro

		Name="opensslDir"

		Value="$(externalsDir)\openssl-0.9.8y"

	/>

    ...

So first we’re going to update sqlite. No risk, no fun! Head over to sqlite.org and download the latest amalgamation sources; at the time of the writing these were 3.8.10. Extract the archive, you should have 4 files in there. Now, where to move the files: notice that..\.. is the externals directory, which is on the same include level as Python-2.7.10. Add a folder sqlite-3.8.10 and copy the amalgamation files there.

Next, you’ll also need bzip2-1.0.6. Download the source tarball at http://www.bzip.org/downloads.html and install it in the correct directory.

Next, you’ll also need db-4.7.25.0. Do a svn export http://svn.python.org/projects/external/db-4.7.25.0 so that the folder is right next to the Python-2.7.10 one.

Next, openssl. There is a version 1.0.1e in subversion, but 1.0.1m is current, and we want to have a current SSL, don't we? At this point, I should probably point out that I have written together specific instructions on building version 1.0.1g.

The TCL/TK problem child

Building TCL/TK support for Python is insert curse-word here, and in the first version of this document I refused to build it. But I've since managed to build it, and I include the instructions here even though I still believe it to be pretty pointless. But anyway, I digress.

• Go to your Python source directory.
• Run Tools\buildbot\external.bat

Ok, this will fail, for two reasons:

1. tcl-8.5.2.1\generic\tclPosixStr.c uses some POSIX cases more than once (at least under the defines available on Windows). I've modified the source, it is part of the download. For reference, the problem looksl ike this:

   tclPosixStr.c
   
   ..\generic\tclPosixStr.c(347) : error C2196: case value '123' already used
   
   ..\generic\tclPosixStr.c(407) : error C2196: case value '121' already used
   
   ..\generic\tclPosixStr.c(795) : error C2196: case value '123' already used
   
   ..\generic\tclPosixStr.c(855) : error C2196: case value '121' already used
   

2. The makefiles use the obsolete syntax -debug:Full for the linker; this must be replaced by -debug. To find out which files are affected, do this:
   

   C:\Python27\Source>grep -r "debug:full" *
   
   Python-2.7.10/output.txt:        link -nologo -machine:IX86 -ltcg -debug:full -debugtype:cv -dll -base:@..\win\coffbase.txt,tcl -out:.\Debug_VC9\tcl85g.dll  kern
   
   el32.lib user32.lib ws2_32.lib @C:\Users\Gerson\AppData\Local\Temp\nmFA32.tmp
   
   Python-2.7.10/output.txt:LINK : fatal error LNK1117: syntax error in option 'debug:full'
   
   Python-2.7.10/output.txt:        link -nologo -machine:IX86 -ltcg -debug:full -debugtype:cv -dll -base:@"..\..\tcl-8.5.2.1\win\coffbase.txt",tk -out:".\Debug_VC9
   
   \tk85g.dll" kernel32.lib user32.lib gdi32.lib  "..\..\tcl-8.5.2.1\win\Debug_VC9\tclstub85.lib" @C:\Users\Gerson\AppData\Local\Temp\nm5B23.tmp
   
   Python-2.7.10/output.txt:LINK : fatal error LNK1117: syntax error in option 'debug:full'
   
   tcl-8.5.2.1/win/configure:      LDFLAGS_DEBUG="-debug:full"
   
   tcl-8.5.2.1/win/makefile.vc:ldebug      = -debug:full -debugtype:cv
   
   tcl-8.5.2.1/win/makefile.vc:@LDFLAGS_DEBUG@      -nologo -machine:$(MACHINE) -debug:full -debugtype:cv
   
   tcl-8.5.2.1/win/tcl.m4: LDFLAGS_DEBUG="-debug:full"
   
   tk-8.5.2.0/win/configure:       LDFLAGS_DEBUG="-debug:full"
   
   tk-8.5.2.0/win/makefile.vc:ldebug       = -debug:full -debugtype:cv
   
   tk-8.5.2.0/win/tcl.m4:  LDFLAGS_DEBUG="-debug:full"
   

Once you've done this, the external.bat runs through ... and creates debug binaries. Great, just what we needed, right

No. You probably want to make a release build as well, and for that you need to patch external.bat once more:

1. Run notepad Tools\buildbot\external.bat and replace DEBUG=1 with an empty string.
2. Still in notepad, replace tcl85g.dll with tcl85.dll, and tk85g.dll with tk85.dll
3. Close the file, continuing to hate TCL/TK.

Doing the actual build (for 32-bit Windows)

OK, time to open Pcbuild.sln. I am doing this with Visual Studio 2010, the IDE asks me whether I want to upgrade the projects: yes, I do

Open the batch build options dialog and select all 32-bit debug and release binaries. Batch build ahead!

You’ll see a ton of warnings like this:

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(990,5): warning MSB8012: TargetPath(C:\Python27\Source\Python-2.7.10\PCbuild\\pythoncore.dll) does not match the Linker's OutputFile property value (C:\Python27\Source\Python-2.7.10\PCbuild\python27.dll). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile).

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Microsoft.CppBuild.targets(992,5): warning MSB8012: TargetName(pythoncore) does not match the Linker's OutputFile property value (python27). This may cause your project to build incorrectly. To correct this, please make sure that $(OutDir), $(TargetName) and $(TargetExt) property values match the value specified in %(Link.OutputFile).

Apparently, this is a well-known issue and you basically have to either ignore this warning, or edit all projects (mostly in the debug build of the project) as per the *.com recommendation; like this:

• In the project settings, for the Debug build, if you have a Linker/Output File like this: $(OutDir)$(ProjectName)_d.pyd (example: unicodedata), you need to change
  • General/Target Name from $(ProjectName)_d to $(ProjectName)_d.
  • General/Target Extension from .dll to .pyd.
• In the project settings, for the Release build, if you have a Linker/Output File like this: $(OutDir)$(ProjectName).pyd (example: unicodedata), you need to change
  • General/Target Extension from .dll to .pyd.

You get the idea...

Note: Starting with Revision 4 of this document, the source download includes project files that have been patched to correct this issue. In other words, if you download the source on this page and rebuild for yourself, you will not get these warnings any longer.

After the build completes, for me I get a single project not done: bsddb. This is actually a problem in the project conversion, because you see ton's of erros like this:

19>c1 : fatal error C1083: Cannot open source file: 'repmgr\repmgr_windows.c': No such file or directory

19>  rijndael-alg-fst.c

which means the project expects db-4.7.25.0 to be a subdirectory of PCBuild rather than in the location where it is actually is. Do this:

• Close Visual Studio (or unload the project).
• Open _bsddb.vcxproj in an editor and change references from

  <ClCompile Include="btree\bt_upgrade.c" />
  

  to

  <ClCompile Include="..\..\db-4.7.25.0\btree\bt_upgrade.c" />
  

• Open _bsddb.vcxproj.filters in an editor and do basically the same

You'll also have problems building tkinter.pyd. After all the shit you've had to go through above, this seems a bit unfair, but well: that is life.

Error	110	error LNK1181: cannot open input file '..\..\tcltk\lib\tcl85.lib ..\..\tcltk\lib\tk85.lib'	C:\Python27\Source\Python-2.7.10\PCbuild\LINK	_tkinter

AAAAAAAAAAAAAAAAAAA. Did I mention that TCL/TK is shit? Add hardcoded lines to the lib files you just built, and it'll find the files.

Understanding the mysteries surrounding the Python Debug builds

Why Batch Building both Debug and Release targets fails

The Visual Studio solution contains both Debug and Release targets. These have separate names following the standard conventions: so for example there is a _ctypes.pyd in Release Build, and a _ctypes_d.pyd in Debug Build.

This holds true for allmost all projects in the solution: except for four:

1. make_buildinfo.exe. This is the one that is causing the problem, and the problem is that this project is built first to create a configuration file used in subsequent builds. And if you do batch-build, then this file is created only once, so its settings are overwritting the settings for the other projects. Solution: do a separate debug and release build, and you'll be fine
2. w9xpopen.exe. Let me quote the documentation: " Serves as an intermediate stub Win32 console application to avoid a hanging pipe when redirecting 16-bit console based programs (including MS-DOS console based programs and batch files) on Window 95 and Windows 98.". OK, this is for Windows 95, a version that will be 20 years old soon. If you are still use the Win9x series, you don't deserve an upgraded Python build :) so I choose not to include it in the binary distribution.
3. kill_python.exe. This is a helper program, quote, " for killing lingering python[_d].exe processes before building, thus attempting to avoid build failures due to files being locked.". OK, this is basically the same in both debug and release builds.
4. make_versioninfo.exe. This file generates the resource file version number, and OK this is the same in debug and release builds, so no problems here.

Building an updated install

• First, you install the standard windows distribution for 2.7.
• This installer puts python27.dll in windows\system32 (or in windows\syswow64 if you’re running on 64-bit windows). You need to remove it there, because you want to create a distribution that is fully movable.
• Update the binaries in C:\python27 and C:\python27\dlls. Note that for every .exe, .dll and .pyd, there should be two forms inPCBuild: a blabla.dll and a blabla_d.dll and so on. Copy both.

Doing the actual build for 64-bit Windows

• Most of the above comments apply to the 64-bit build as well.
• Note that make_buildinfo and make_versioninfo are not 64-build compatible. You build the 32-bit version, and then you reuse the 32-bit version in the 64-bit batch build...
• Binaries are built in C:\Python27\Source\Python-2.7.10\PCbuild\amd64, rather than C:\Python27\Source\Python-2.7.10\PCbuild

Modules

The download includes the following modules - sorted alphabetically - that were more or less easy to build. Some of them are included because I use them in my professional life, some are included because they are fashionable, and some are included because people asked for them ;)

Verifying you are using the updated SSL versions:

Python 2.7.10 (default, Jul  6 2015, 11:26:50) [MSC v.1600 32 bit (Intel)] on win32

Type "help", "copyright", "credits" or "license" for more information.

>>> import OpenSSL

>>> import OpenSSL.SSL

>>> OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION)

'OpenSSL 1.0.1p 12 Jun 2015'

>>> ....

>>> import ssl

>>> ssl.OPENSSL_VERSION

'OpenSSL 1.0.1p 12 Jun 2015'

Back to the build overview

Document history

• August 7, 2015: document revision 8
  • Provided builds using Visual Studio 2015
  • Updated OpenSSL to 1.0.2d
• July 12, 2015: document revision 8
  • Updated OpenSSL to 1.0.1p
• July 6, 2015: document revision 7
  • Updated Python to 2.7.10
  • Updated OpenSSL to 1.0.1o
  • Updated a ton of other modules
• May 10, 2015: document revision 6
  • Updated Python to 2.7.9
  • Updated OpenSSL to 1.0.1m
  • Updated PILLOW to 2.8
• April 12, 2014: document revision 5
  • Updated OpenSSL to 1.0.1g for fix of the heartbleed issue
• March 8, 2014: document revision 4
  • Added a detailed section on Debug / Release builds, and how to get those.
  • Added x64 build for all components (and x64 build notes for all)
  • Added tons of debug versions (for example, for win32api)
  • Added many new libraries (for example, SQLAlchemy, psycopg2, beautifulsoup, and others.)
  • Generally, resolved .eggs as standard folders - a matter of personal preference
• Feb 24, 2014: document revision 3
  • Added wxPython 3.0.0
• Feb 23, 2014:: document revision 2
  • Enabled TCL/TK build
  • OpenSSL updated to 1.0.1f
  • SQLite updated to 3.8.3.1
• Jan 5, 2014: First release