Building Scipy/Numpy on Windows with Optimized Numerical Libraries

This involves compiling several libraries (ATLAS, LAPACK, FFTW and UMFPACK) and then building numpy and scipy from SVN source. But as with most things Windows, this turns out to be a slightly tricky affair.

The following has been tested on Windows Vista Enterprise 32bit only, but should theoretically work on other Windows platforms. It also used Python 2.5.

Ideally, a big chunk of this page should move to the scipy/numpy site. And also ideally should become a single script. But it’s also good to know exactly how you got there.

Prerequisites

• You need Windows Vista enterprise/ultimate with SUA enabled and installed or Windows (others, including other Vista variants) with Cygwin installed. You cannot install the SUA package on a non enterprise or ultimate Vista edition.

• MinGW (installer) with gcc 3.4.5 (choose the candidate option when installing) and the msys environment installed. You will need to download the following packages for msys:

  • bzip2-1.0.3-MSYS-1.0.11-snapshot.tar.bz2

  • coreutils-5.97-MSYS-1.0.11-snapshot.tar.bz2

  • diffutils-2.8.7-MSYS-1.0.11-snapshot.tar.bz2

  • gawk-3.1.5-MSYS-1.0.11-snapshot.tar.bz2

  • make-3.81-MSYS-1.0.11-snapshot.tar.bz2

  • msysCORE-1.0.11-2007.01.19-1.tar.bz2

  • binutils-2.17.50-20070129-1.tar.gz

Just unpack all the package contents in a single directory and copy them over to the MinGW installation directory. You may want to add the following to the system path:

set PATH=[PATH TO]\MinGW;[PATH TO]\MinGW\libexec\gcc\mingw32\3.4.5;%PATH%

• Numerical Libraries

  • ATLAS latest developer version

  • LAPACK lapack 3.1 scroll down to Available software

  • FFTW fftw-3.1.2

  • UMFPACK download UMFPACK, UFConfig, AMD

Installation

• Create a directory called BUILDS, BUILDS/lib, BUILDS/include

• Unpack all the numerical library files in BUILDS

• Create subversion check out directories for scipy and numpy in BUILDS

• Start SUA c-shell or cygwin shell

• Start msys.bat:

  PATH=/mingw/libexec/gcc/mingw32/3.4.5:$PATH; export PATH
  

• Change directory to location of BUILDS. (/dev/fs/driveletter/… in SUA, /cygdrive/driveletter/… in cygwin, /driveletter/… in msys)

Compiling ATLAS

• This is done in the SUA/Cygwin shell. In Cygwin you probably want to follow the instructions at Installing Scipy on Windows

• cd ATLAS; mkdir build; cd build

• Run ../configure (This will probably fail but will leave you with xconfig)

• Run ./xconfig –help (to see all options)

• Run ../configure -O 8 -A 16 -m 3189 -b 32 (replacing the values with your machine configuration)

• Edit Make.inc to provide correct L2SIZE

• Run make (leave your computer and go do something else for about an hour)

Compiling LAPACK

• This is done in the msys shell

• cd lapack_XX

• Copy make.inc.example to make.inc

• Edit the following lines in make.inc:

  PLAT = _NT
  OPTS = -funroll-all-loops -O3 -malign-double -msse2
  BLASLIB      = -L/driveletter/[PATH TO]/BUILDS/ATLAS/build/lib -lf77blas -latlas
  

• Run make lib

Combining LAPACK and ATLAS

• Stay in the msys shell after compiling LAPACK

• Go to the ATLAS/build/lib directory

• Execute the following commands:

  mkdir tmp; cd tmp
  cp ../liblapack.a ../liblapack_ATLAS.a
  ar -x ../liblapack.a
  cp [PATH TO]/lapack_NT.a ../liblapack.a
  ar -r ../liblapack.a *.o
  rm *.o
  ar -x ../liblapack.a xerbla.o
  ar -r ../libf77blas.a xerbla.o
  

• Copy liblapack.a, libf77blas.a, libcblas.a, libatlas.a to BUILDS/lib

• Copy the ATLAS/include to BUILDS/include/ATLAS

Compiling UMFPACK

• Stay in msys shell

• Goto UFconfig

• Edit UFConfig/UFconfig.mk:

  BLAS   = -L/driveletter/[PATH TO]/BUILDS/lib -llapack -lf77blas -lcblas -latlas -lg2c
  LAPACK = -L/driveletter/[PATH TO]/BUILDS/lib -llapack -lf77blas -lcblas -latlas -lg2c
  XERBLA =
  

• Run the following commands:

  cd ..\AMD
  make
  cd ..\UMFPACK
  make
  

• Copy libamd.a (from AMD), libumfpack.a (from UMFPACK) to BUILDS/lib

• Copy UMFPACK/include to BUILDS/include/UMFPACK

• Copy UFconfig/ufconfig.h to BUILDS/include

• Copy AMD/include/amd.h to BUILDS/include

Compiling fftw

Note

The latest versions of scipy do not link to FFTW, so this step is no longer useful for scipy

• Stay in msys shell

• Goto fftw_XX

• mkdir build; cd build

• Run the following command:

  ../configure --prefix=/c/DOWNLOADS/BUILDS/ --enable-sse2 --disable-dependency-tracking --enable-threads --with-our-malloc16 --with-windows-f77-mangling --with-combined-threads
  

• Run make OR make -j 4 if you have multiple processors (it’ll make things go faster. This build on msys in vista takes a while)

• Copy .libs/libfftw3.a to BUILDS/lib

• Copy fftw_XX/api/fftw3.h to BUILDS/include

Compiling numpy/scipy

Note

As above, note that the FFTW linking here is no longer useful for the scipy install

• Open a Windows cmd window and make sure you can execute python.

• Make a copy of each of the libs in BUILDS/lib and rename them from libname.a to name.lib

• Rename lapack.lib to flapack.lib

• rename site.cfg.example to site.cfg

• Edit site.cfg in the numpy directory. Replace the blas_opt and lapack_opt section with:

  [atlas]
  libraries = f77blas, cblas, atlas, g2c
  library_dirs = driveletter:\[PATH TO]\MinGW\lib;driveletter:\[PATH TO]\BUILDS\lib
  include_dirs = driveletter:\[PATH TO]\BUILDS\include\ATLAS
  
  [lapack]
  libraries = flapack, f77blas, cblas, atlas
  library_dirs = driveletter:\[PATH TO]\MinGW\lib;driveletter:\[PATH TO]\BUILDS\lib
  
  [amd]
  library_dirs = driveletter:\[PATH TO]\MinGW\lib;driveletter:\[PATH TO]\BUILDS\lib
  include_dirs = driveletter:\[PATH TO]\BUILDS\include
  libraries = amd
  
  [umfpack]
  library_dirs = driveletter:\[PATH TO]\MinGW\lib;driveletter:\[PATH TO]\BUILDS\lib
  include_dirs = driveletter:\[PATH TO]\BUILDS\include\UMFPACK
  libraries = umfpack
  
  [fftw3]
  library_dirs = driveletter:\[PATH TO]\MinGW\lib;driveletter:\[PATH TO]\BUILDS\lib
  include_dirs = driveletter:\[PATH TO]\BUILDS\include
  libraries = fftw3
  

• Edit numpy/distutils/fcompiler/gnu.py. Find the line that says opt.append(‘gcc’) and comment it # opt.append(‘gcc’). This is probably a Vista SUA thing and perhaps won’t be required when using Cygwin to compile ATLAS.

• Copy site.cfg to ../scipy/site.cfg

• Compile numpy:

  cd numpy
  python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst
  

• Install numpy from the numpy/dist folder

• Compile scipy:

  cd scipy
  python setup.py config --compiler=mingw32 build --compiler=mingw32 bdist_wininst
  

• Install scipy from the scipy/dist folder

• Test installations. In python run:

  import numpy
  import scipy
  numpy.test()
  scipy.test()
  numpy.show_config()
  scipy.show_config()