Accessing HDF Data from Python on Mac OS X

HDF, Python, and Mac OS X: can this three-way marriage work? Yes! This document tells how.

The Python programming language offers an agile and highly productive environment for developing a variety of applications, including numeric and scientific applications. Scientific analysis often requires reading and writing data in the Hierarchical Data Format (HDF). Accessing HDF files is possible from Python by compiling and installing add-on libraries.

Installing HDF4 and HDF5 libraries for Python on Mac OS X requires a fairly involved procedure. This document describes the steps you can take to enable Python-based HDF libraries. It's divided into three sections:

Installing Common Components. Follow these steps regardless of whether you want HDF4 or HDF5.
Installing HDF4 Components. Follow these steps if you wish to access HDF4 files using the "pyhdf" package.
Installing HDF5 Components. Follow these steps if you wish to access HDF5 files using the "h5py" package.

I tested these procedures on an Intel-based Apple iMac running Mac OS X 10.5.5 "Leopard". Your own experience may vary. The procedures described here assume you have administrative access to your computer. If not, pass this document onto your system administrator.

Installing Common Components

Both HDF4 and HDF5 depend on a number of base-packages. Follow these instructions to install the common components required by both HDF4 and HDF5.

Python 2.6

Mac OS X 10.5 ships with Python 2.5 as the system Python framework. However, I prefer not to mess with the system-supplied frameworks as much as possible, so I recommend installing Python 2.6. Follows these steps:

Download http://www.python.org/ftp/python/2.6/python-2.6-macosx.dmg.
Open the disk image if it doesn't open automatically and launch the installer package if it doesn't open automatically.
Follow the installer's prompts as presented and complete the installation.

You now have a Python 2.6 framework installed in /Library/Frameworks. Don't worry, Python 2.5 is still available if you need it in /System/Library/Frameworks.

Imaging Libraries

HDF4 and HDF5 make use of various image file formats, including JPEG and PNG. You can easily add libraries that support these formats as follows:

Download this installer.
Open the disk image if it doesn't open automatically and launch the installer package if it doesn't open automatically.
Follow the installer's prompts as presented and complete the installation.

SZIP

SZIP is a lossless compression library specifically for scientific data. Both HDF4 and HDF5 make use of it. Follow these instructions in order to install SZIP:

Make a temporary directory in which to work:
```
mkdir $HOME/Documents/src
```
Change that directory to the current working directory:
```
cd $HOME/Documents/src
```

Download and extract SZIP:

curl ftp://ftp.hdfgroup.org/lib-external/szip/2.1/src/szip-2.1.tar.gz | tar xzf -

Build, test, and install SZIP using the following commands:

cd szip-2.1
./configure --prefix=/Library/Frameworks/Python.framework/Versions/Current --enable-shared --enable-production
make
make check
sudo make install

When prompted for your password, enter it.

Numpy

Numpy is a numeric and scientific computing addon to Python. Both of the Python extensions that use HDF4 and HDF5 require Numpy to be present as well. Follow these instructions to install Numpy:

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download and extract Numpy:

curl http://superb-east.dl.sourceforge.net/sourceforge/numpy/numpy-1.2.1.tar.gz | tar xzf -

Install Numpy:

cd numpy-1.2.1
sudo /Library/Frameworks/Python.framework/Versions/Current/bin/python setup.py install

Setuptools

Setuptools is the de facto package management system for Python. Although not specifically required by the processes described in this document, no Python installation should be without it. Setuptools provides the easy_install command that makes installing many Python extensions quick and painless. Follow these steps:

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download the ez_setup.py program:

curl -O http://peak.telecommunity.com/dist/ez_setup.py

Execute it:

sudo /Library/Frameworks/Python.framework/Versions/Current/bin/python ez_setup.py

Installing HDF4 for Python

After completing the installation of the common components described above, you're ready to install the HDF4 library and the "pyhdf" Python package that makes use of it.

Begin by installing HDF4 using the following steps:

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download and extract the HDF4 source code:

curl ftp://ftp.hdfgroup.org/HDF/HDF_Current/src/HDF4.2r4.tar.gz | tar xzf -

Make the source directory the current working directory:
```
cd HDF4.2r4
```

Configure the source using the following (long) command:

./configure \
    --prefix=/Library/Frameworks/Python.framework/Versions/Current \
    --disable-fortran \
    --enable-production \
    --with-szlib=/Library/Frameworks/Python.framework/Versions/Current \
    CPPFLAGS=-I/Library/Frameworks/Python.framework/Versions/Current/include \
    LDFLAGS=-L/Library/Frameworks/Python.framework/Versions/Current/lib

Compile, test, and install HDF5 using these commands:
```
make
make check
sudo make install
```

Now, install "pyhdf":

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download and extract the pyhdf source code:

curl http://voxel.dl.sourceforge.net/sourceforge/pysclint/pyhdf-0.8.3.tar.gz | tar xzf -

Make the source directory the current working directory:
```
cd pyhdf-0.8.3
```

Install the software with the following (long) command:

sudo env \
    INCLUDE_DIRS=/Library/Frameworks/Python.framework/Versions/Current/include \
    LIBRARY_DIRS=/Library/Frameworks/Python.framework/Versions/Current/lib \
    /Library/Frameworks/Python.framework/Versions/Current/bin/python \
    setup.py \
    install

You can test the installation by running this command:

python2.6 -c 'import pyhdf'

If you receive no error message, then the installation was successful. If you see an ImportError or other error message, then something went wrong. (Do let me know!) Regardless, you can go ahead and throw away your temporary work area, $HOME/Documents/src.

Installing HDF5 for Python

After completing the installation of the common components described above in "Installing Common Components", you're ready to install the HDF5 library and the "h5py" Python package that makes use of it.

Begin by installing HDF5 using the following steps:

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download and extract the HDF5 source code:

curl ftp://ftp.hdfgroup.org/HDF5/current/src/hdf5-1.8.2.tar.gz | tar xzf -

Make the source directory the current working directory:
```
cd hdf5-1.8.2
```

Configure the software with the following (long) command:

./configure \
    --prefix=/Library/Frameworks/Python.framework/Versions/Current \
    --enable-shared \
    --enable-production \
    --enable-threadsafe \
    --with-szlib=/Library/Frameworks/Python.framework/Versions/Current \
    CPPFLAGS=-I/Library/Frameworks/Python.framework/Versions/Current/include \
    LDFLAGS=-L/Library/Frameworks/Python.framework/Versions/Current/lib

Compile, test, and install HDF4 using these commands:
```
make
make check
sudo make install
```

Now install "h5py":

Change again to your temporary work directory:
```
cd $HOME/Documents/src
```

Download and extract the h5py source code:

curl http://h5py.googlecode.com/files/h5py-1.0.0-1.8.tar.gz | tar xzf -

Make the source directory the current working directory:
```
cd h5py-1.0.0
```

Build the software using the following (long) command:

/Library/Frameworks/Python.framework/Versions/Current/bin/python \
    setup.py \
    build \
    --api=18 \
    --hdf5=/Library/Frameworks/Python.framework/Versions/Current

Install the software using the following (longish) command:

sudo \
    /Library/Frameworks/Python.framework/Versions/Current/bin/python \
    setup.py \
    install