A small introduction to Python Eggs - mrtopf.demrtopf.de

A small introduction to Python Eggs

Python Eggs(Attention: This is a very technical post mostly for Python developerts. If you are not a programmer you might want to skip this).

During a conversation esp. with Ryan Williams (Which Linden) from Linden Lab we talked about their own Python libraries eventlet and mulib and how great it would be to have them available as Python Eggs (eventlet is a co-routines based networking library and mulib is a REST framework build on top of it. Both are used internally by Linden Lab for the Second Life infrastructure).

If you don’t know what a Python egg is, it’s simply a way of distributing Python packages, similar to RPM. There is also an easy method of installing them, using easy_install.

Where do I find Python Eggs?

You can find Python Eggs on quite a few places on the web, e.g. at a package author’s website. The biggest repository of eggs is the Cheeseshop (or PyPI) though), an index for Python packages. In order to be able to install eggs you simply need to install easy_install which is easily done by downloading ez_install.py (you can download it here) and calling it (you need to have rights to install components in your python installation of course).

Once you have done this you can simply install an egg by calling:

easy_install somepackage.egg

You can also give a URL to an egg and use

easy_install http://somehost.somedomain.com/somepackage.egg

If an egg is not found at that location or in the directory you give, easy_install will automatically query the Cheeseshop for the egg location. So if you want to install SimpleJSON you simply give

easy_install simplejson

and it will download and install the most recent version. If you are not running as root and you have Python installed as root you of course need to use something like „sudo easy_install“ instead (e.g. on MacOSX).

A great additional feature is also that Eggs can define dependencies on other packages which easy_install will then try to automatically download and install aswell.

BTW, the easy_install program is part of the setuptool package by Philip Eby and is based on the distutils package which is part of the standard python distribution.

Creating Python Eggs

Now we know how to install Python Eggs, how can we actually create them? There are actually two ways to do that, one is more manual and the other directly creates a skeleton for you to work with which might be handy for projects you start from scratch.

As an example we will use the above mentioned eventlet package (you can find the SVN repository here). The directory structure right now looks like this:


So as we can see it already has a setup.py which is part of the distutils package and makes it easy to install eventlet by simply typing

python setup.py install

This will build and install the library in your Python installation. You need to make sure you have greenlet installed though.

The setup.py now looks like this:

#!/usr/bin/env python

from distutils.core import setup

    description='Coroutine-based networking library',
    author='Linden Lab',

So we can see some metadata about this package, like name, version, description, author and so on, but also a list of packages to be installed. In this case it’s „eventlet“ and this means that the directory „eventlet“ on the filesystem will be used as the package to install so you later can say from eventlet import .... The directory eventlet will be what will be in your PYTHONPATH.

Now so far it’s great but we might miss some features of eggs here, namely the ability to package it as one file, to register it with the cheeseshop and to define dependencies. So to add this we have to do just a few changes to setup.py.

The first one is to import not from distutils but from setuptools instead:

from setuptools import setup

And then we need to extend the amount of data mostly by further metadata we need for the cheeseshop, such as license and categories:

    description='Coroutine-based networking library',
    author='Linden Lab',
      eventlet is a coroutines-based network library for python ...
          "License :: OSI Approved :: GNU General Public License (GPL)",
          "Programming Language :: Python",
          "Development Status :: 4 - Beta",
          "Intended Audience :: Developers",
          "Topic :: Internet",
      keywords='networking eventlet nonblocking internet',

(actually I haven’t checked the license of eventlet/mublib so this might be wrong but you can change this accordingly).

So we added those classifiers to put it into the right categories on Cheeseshop and most importantly we added the requirement „greenlet“. If you then install the egg setuptools will automatically install greenlet as well.

Once you have this you can create an egg out of this by calling

python setup.py bdist_egg

This will create an egg distribution file which you can find in dist/eventlet-0.1-py2.4.egg (if you used Python 2.4)

You can check the contents by calling unzip -t eventlet-0.1-py2.4.egg and as you can see it’s basically the contents of the eventlet directory plus an EGG_INFO directory with metadata. Also included are the .pyc files as it’s a binary distribution.

The resulting .egg file can now be distributed by whatever means and installed by the above easy_install command.

Starting from scratch with Python Paste

If you just want to start your project there is an easier way to create the initial structure. You simply use Python Paste and a template. What we are interested in is especially paste.script which is some sort of template based automation system.

So here is what you do to install the right packages:

easy_install -U ZopeSkel

You can alternatively just do

easy_install -U PasteScript

but ZopeSkel will install PasteScript as well and offers additionally templates which might be of use to you.

Now that PasteScript is installed you can list the available templates:

  $ paster create --list-templates
  Available templates:
  archetype:          A Plone project that uses Archetypes
  basic_namespace:    A project with a namespace package
  basic_package:      A basic setuptools-enabled package
  basic_zope:         A Zope project
  nested_namespace:   A project with two nested namespaces.
  paste_deploy:       A web application deployed through paste.deploy
  plone:              A Plone project
  plone2.5_buildout:  A buildout for Plone 2.5 projects
  plone2.5_theme:     A Theme for Plone 2.5
  plone2_theme:       A Theme Product for Plone 2.1 & Plone 2.5
  plone3_buildout:    A buildout for Plone 3 projects
  plone3_portlet:     A Plone 3 portlet
  plone3_theme:       A Theme for Plone 3.0
  plone_app:          A Plone App project
  plone_hosting:      Plone hosting: buildout with ZEO and any Plone version
  tgbase:             tg base template
  tgbig:              For more complex projects
  tgwidget:           TurboGears widget projects
  turbogears:         web framework

(I have Turbogears installed as well which is why here are more templates listed than you have)

The interesting ones for Python are basic_package and basic_namespace. We will use basic_package for now as it creates a package ready for egg distribution.

So we can call it like this:

paster create -t basic_package

and it will ask us a lot of questions, basically all the metadata we have to put into the setup.py file. After it’s finished you should have a new directory with a basic egg supporting structure. Now you just need to code your actuall component but that’s left as an exercise for the reader!

Tags: , , , , , , , , ,

29 Kommentare » Schreibe einen Kommentar

  1. Great introduction and something I wasn't aware of. As a programming team we've just recently moved to Python from PHP so its good to see what's out there.

  2. Thanks :-) It's indeed quite handy. As well as buildout which is something I maybe also should do a tutorial on (but I think there actually is one). buildout builds on eggs and it's definitely quite useful for teams to make sure they work in equally setup sandboxes.

  3. Pingback: Topic Maps 2008: Tutorial Day | TopicObserver.com

  4. Pingback: Announcing pydataportability 0.1 (technical) — mrtopf.de

  5. Pingback: A first shot at refactoring the OGP login in Python (pyogp) (technical) — mrtopf.de

  6. Pingback: python GASP - openSUSE Forums

  7. Thank you for this introduction.
    I didn't know about paster. Very interesting.

  8. Pingback: Getting pushloghtml Up and Running Part One | The Unheralded Perspective

  9. Pingback: Tagz | "A small introduction to Python Eggs — mrtopf.de" | Comments

  10. [[[scream]]]] How are you supposed to get easy_install setup? I have Python 2.6.2 set up on my PC but it doesn't seem to have easy_install anywhere I can find.

  11. Mr. Topf, do you know of a site that will tell you examples of when to use an egg? I mean… I need to write some code in python for a certain project in zope/plone. But my boss is thinking about rewriting this app in either Zope 3 or Django… So, I wonder if I should create an egg for this.

  12. Pingback: Trying out mod_fcgid « Teh Tech Blogz0r by Luke Meyer

  13. Nice article – Thanks.
    One little correction:
    That would be dependEncies, not dependAncies throughout.
    And this would be nitpicking

  14. Pingback: Python: distribution systems world | Supportex.net

  15. Pingback: python

  16. There is a typo…

    in the second setup.py ,
    instead of


    you should put:

    packages=[‚eventlet‘] ,

  17. I am trying to install NetworkX, which I can download as a Python egg. Unfortunately, I don’t understand what to do with this.

    I have read this page and others and I can’t make any sense of it, unfortunately.

    Trying to download Easy Installer at this site:
    I am thoroughly confused. That site is just a page of code. If I run it in Python, nothing seems to happen.

    „Once you have done this you can simply install an egg by calling: easy_install somepackage.egg“

    What does this mean? Do you just type „easy_install somepackage.egg“ into Python and run it?

    I have previously downloaded matplotlib and numpy etc with no problems, but this is causing me a lot of problems. Could you break down the instructions a bit further, please.


    • Regarding ez_setup: You simply save it and running it via python ez_setup.py should install setuptools for your python installation. Today most linux based systems come with some package like python-setuptools already though so you might simply install this instead (maybe it is installed already). Make sure to run ez_setup.py as root though.

      After that you have a new command „easy_install“ available and if a package is on PyPI you can in many cases simply say

      easy_install NetworkX

      (again as root in case you are not using a virtual python environment). I just did that and it worked.

      In this case it will search pypi for that package name, download and install it. It’s not working with all packages but with quite a lot. If it does not work you have to download the package manually and follow installation instructions as usual (many times it’s python setup.py install).

      In case you have downloaded an .egg file you can install it as you wrote with easy_install somepackage.egg

      I hope this helps!

  18. OK, my next question is, what does „as root“ mean.

    When I saved the code and ran it as a module from IDLE, I got the following message:

    Processing setuptools-0.6c11-py2.6.egg
    Copying setuptools-0.6c11-py2.6.egg to /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
    Adding setuptools 0.6c11 to easy-install.pth file
    Installing easy_install script to /Library/Frameworks/Python.framework/Versions/2.6/bin
    Installing easy_install-2.6 script to /Library/Frameworks/Python.framework/Versions/2.6/bin

    Installed /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
    Processing dependencies for setuptools==0.6c11
    Finished processing dependencies for setuptools==0.6c11

    This seemed promising. However, trying to run „easy_install NetworkX“ or „easy_install networkx-1.5-py2.6.egg“ just returns an invalid syntax error message. I’m guessing that my lack of understanding of running script „as root“ is the issue.

    Thanks again.


  19. Pingback: Python 101: Setting up Python on Windows « The Mouse Vs. The Python

  20. Pingback: Hooking into the whole Python way of distributing code | skipperkongen.dk