• flickr
  • twitter
  • facebook
  • delicious
  • Google Buzz

A small introduction to Python Eggs

by Christian Scholz on November 21, 2007

Python Eggs(Attention: This is a very technical post mostly for Python developerts. If you are not a programmer you might want to skip this).

During a conversation esp. with Ryan Williams (Which Linden) from Linden Lab we talked about their own Python libraries eventlet and mulib and how great it would be to have them available as Python Eggs (eventlet is a co-routines based networking library and mulib is a REST framework build on top of it. Both are used internally by Linden Lab for the Second Life infrastructure).

If you don’t know what a Python egg is, it’s simply a way of distributing Python packages, similar to RPM. There is also an easy method of installing them, using easy_install.

Where do I find Python Eggs?

You can find Python Eggs on quite a few places on the web, e.g. at a package author’s website. The biggest repository of eggs is the Cheeseshop (or PyPI) though), an index for Python packages. In order to be able to install eggs you simply need to install easy_install which is easily done by downloading ez_install.py (you can download it here) and calling it (you need to have rights to install components in your python installation of course).

Once you have done this you can simply install an egg by calling:

easy_install somepackage.egg

You can also give a URL to an egg and use

easy_install http://somehost.somedomain.com/somepackage.egg

If an egg is not found at that location or in the directory you give, easy_install will automatically query the Cheeseshop for the egg location. So if you want to install SimpleJSON you simply give

easy_install simplejson

and it will download and install the most recent version. If you are not running as root and you have Python installed as root you of course need to use something like “sudo easy_install” instead (e.g. on MacOSX).

A great additional feature is also that Eggs can define dependencies on other packages which easy_install will then try to automatically download and install aswell.

BTW, the easy_install program is part of the setuptool package by Philip Eby and is based on the distutils package which is part of the standard python distribution.

Creating Python Eggs

Now we know how to install Python Eggs, how can we actually create them? There are actually two ways to do that, one is more manual and the other directly creates a skeleton for you to work with which might be handy for projects you start from scratch.

As an example we will use the above mentioned eventlet package (you can find the SVN repository here). The directory structure right now looks like this:

README
eventlet
examples
setup.py

So as we can see it already has a setup.py which is part of the distutils package and makes it easy to install eventlet by simply typing

python setup.py install

This will build and install the library in your Python installation. You need to make sure you have greenlet installed though.

The setup.py now looks like this:

#!/usr/bin/env python

from distutils.core import setup

setup(
    name='eventlet',
    version='0.1',
    description='Coroutine-based networking library',
    author='Linden Lab',
    author_email='sldev@lists.secondlife.com',
    url='http://wiki.secondlife.com/wiki/Eventlet',
    packages=['eventlet'])

So we can see some metadata about this package, like name, version, description, author and so on, but also a list of packages to be installed. In this case it’s “eventlet” and this means that the directory “eventlet” on the filesystem will be used as the package to install so you later can say from eventlet import .... The directory eventlet will be what will be in your PYTHONPATH.

Now so far it’s great but we might miss some features of eggs here, namely the ability to package it as one file, to register it with the cheeseshop and to define dependencies. So to add this we have to do just a few changes to setup.py.

The first one is to import not from distutils but from setuptools instead:

from setuptools import setup

And then we need to extend the amount of data mostly by further metadata we need for the cheeseshop, such as license and categories:

setup(
    name='eventlet',
    version='0.1',
    description='Coroutine-based networking library',
    author='Linden Lab',
    author_email='sldev@lists.secondlife.com',
    url='http://wiki.secondlife.com/wiki/Eventlet',
    packages=['eventlet'],
      long_description="""\
      eventlet is a coroutines-based network library for python ...
      """,
      classifiers=[
          "License :: OSI Approved :: GNU General Public License (GPL)",
          "Programming Language :: Python",
          "Development Status :: 4 - Beta",
          "Intended Audience :: Developers",
          "Topic :: Internet",
      ],
      keywords='networking eventlet nonblocking internet',
      license='GPL',
      install_requires=[
        'setuptools',
        'greenlet',
      ],
      )

(actually I haven’t checked the license of eventlet/mublib so this might be wrong but you can change this accordingly).

So we added those classifiers to put it into the right categories on Cheeseshop and most importantly we added the requirement “greenlet”. If you then install the egg setuptools will automatically install greenlet as well.

Once you have this you can create an egg out of this by calling

python setup.py bdist_egg

This will create an egg distribution file which you can find in dist/eventlet-0.1-py2.4.egg (if you used Python 2.4)

You can check the contents by calling unzip -t eventlet-0.1-py2.4.egg and as you can see it’s basically the contents of the eventlet directory plus an EGG_INFO directory with metadata. Also included are the .pyc files as it’s a binary distribution.

The resulting .egg file can now be distributed by whatever means and installed by the above easy_install command.

Starting from scratch with Python Paste

If you just want to start your project there is an easier way to create the initial structure. You simply use Python Paste and a template. What we are interested in is especially paste.script which is some sort of template based automation system.

So here is what you do to install the right packages:

easy_install -U ZopeSkel

You can alternatively just do

easy_install -U PasteScript

but ZopeSkel will install PasteScript as well and offers additionally templates which might be of use to you.

Now that PasteScript is installed you can list the available templates:

  $ paster create --list-templates
  Available templates:
  archetype:          A Plone project that uses Archetypes
  basic_namespace:    A project with a namespace package
  basic_package:      A basic setuptools-enabled package
  basic_zope:         A Zope project
  nested_namespace:   A project with two nested namespaces.
  paste_deploy:       A web application deployed through paste.deploy
  plone:              A Plone project
  plone2.5_buildout:  A buildout for Plone 2.5 projects
  plone2.5_theme:     A Theme for Plone 2.5
  plone2_theme:       A Theme Product for Plone 2.1 & Plone 2.5
  plone3_buildout:    A buildout for Plone 3 projects
  plone3_portlet:     A Plone 3 portlet
  plone3_theme:       A Theme for Plone 3.0
  plone_app:          A Plone App project
  plone_hosting:      Plone hosting: buildout with ZEO and any Plone version
  tgbase:             tg base template
  tgbig:              For more complex projects
  tgwidget:           TurboGears widget projects
  turbogears:         web framework

(I have Turbogears installed as well which is why here are more templates listed than you have)

The interesting ones for Python are basic_package and basic_namespace. We will use basic_package for now as it creates a package ready for egg distribution.

So we can call it like this:

paster create -t basic_package

and it will ask us a lot of questions, basically all the metadata we have to put into the setup.py file. After it’s finished you should have a new directory with a basic egg supporting structure. Now you just need to code your actuall component but that’s left as an exercise for the reader!

Tags: , , , , , , , , ,

29 comments

Thanks for the nice introduction!

by Claus Conrad on 24.11.2007 at 10:29. #

Great introduction and something I wasn't aware of. As a programming team we've just recently moved to Python from PHP so its good to see what's out there.

by Tenders on 25.11.2007 at 23:45. #

Thanks :-) It's indeed quite handy. As well as buildout which is something I maybe also should do a tutorial on (but I think there actually is one). buildout builds on eggs and it's definitely quite useful for teams to make sure they work in equally setup sandboxes.

by Christian Scholz on 26.11.2007 at 00:53. #

[...] ZTM3 uses the new architecture/design of Zope3 and will be module based for flexibility (interfaces, object storage, adapters, utilities, site managers, views, layers, skins, Python eggs). [...]

by Topic Maps 2008: Tutorial Day | TopicObserver.com on 2.4.2008 at 22:29. #

[...] To show how it works I created an example buildout. If you don’t know what an buildout is then don’t worry, you don’t need to know for the example (basically it allows you to create a sandbox and install various eggs in it. Very useful for developing in a group or deploying software. Check out this screencast for a demo). In case you don’t know what Python eggs are, then please read my tutorial on them. [...]

by Announcing pydataportability 0.1 (technical) — mrtopf.de on 21.4.2008 at 08:49. #

[...] this by using the Zope Component Architecture (for an intro see here), buildout and eggs (for an egg introduction see here, buildout needs to be written). The result can be found in a base library package on Google [...]

by A first shot at refactoring the OGP login in Python (pyogp) (technical) — mrtopf.de on 24.6.2008 at 17:36. #

[...] python GASP A small introduction to Python Eggs — mrtopf.de Python Package Index : gasp 0.4.5 Should have enough info unsure how to make a proper pkg, or how [...]

by python GASP - openSUSE Forums on 29.7.2008 at 20:01. #

Thank you for this introduction.
I didn't know about paster. Very interesting.

by Cyberdrow on 31.7.2008 at 14:44. #

[...] had told me, use ez_install.py to install these libs. So I hunted down and found the file on this blog (there doesn’t seem to be an official place where this is housed). I downloaded and installed [...]

by Getting pushloghtml Up and Running Part One | The Unheralded Perspective on 6.10.2008 at 03:30. #

Nice one!!!! helped me a lot :-)

by Sandeep Kekuda on 12.2.2009 at 14:18. #

[...] [upmod] [downmod] A small introduction to Python Eggs — mrtopf.de (mrtopf.de) 1 points posted 10 months, 1 week ago by jeethu tags python eggs tutorial [...]

by Tagz | "A small introduction to Python Eggs — mrtopf.de" | Comments on 16.5.2009 at 19:04. #

[[[scream]]]] How are you supposed to get easy_install setup? I have Python 2.6.2 set up on my PC but it doesn't seem to have easy_install anywhere I can find.

by Jason S on 10.7.2009 at 01:26. #

As described above, simply download it here:
http://peak.telecommunity.com/dist/ez_setup.py

and run it with

python ez_setup.py

this will install easy_install. You need to be root (or use sudo) in case you are on unix and want to install it in your system python.

by Christian Scholz on 13.7.2009 at 13:23. #

Mr. Topf, do you know of a site that will tell you examples of when to use an egg? I mean… I need to write some code in python for a certain project in zope/plone. But my boss is thinking about rewriting this app in either Zope 3 or Django… So, I wonder if I should create an egg for this.

by Eric on 1.8.2009 at 01:55. #

depletion disease china output paleoclimatology governments

by parrishmar on 2.8.2009 at 19:06. #

regional combined december population led exempt nations relates

by skeltonfer on 2.8.2009 at 20:09. #

[...] yum install python-setuptools so I could use the easy_install to install the flup egg (thanks to this blog for introducing [...]

by Trying out mod_fcgid « Teh Tech Blogz0r by Luke Meyer on 7.9.2010 at 20:03. #

s/dependancies/dependencies/g

by Ewan on 12.1.2011 at 19:03. #

Nice article – Thanks.
One little correction:
That would be dependEncies, not dependAncies throughout.
And this would be nitpicking

by tomm174 on 3.8.2011 at 23:06. #

[...] A small introduction to Python Eggs by by Christian Scholz [...]

by Python: distribution systems world | Supportex.net on 5.8.2011 at 13:08. #

[...] A small introduction to Python Eggs by by Christian Scholz [...]

by python on 6.8.2011 at 12:36. #

There is a typo…

in the second setup.py ,
instead of

packages=['eventlet'])

you should put:

packages=['eventlet'] ,

by Zvi Fer on 1.11.2011 at 17:32. #

thanks, fixed!

by Christian Scholz on 3.11.2011 at 16:44. #

I am trying to install NetworkX, which I can download as a Python egg. Unfortunately, I don’t understand what to do with this.

I have read this page and others and I can’t make any sense of it, unfortunately.

Trying to download Easy Installer at this site:
http://peak.telecommunity.com/dist/ez_setup.py
I am thoroughly confused. That site is just a page of code. If I run it in Python, nothing seems to happen.

“Once you have done this you can simply install an egg by calling: easy_install somepackage.egg”

What does this mean? Do you just type “easy_install somepackage.egg” into Python and run it?

I have previously downloaded matplotlib and numpy etc with no problems, but this is causing me a lot of problems. Could you break down the instructions a bit further, please.

Thanks

by Thomas Evans on 15.11.2011 at 13:39. #

Regarding ez_setup: You simply save it and running it via python ez_setup.py should install setuptools for your python installation. Today most linux based systems come with some package like python-setuptools already though so you might simply install this instead (maybe it is installed already). Make sure to run ez_setup.py as root though.

After that you have a new command “easy_install” available and if a package is on PyPI you can in many cases simply say

easy_install NetworkX

(again as root in case you are not using a virtual python environment). I just did that and it worked.

In this case it will search pypi for that package name, download and install it. It’s not working with all packages but with quite a lot. If it does not work you have to download the package manually and follow installation instructions as usual (many times it’s python setup.py install).

In case you have downloaded an .egg file you can install it as you wrote with easy_install somepackage.egg

I hope this helps!

by Christian Scholz on 15.11.2011 at 15:26. #

Thank you

by Thomas Evans on 15.11.2011 at 16:10. #

OK, my next question is, what does “as root” mean.

When I saved the code and ran it as a module from IDLE, I got the following message:

Processing setuptools-0.6c11-py2.6.egg
Copying setuptools-0.6c11-py2.6.egg to /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages
Adding setuptools 0.6c11 to easy-install.pth file
Installing easy_install script to /Library/Frameworks/Python.framework/Versions/2.6/bin
Installing easy_install-2.6 script to /Library/Frameworks/Python.framework/Versions/2.6/bin

Installed /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg
Processing dependencies for setuptools==0.6c11
Finished processing dependencies for setuptools==0.6c11

This seemed promising. However, trying to run “easy_install NetworkX” or “easy_install networkx-1.5-py2.6.egg” just returns an invalid syntax error message. I’m guessing that my lack of understanding of running script “as root” is the issue.

Thanks again.

Thomas

by Thomas Evans on 16.11.2011 at 11:54. #

[...] easy_install was that it was the defacto standard for install Python Eggs, which you can read about here or here. Now you can use pip or distribute to do the same thing. All three can also install [...]

by Python 101: Setting up Python on Windows « The Mouse Vs. The Python on 24.11.2011 at 17:32. #