(Attention: This is a very technical post mostly for Python developerts. If you are not a programmer you might want to skip this).
During a conversation esp. with Ryan Williams (Which Linden) from Linden Lab we talked about their own Python libraries eventlet and mulib and how great it would be to have them available as Python Eggs (eventlet is a co-routines based networking library and mulib is a REST framework build on top of it. Both are used internally by Linden Lab for the Second Life infrastructure).
If you don’t know what a Python egg is, it’s simply a way of distributing Python packages, similar to RPM. There is also an easy method of installing them, using easy_install.
Where do I find Python Eggs?
You can find Python Eggs on quite a few places on the web, e.g. at a package author’s website. The biggest repository of eggs is the Cheeseshop (or PyPI) though), an index for Python packages. In order to be able to install eggs you simply need to install easy_install which is easily done by downloading ez_install.py (you can download it here) and calling it (you need to have rights to install components in your python installation of course).
Once you have done this you can simply install an egg by calling:
easy_install somepackage.egg
You can also give a URL to an egg and use
easy_install http://somehost.somedomain.com/somepackage.egg
If an egg is not found at that location or in the directory you give, easy_install will automatically query the Cheeseshop for the egg location. So if you want to install SimpleJSON you simply give
easy_install simplejson
and it will download and install the most recent version. If you are not running as root and you have Python installed as root you of course need to use something like „sudo easy_install“ instead (e.g. on MacOSX).
A great additional feature is also that Eggs can define dependencies on other packages which easy_install will then try to automatically download and install aswell.
BTW, the easy_install program is part of the setuptool package by Philip Eby and is based on the distutils package which is part of the standard python distribution.
Creating Python Eggs
Now we know how to install Python Eggs, how can we actually create them? There are actually two ways to do that, one is more manual and the other directly creates a skeleton for you to work with which might be handy for projects you start from scratch.
As an example we will use the above mentioned eventlet package (you can find the SVN repository here). The directory structure right now looks like this:
README eventlet examples setup.py
So as we can see it already has a setup.py which is part of the distutils package and makes it easy to install eventlet by simply typing
python setup.py install
This will build and install the library in your Python installation. You need to make sure you have greenlet installed though.
The setup.py now looks like this:
#!/usr/bin/env python
from distutils.core import setup
setup(
name='eventlet',
version='0.1',
description='Coroutine-based networking library',
author='Linden Lab',
author_email='sldev@lists.secondlife.com',
url='http://wiki.secondlife.com/wiki/Eventlet',
packages=['eventlet'])
So we can see some metadata about this package, like name, version, description, author and so on, but also a list of packages to be installed. In this case it’s „eventlet“ and this means that the directory „eventlet“ on the filesystem will be used as the package to install so you later can say from eventlet import .... The directory eventlet will be what will be in your PYTHONPATH.
Now so far it’s great but we might miss some features of eggs here, namely the ability to package it as one file, to register it with the cheeseshop and to define dependencies. So to add this we have to do just a few changes to setup.py.
The first one is to import not from distutils but from setuptools instead:
from setuptools import setup
And then we need to extend the amount of data mostly by further metadata we need for the cheeseshop, such as license and categories:
setup(
name='eventlet',
version='0.1',
description='Coroutine-based networking library',
author='Linden Lab',
author_email='sldev@lists.secondlife.com',
url='http://wiki.secondlife.com/wiki/Eventlet',
packages=['eventlet'],
long_description="""\
eventlet is a coroutines-based network library for python ...
""",
classifiers=[
"License :: OSI Approved :: GNU General Public License (GPL)",
"Programming Language :: Python",
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"Topic :: Internet",
],
keywords='networking eventlet nonblocking internet',
license='GPL',
install_requires=[
'setuptools',
'greenlet',
],
)
(actually I haven’t checked the license of eventlet/mublib so this might be wrong but you can change this accordingly).
So we added those classifiers to put it into the right categories on Cheeseshop and most importantly we added the requirement „greenlet“. If you then install the egg setuptools will automatically install greenlet as well.
Once you have this you can create an egg out of this by calling
python setup.py bdist_egg
This will create an egg distribution file which you can find in dist/eventlet-0.1-py2.4.egg (if you used Python 2.4)
You can check the contents by calling unzip -t eventlet-0.1-py2.4.egg and as you can see it’s basically the contents of the eventlet directory plus an EGG_INFO directory with metadata. Also included are the .pyc files as it’s a binary distribution.
The resulting .egg file can now be distributed by whatever means and installed by the above easy_install command.
Starting from scratch with Python Paste
If you just want to start your project there is an easier way to create the initial structure. You simply use Python Paste and a template. What we are interested in is especially paste.script which is some sort of template based automation system.
So here is what you do to install the right packages:
easy_install -U ZopeSkel
You can alternatively just do
easy_install -U PasteScript
but ZopeSkel will install PasteScript as well and offers additionally templates which might be of use to you.
Now that PasteScript is installed you can list the available templates:
$ paster create --list-templates Available templates: archetype: A Plone project that uses Archetypes basic_namespace: A project with a namespace package basic_package: A basic setuptools-enabled package basic_zope: A Zope project nested_namespace: A project with two nested namespaces. paste_deploy: A web application deployed through paste.deploy plone: A Plone project plone2.5_buildout: A buildout for Plone 2.5 projects plone2.5_theme: A Theme for Plone 2.5 plone2_theme: A Theme Product for Plone 2.1 & Plone 2.5 plone3_buildout: A buildout for Plone 3 projects plone3_portlet: A Plone 3 portlet plone3_theme: A Theme for Plone 3.0 plone_app: A Plone App project plone_hosting: Plone hosting: buildout with ZEO and any Plone version tgbase: tg base template tgbig: For more complex projects tgwidget: TurboGears widget projects turbogears: web framework
(I have Turbogears installed as well which is why here are more templates listed than you have)
The interesting ones for Python are basic_package and basic_namespace. We will use basic_package for now as it creates a package ready for egg distribution.
So we can call it like this:
paster create -t basic_package
and it will ask us a lot of questions, basically all the metadata we have to put into the setup.py file. After it’s finished you should have a new directory with a basic egg supporting structure. Now you just need to code your actuall component but that’s left as an exercise for the reader!
Tags: python, eggs, egg, rpm, package, introduction, howto, tutorial, development, lindenlab