In Defense of PyPI

Everyone on the Python Planet is probably already familiar with Peter Fein’s recent article about PyPI use (or lack thereof). But in case not, particularly striking was the number of folks who joined the “PyPI bashing” in the comments. In fact, it has inspired me to write this post “in defense of PyPI”. I would like to offer the Python community a summary of what I think are the general criticisms, along with my responses as a “sysadmin / developer type”.

First let me say this: I love PyPI! And I agree with Peter, if your package isn’t on PyPI it  “doesn’t exist”. I wouldn’t put it quite like that; but I would say it’s fairly important if you are publishing open source Python code, to consider uploading it to the Python Package Index.

Why?

Because Everybody Wins

Believe it or not, the general Python community is interested in seeing your code. Whether to use it for an example, or to avoid reinventing the wheel, or whatever the reason; we’d like a chance to see your code. But if you don’t publish it to PyPI, we may never get that chance!

For better or worse, PyPI is the canonical place on Earth for Python packages. It’s the CPAN of Python. I understand that not everyone is 100% comfortable with this, but that doesn’t make it any less true. If you accept that “open source is good”, and that “Python rules”, then you simply must take this next leap of faith: “PyPI is the place for Python packages”.

[waves hand]

Moving on, why else should you consider uploading your packages to PyPI?

Because It Is The “Right” Thing To Do

Another thing that struck me is the number of folks who (appear to) confuse “version control” with “distribution”. If I’m not mistaken, Launchpad, Github, and Bitbucket are primarily designed for Bazaar, Git, and Mercurial hosting respectively. These sites can host your distribution tarballs, but they certainly weren’t designed and built to do so. Rather, they were designed and built to host your source code.

In some cases, a project may wish to host it’s own distribution server. Whether it be for redundancy (although PyPI has begun to tackle this) or “branding” or other reasons, I would argue this is the preferred way of handling it: in addition to uploading to PyPI, not in place of it.

Why?

Because It Is Not That Hard

Ahem… we get it. The situation with easy_install is “less than ideal”. But this is something to be fixed, not avoided. If you are receiving too many support requests, may I suggest simply telling people not to use easy_install. Or, if the problem is proper packaging, learn how to test your packages before uploading them. Due to the large number of screwed up releases I’ve made, I’ve come to rely on a local PyPI and a virtualenv to test installations. Others use even simpler methods. And with tools like mkrelease, it’s easy to upload your package to multiple PyPI locations with just a single command (although leaping-tall-buildings-in-a single-bound is not yet supported.)

The point is, please consider helping the community fix the problem rather than simply avoiding it. There are folks actively trying to improve the situation right now.

Let’s see, what else?

Because It Does Not Have To Be Perfect

Over the years I’ve seen various and sundry criticisms of the PyPI user interface. Fine. I have not looked into the current development process, but I assume the author/maintainers would be open to some constructive criticism and/or development assistance.

It doesn’t have to be Github-sexy to be useful. If you would like to report a bug or feature request, do it here (at least, I think that is the right place.)

Conclusion

I hope this convinces at least some folks to consider uploading their packages to PyPI. If it doesn’t, please let me know why in the comments.

Did you enjoy reading this article? If so, please consider helping me help Plone.

Hostout rules

As in “dominates”, not a “list of rules” :-)

I spent a long time avoiding collective.hostout for reasons I will call “technical stylistic differences” with the author Dylan Jay.

But then I grew up (and in cases of Plone growth like this, I usually end up crediting the probably-much-younger-than-me-in-years, but-certainly-much-more-possessive-of-sage-like-wisdom-than-me-Martin-Aspeli; and this time is certainly no exception. So thanks Martin!)

As it turns out, Hostout (as I like to call it now, because we are on a first name basis) is awesome! Let me break down the awesomeness for you.

Buildout

It starts with Buildout. I won’t bore you with the details of why Buildout is awesome, but I will mention one key feature:

  • INI-style configuration, y’all.

Sometimes you just want to specify something like:

[foo]
bar = baz

and be done. I am not making this up folks. There is something about INI-style configuration that transcends time and space, and reaches across partisan-aisles to bring folks together in a “let’s just get this done” sort of way. Of course not everyone likes them, but if you are a technical person and don’t like them, I guarantee you at least understand them and can see why they might be appropriate in some cases over another technology.

Anyway, back to the awesomeness.

Fabric

Fabric is awesome! I won’t bore you with the details of why Fabric is awesome. But I will mention one key feature:

  • SSH, people. SSH.

Sometimes you just want to type:

$ fab -H foo.com bar

and be done. Fabric lets you execute the Python code you wrote in a function called “bar”, inside a (local) file called fabfile.py, on a (remote) host called foo.com. All with the touch of a button, so to speak. That is to say, you create a local fabfile.py file, and Fabric runs the code remotely for you on any number of hosts you give it. Brilliant.

Now, what could be better than these two technologies? These two technologies together, that’s what. (See where I am going with this?) And what could be better than joining these two technologies? Joining them with absolutely no effort required on your part whatsoever, that’s what.

I have a long history of touting software that makes my job (and life) easier, and Hostout fits right in with that tradition. I’m just sorry it took me this long to try it out.

Hostout

As compelling as it is to write “pure Python” in fab files, and it is, sometimes you just want to be done. And Hostout helps you get there. I will explain in a minute, but first a slight detour.

Back story

I recently blogged about the relaunch of this website here. Shortly after the launch, I realized I needed a quick and easy way to deploy my staging site to production. I pitched the idea of a utility called “mr.pusher” to accomplish this task to Dylan, with the caveat that I was interested in making it work with Hostout.

As it turns out, between Hostout and a recipe I created years ago called collective.recipe.rsync, “mr.pusher” almost already existed. Now back to the story.

Fabric integration made easy

Since I knew I wanted to be able to execute a “push” command remotely, I figured I would finally get acclimated with Hostout.

After a small wrestling match in my head with the docs, I came up with this:

# Hostout makes Fabric integration easy
[staging]
recipe = collective.hostout
host = aclark.net
path = /srv/staging

[production]
recipe = collective.hostout
host = aclark.net
path = /srv/aclark_net_website

At this point (after running buildout) I was able to perform such great feats of remote administration (on my already-deployed-sans-hostout sites) as:

$ bin/hostout staging run git pull
$ bin/hostout staging run bin/buildout -c staging.cfg

Armed with this ability, I added the following to my staging.cfg:

# Create scripts to deploy staging data to production.
# Be VERY careful with this. You could easily overwrite your
# live production data if you either forget to use the script
# option, or accidentally run the bin/rsync-filestorage-to-production
# script without stopping the production site first.
[filestorage-to-production]
recipe = collective.recipe.rsync
source = var/filestorage/Data.fs
target = ../aclark_net_website/var/filestorage/Data.fs
script = true

[blobstorage-to-production]
recipe = collective.recipe.rsync
source = var/blobstorage/
target = ../aclark_net_website/var/blobstorage/
script = true

The result was two scripts I could use to “push” staging to production, but only after stopping the production site first:

$ bin/hostout production run bin/supervisorctl shutdown
$ bin/hostout staging run bin/rsync-filestorage-to-production
$ bin/hostout staging run bin/rsync-blobstorage-to-production
$ bin/hostout production run bin/supervisord

And we’re deployed! This technique is particularly elegant when used in combination with a custom 503.

Conclusion

I really like this setup, and I really appreciate what Hostout gets me “for free”; I didn’t have to create a fabfile.py, I just configured my staging and production host parameters in buildout.cfg and off I went.

Of course, there is always room for improvement. The next most logical step for me would be to add cloud integration. If I could provision a new Rackspace cloud server via buildout, and then host-it-out, I think the future will have arrived. And since that is exactly what hostout.cloud appears to do, welcome to the future!

Alex Clark (January 2011)

Did you enjoy reading this article? Please consider helping me help Plone in February 2011.

You should buy “Plone 3 for Education” right now

With what must be the fifth or sixth round of apologies to both Erik Rose and PACKT publishing for the delay (very sorry guys), here is my review! Better late than never.

I get bored with the same ol’ blog entry titles:

“A review of so and so’s book, by such and such…”

Not that there is anything wrong with these titles, they serve their purpose. It’s not you… it’s me. Can we still be friends?

Anyway, I will cut to the chase with this one: if you care about Plone and you have not done so already, you should buy Plone 3 for Education right now.

Why?

Several reasons:

  • Erik Rose cares about Plone. He cares so much, he spent months of his life pouring his professional experiences into a book so you can benefit!
  • PACKT publishing will donate a percentage of the proceeds to the Plone Foundation.
  • You will learn a lot about Plone, from the perspective of someone who has supported its deployment and maintenance within a large educational institution.

That is not to say every Plone book is for you. You have to decide for yourself if you want this beautiful creation (see above) of Erik’s to shine on your mantle piece for years to come. Even if you are a giant Plone supporter, and even if you make your living off of Plone, you may still not want to purchase this book. That’s just the way that it goes. But, I will to try to convince you anyway! Here is my chapter-by-chapter review:

Preface

The preface sets the tone of the book, and I like the tone Erik sets. Erik has used Plone for many years at Penn State and you will find his experiences valuable. He’s made mistakes so you don’t have to!

Chapter 1

Chapter 1 covers creating courses (of course!) Erik makes the argument that Plone can hold it’s own against expensive course management software, and I believe him. Of particular interest are Erik’s recommendations to:

  • Put courses in large Plone folders
  • Use news items, events and collections in the context of a course.
  • Cut and paste the course framework once you are done creating it.

Chapter 2

Chapter 2 introduces calendaring in Plone, and Erik suggests using Plone4artists calendar to make your course content easily browsable. Of particular interest are Erik’s recommendations to:

  • Replace the stock Plone events folder (which is a collection contained within a folder) with (just) a collection.
  • Make clever use of keywords to support lots of events.
  • Reorder subfolders in a collection “the hard way”. (Also interesting that he opened this ticket: https://dev.plone.org/plone/ticket/9109 and that it has received no “love” to date. Perhaps the new collections UI: http://dev.plone.org/plone/ticket/10902 will save the day!)

Chapter 3

Chapter 3 showcases the Faculty/Staff Directory (FSD) add-on for Plone which allows you to create an online personnel directory. Of particular interest is Erik’s explanation of how to categorize and correlate people within FSD.

Chapter 4

Chapter 4 covers customizing Faculty/Staff Directory. Of particular interest is Erik’s coverage of AT Schema Extender (which is a great way to customize Plone these days).

Chapter 5

Chapter 5 covers blogs and forums. Of particular interest is Erik’s coverage of Scrawl, and the need to exercise extreme caution when installing add-ons (always good advice).

Chapter 6

Chapter 6 covers audio and video. Of particular interest is Erik’s coverage of collective.flowplayer which makes Flowplayer easy to use in Plone.

Chapter 7

Chapter 7 covers forms. Of particular interest is Erik’s comparison of PloneFormGen to Archetypes content objects, and his explanation of when to use one over the other.

Chapter 8

Chapter 8 covers theming. Of particular interest is Erik’s coverage of “new style” template customizations via z3c.jbot (which you don’t even need a package for!)

Chapter 9

Chapter 9 covers deployment. Of particular interest is Erik’s coverage of a typical production software stack that includes ZEO, Zope2, Apache and of course Plone. Also covered is CacheFu, which has been superseded by plone.app.caching in Plone 4.

Chapter 10

Chapter 10 covers maintenance. Of particular interest is Erik’s coverage of collective.recipe.backup to automate the creation of repozo scripts. I would add to that, you can use z3c.recipe.usercrontab to automate the creation and removal of cron jobs to execute those scripts.

Conclusion

All in all, I enjoyed reading this book. And, I appreciate whenever a member of the community shares his/her experiences. So thank you, Erik!

That’s it! I hope you will consider purchasing a copy of Plone 3 for Education right now.

And if you enjoyed reading this review, please consider helping me help Plone in February 2011.

Alex Clark (January 2011)