(I hope my tech colleagues will indulge my thoughts on the recent passing of Beastie Boy Adam Yauch.)

TL;DR: I <3 the Beastie Boys and will miss Adam Yauch; and I suspect many others in the tech world do, and will, too

Where I came from

For as long as I can remember, I’ve been trying to emulate my heroes. It started with pop music in the 80s and artists like Michael Jackson, Madonna, and of course the Beastie Boys. While I was obsessed with Michael Jackson back then, I don’t recall trying too hard to emulate the Beastie Boys… until the 90s.

In the early 90s, I was just out of high school (where I discovered Led Zeppelin, Pink Floyd, Beatles, Stones etc), was working full time at Chili’s, and had no serious plans for college. I was really into music, but didn’t have the social skills required to participate in any meaningful activities, like playing in a band.

But I loved music.

I listened to Nirvana, Beastie Boys, and any number of other pop, rock and rap acts of the 90s. I went to rock shows at the 8×10 in Baltimore and 9:30 club in DC. I went to festivals like the HFSistival and Lollapalooza. Throughout the decade, I formulated my thoughts on how I would, one day, start a band.

To say the Beastie Boys were inspirational to me during this time period would be a colossal understatement (remember the first time you saw the “So Watcha Want?” video?) While I wholly identified with the angst-ridden music of Nirvana, I also equally identified with the good natured spirit of, and dope-ass-jams from, the Beastie Boys: three guys producing kick ass music with, among other instruments, a “bullshit mic, made out of plastic”. Most importantly: they never took themselves too seriously. They made great music, but they were also just like you and me. They made you feel like you could do it, too.

In the late 90s and early to mid 2000s, I finally got a chance to play music. I started a band, played some shows, recorded a few albums, and had a lot of fun. Most importantly: I proved to myself I could do it. The shows I put on with my band were awesome; they were true rock shows: those in which the audience responded positively, viscerally. There is no better feeling in the world than that type of self-expression.

Who I am now

Eventually the band imploded, and I found a new obsession: technology. I have always been obsessed with tech, but never focused on it exclusively until the mid to late 2000s. Now I have a new set of heroes to emulate, and many new things to prove to myself I can do. Maybe I’ll even play music again one day, but if I don’t, I know that I accomplished, on some level, what I set out to do. And that is a great feeling.

Where we are all headed

We are all headed forward, now, without Adam Yauch to inspire us anew. This sucks more than words can express; but it is, alas, a part of life. Without death to make life precious, there would be no joy in living. We all only have a short period of time in life to accomplish our goals, so it is important that we spend time trying to do so. (And if your goals include helping other people, bonus points from Adam.)

I cannot thank Adam Yauch (MCA), Adam Horovitz (Ad-Rock), and Mike D enough; for providing me with all the inspiration I needed, when I needed it, to live my life; it has made all the difference. I know that Ad-Rock and Mike D will continue to live their lives; and whether that includes performing as “Beastie Boys” or not, does not matter to me; I will still be watching, waiting to be inspired again. And to Adam Yauch who has rapped, joked, snowboarded, and philanthropized his way through all our lives: thank you for the inspiration; I use it every day, and I’ll never forget it was you and your closest two brothers Ad-Rock and Mike D who gave it to me.

Operating system

Mac OS X Latest (Lion, at the time of this writing)

Terminal

Mac OS X Terminal

Editor

Vim

Python

Yes. All versions, via the collective Python buildout:

• https://github.com/collective/buildout.python

Plone

Yes. All versions, via pythonpackages.com:

• http://docs.pythonpackages.com/en/latest/advanced.html

Additional tools

A shell script to run virtualenv, buildout, etc:

#!/bin/sh
if ! [ -n "$1" ]
then
    echo "Usage:\n\n$0 <dir>\n"
    exit 1
fi
if ! [ -d $1 ]
then
    mkdir $1
fi
cd $1
virtualenv-2.7 .
bin/pip install zc.buildout
bin/buildout init
cat << EOF > buildout.cfg
[buildout]
extends = http://build.pythonpackages.com/buildout/plone/4.2.x-dev
EOF
bin/buildout
bin/plone start
sleep 4
echo "Adding Plone site..."
curl -d form.submitted:boolean="True" http://admin:admin@localhost:8080/@@plone-addsite\?site_id=Plone
bin/plone stop
bin/plone fg

A Python script to create temporary directories:

#!/usr/bin/env python

import os
import tempfile
print os.path.abspath(tempfile.mkdtemp())

A default buildout config file:

[buildout]
eggs-directory = /Users/aclark/Developer/eggs-directory
download-cache = /Users/aclark/Developer/download-cache
extends-cache = /Users/aclark/Developer/extends-cache

Workflow

My typical workflow looks like this:

• Hang in #plone on irc.freenode.net and wait for questions
• Hear question and get inspired to run Plone
• Run: $ test-plone `tmp`

~ 1 minute later I’m browsing http://localhost:8080/Plone and can install add-ons, check ZMI settings, etc.

Enjoy.

Here are some of my favorite results.

Vanity

$ vanity vanity -v
     vanity-1.0.zip     2011-04-14      352
   vanity-1.1.0.zip     2011-10-26      139
   vanity-1.1.1.zip     2011-10-28      114
   vanity-1.1.2.zip     2011-10-28      145
vanity-1.2.0.tar.gz     2012-01-30        0
-------------------------------------------
Package `vanity` has been downloaded 750 times!

Django

$ vanity django -v
Django-1.1.3.tar.gz     2010-12-23    2,618
Django-1.1.4.tar.gz     2011-02-09    4,476
  Django-1.2.tar.gz     2010-05-17   15,876
Django-1.2.1.tar.gz     2010-05-24   65,120
Django-1.2.2.tar.gz     2010-09-09    2,467
Django-1.2.3.tar.gz     2010-09-11   73,984
Django-1.2.4.tar.gz     2010-12-23   49,904
Django-1.2.5.tar.gz     2011-02-09   63,977
Django-1.2.6.tar.gz     2011-09-10      427
Django-1.2.7.tar.gz     2011-09-11    6,825
  Django-1.3.tar.gz     2011-03-23  238,504
Django-1.3.1.tar.gz     2011-09-10  176,054
-------------------------------------------
Package `Django` has been downloaded 700,232 times!

Plone

$ vanity plone -v
     Plone-3.2.zip     2008-12-31    1,690
   Plone-3.2.1.zip     2009-02-04    2,466
   Plone-3.2.2.zip     2009-03-03    4,893
   Plone-3.2.3.zip     2009-06-20    2,731
   Plone-3.2a1.zip     2008-10-10      976
  Plone-3.2rc1.zip     2008-12-15      758
     Plone-3.3.zip     2009-08-19    4,611
   Plone-3.3.1.zip     2009-09-09    4,148
Plone-3.3.2.tar.gz     2009-11-03    3,043
   Plone-3.3.3.zip     2009-12-08    1,733
   Plone-3.3.4.zip     2010-01-14    4,906
   Plone-3.3.5.zip     2010-03-03   12,131
Plone-3.3.6.tar.gz     2011-07-19      787
   Plone-3.3b1.zip     2009-03-12      940
  Plone-3.3rc1.zip     2009-03-30      743
  Plone-3.3rc2.zip     2009-04-05    1,822
  Plone-3.3rc3.zip     2009-05-23    2,036
  Plone-3.3rc4.zip     2009-07-07    2,163
  Plone-3.3rc5.zip     2009-08-01    1,211
     Plone-4.0.zip     2010-08-30    3,659
   Plone-4.0.1.zip     2010-10-04    4,224
   Plone-4.0.2.zip     2010-11-22    4,022
   Plone-4.0.3.zip     2011-01-21    3,339
   Plone-4.0.4.zip     2011-03-01    2,711
   Plone-4.0.5.zip     2011-04-09    3,152
   Plone-4.0.6.zip     2011-05-22    1,469
   Plone-4.0.7.zip     2011-06-06    2,192
   Plone-4.0.8.zip     2011-07-17      509
   Plone-4.0.9.zip     2011-07-29      958
  Plone-4.0.10.zip     2011-10-12      495
   Plone-4.0a1.zip     2009-11-19      946
   Plone-4.0a2.zip     2009-12-04      921
   Plone-4.0a3.zip     2009-12-21    1,272
   Plone-4.0a4.zip     2010-02-01    1,121
   Plone-4.0a5.zip     2010-02-19      850
   Plone-4.0b1.zip     2010-03-09    1,308
   Plone-4.0b2.zip     2010-04-10    1,028
   Plone-4.0b3.zip     2010-05-04    1,722
   Plone-4.0b4.zip     2010-06-13    1,642
Plone-4.0b5.tar.gz     2010-07-08    1,995
  Plone-4.0rc1.zip     2010-08-06    1,598
     Plone-4.1.zip     2011-07-17    4,479
Plone-4.1.1.tar.gz     2011-09-21      429
Plone-4.1.2.tar.gz     2011-10-08    2,187
Plone-4.1.3.tar.gz     2011-11-29    1,883
   Plone-4.1a1.zip     2011-01-21      699
   Plone-4.1a2.zip     2011-02-11      451
   Plone-4.1a3.zip     2011-02-15      680
   Plone-4.1b1.zip     2011-03-08      958
   Plone-4.1b2.zip     2011-04-09    1,040
  Plone-4.1rc2.zip     2011-05-22      797
  Plone-4.1rc3.zip     2011-06-06    1,329
   Plone-4.2a1.zip     2011-08-10      521
   Plone-4.2a2.zip     2011-09-16      884
Plone-4.2b1.tar.gz     2011-12-06      619
------------------------------------------
Package `Plone` has been downloaded 111,877 times!

Pyramid

$ vanity pyramid -v
   pyramid-1.0.tar.gz     2011-01-31   24,055
 pyramid-1.0.1.tar.gz     2011-08-13      460
 pyramid-1.0.2.tar.gz     2011-12-15      185
 pyramid-1.0a1.tar.gz     2010-11-05    1,128
 pyramid-1.0a2.tar.gz     2010-11-09      952
 pyramid-1.0a3.tar.gz     2010-11-16      803
 pyramid-1.0a4.tar.gz     2010-11-21    1,732
 pyramid-1.0a5.tar.gz     2010-12-15      639
 pyramid-1.0a6.tar.gz     2010-12-16      834
 pyramid-1.0a7.tar.gz     2010-12-20      912
 pyramid-1.0a8.tar.gz     2010-12-27    1,233
 pyramid-1.0a9.tar.gz     2011-01-08    1,313
pyramid-1.0a10.tar.gz     2011-01-18      960
 pyramid-1.0b1.tar.gz     2011-01-22      871
 pyramid-1.0b2.tar.gz     2011-01-25    1,003
 pyramid-1.0b3.tar.gz     2011-01-28      816
   pyramid-1.1.tar.gz     2011-07-22    5,674
 pyramid-1.1.1.tar.gz     2011-08-13    1,057
 pyramid-1.1.2.tar.gz     2011-08-17    1,560
 pyramid-1.1.3.tar.gz     2011-12-15      160
 pyramid-1.1a1.tar.gz     2011-06-20    1,146
 pyramid-1.1a2.tar.gz     2011-06-23    1,003
 pyramid-1.1a3.tar.gz     2011-06-26    1,252
 pyramid-1.1a4.tar.gz     2011-07-01    1,517
 pyramid-1.1b1.tar.gz     2011-07-10      981
 pyramid-1.1b2.tar.gz     2011-07-13      844
 pyramid-1.1b3.tar.gz     2011-07-15      742
 pyramid-1.1b4.tar.gz     2011-07-18    1,094
   pyramid-1.2.tar.gz     2011-09-13    6,450
 pyramid-1.2.1.tar.gz     2011-09-28   10,357
 pyramid-1.2.2.tar.gz     2011-11-20      322
 pyramid-1.2.3.tar.gz     2011-11-21    3,078
 pyramid-1.2.4.tar.gz     2011-12-06    1,193
 pyramid-1.2.5.tar.gz     2011-12-15      555
 pyramid-1.2.6.tar.gz     2012-01-05      514
 pyramid-1.2.7.tar.gz     2012-01-20      472
 pyramid-1.2a1.tar.gz     2011-08-24      907
 pyramid-1.2a2.tar.gz     2011-08-27      696
 pyramid-1.2a3.tar.gz     2011-08-29    3,268
 pyramid-1.2a4.tar.gz     2011-09-02      593
 pyramid-1.2a5.tar.gz     2011-09-04      840
 pyramid-1.2a6.tar.gz     2011-09-07      602
 pyramid-1.2b1.tar.gz     2011-09-08      493
 pyramid-1.2b2.tar.gz     2011-09-08      700
 pyramid-1.2b3.tar.gz     2011-09-11      674
 pyramid-1.3a1.tar.gz     2011-12-09    1,264
 pyramid-1.3a2.tar.gz     2011-12-14    1,641
 pyramid-1.3a3.tar.gz     2011-12-21    2,716
 pyramid-1.3a4.tar.gz     2012-01-05      675
 pyramid-1.3a5.tar.gz     2012-01-09    2,369
 pyramid-1.3a6.tar.gz     2012-01-20    1,917
---------------------------------------------
Package `pyramid` has been downloaded 97,222 times!

Enjoy the new release.

New Plone themes are:

• HTML/CSS/JavaScript/images (Sound familiar? You don’t need to know any Plone to understand this part.)
• A set of rules that map your HTML/CSS/JavaScript/images to Plone content. (This part is Diazo-specific; and while Diazo is not part of Plone, integration is provided by a Python package called plone.app.theming.)

The Diazo rules can get complex, but it’s very easy to get your first rule done and working and then off you go. You may find this new tutorial on pythonpackages.com. Enjoy, and get busy theming Plone.

It was fun answering in 2009, so I thought I’d play again.

What’s the coolest Python application, framework or library you have discovered in 2011?

Pyramid. In early 2011 I released vanity, which later became pythonpackages.com (more or less). I used building the site as an excuse to learn Pyramid, and it was well worth it.

What new programming technique did you learn in 2011?

I learned to not reinvent the wheel, and to collaborate with others on solving hard problems (which in and of itself is hard). Long story short, I took a little bit of heat within the Plone CMS community for creating a “one off” migration script:

• http://pythonpackages.com/info/parse2plone

I’ve since learned my lesson and have been collaborating meaningfully with Dylan Jay on the more widely-supported mr.migrator and friends.

What’s the name of the open source project you contributed to the most in 2011? What did you do?

I contributed to the Plone project quite a bit (as I have been doing for years now), but I also fell in love with Mozilla (and their mission) and began working on various things, including a virtual machine to help with the development of support.mozilla.com:

• https://github.com/aclark4life/kitsune-vagrant

What was the Python blog or website you read the most in 2011?

Hacker news. I have the start-up bug.

What are the three top things you want to learn in 2012?

JavaScript, JavaScript, and JavaScript. I first heard the term 15 years ago and have been avoiding it ever since. Time to jump in.

What is the top software, application or library you wish someone would write in 2012?

I hope someone (me) writes a “secret” feature for pythonpackages.com and launches it in first quarter of 2012; and I hope folks find it useful.

I have this overwhelming desire to know how many times my favorite Python packages have been downloaded. Don’t you? If so, look no further than pythonpackages.com:

Count downloads

Where you can enter a package like Django and find out:

Count favorites

Or click on Count favorites to reveal how many times it has been favorited:

Find trash

Or even poke some good natured fun at it, for its failure to provide adequate package metadata:

Discuss packages

You can even discuss packages on pythonpackages.com:

Recent activity

But wait there is more! You can also browse the latest changelog activity from PyPi right on the site:

Package info

Lastly, if you get bored counting package downloads and favorites you can look at the actual package metadata.

Conclusion

Sound interesting? Or utterly silly, perhaps? I think pythonpackages.com is the best of both. It started off as a front end to the command line utility vanity, but has taken on a life of it’s own.

After a small run on Python reddit and Hacker News, I’m looking forward to seeing what Python Planet folks think. So please check out pythonpackages.com and let me know. You can leave comments about the site here:

• http://pythonpackages.com/about

Or open a new ticket here:

• https://bitbucket.org/pythonpackages/pythonpackages.com/issues/new

I am planning to do some work on the plone.org server and website by the end of the year, including:

• Upgrade to the latest 4.2.x series
• Switch from xdv to plone.app.theming
• Reduce the number of NGINXes running on the server by 1 (we currently have an nginx doing the xdv transform)
• Repackage the current plone.org theme as plonetheme.ploneorg
• Clean up the server
• Separate vendor package configs from buildout generated configs
• Remove archived files
• Prune tickets on dev.plone.org

I’ve done some of this work already, earlier this year:

• https://github.com/plone/Products.PloneOrg/commits/4.1-compat

Pitch

To ensure it gets done by the end of the year, I would like to add this project to my calendar as paid work. So if you are able to help out by donating some portion of the goal, I would appreciate it. Please use the chip-in below to contribute to this effort.

This just in from the production department: use Varnish. (And please forgive the heavily meme-laden approach to describing these techniques .)

Cache ALL the hosts

Our ability to use Varnish in production is no secret by now, or at least it shouldn’t be. What is often less clear is exactly how to use it. One way I like[1], is to run Varnish on your public IP port 80 and make Apache listen on your private IP port 80. Then proxy from Varnish to Apache and enjoy easy caching goodness on all your virtual hosts in Apache.

Configuration

This should require less than five minutes of down time to implement. First, configure the appropriate settings. (Well, first install Apache and Varnish if you haven’t already: `aptitude install varnish apache2` on Ubuntu Linux[0].)

Varnish

To modify the listen IP address and port, we typically edit a file like /etc/default/varnish (in Ubuntu). However you do it, configure the equivalent of the following on your system:

DAEMON_OPTS="-a 174.143.252.11:80 \
             -T localhost:6082 \
             -f /etc/varnish/default.vcl \
             -s malloc,256m"

This environment variable is then passed to varnishd on the command line. Next, pass traffic to Apache like so (in /etc/varnish/default.vcl on Ubuntu):

backend default {
 .host = "127.0.0.1";
 .port = "80";
 }

Now on to Apache.

Please note that the syntax above is for Varnish 3.x and the syntax has (annoyingly) changed from 2.x to 3.x.

Apache

The Apache part is a bit simpler. You just need to change the listen port (on Ubuntu this is done in /etc/apache2/ports.conf), typically from something like:

Listen *:80

to:

Listen 127.0.0.1:80

Restart ALL the services

Now restart both services. If all goes well you shouldn’t notice any difference, except better performance, and when you make a website change and need to clear the cache[2]. For this, I rely on telnetting to the varnish port and issuing the `ban.url` command (formerly `url.purge` in 2.x):

$ telnet localhost 6082
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
200 205     
-----------------------------
Varnish Cache CLI 1.0
-----------------------------
Linux,2.6.35.4-rscloud,x86_64,-smalloc,-smalloc,-hcritbit

Type 'help' for command list.
Type 'quit' to close CLI session.

ban.url /
200 0

Cache ALL the disks

This site has Varnish and Apache configured as described in this article. It also has disk caching in Apache enabled, thanks to Elizabeth Leddy’s article:

• http://plonechix.blogspot.com/2011/08/10-minute-caching-with-apache.html

As a result, it’s PEPPY AS THE DICKENS™ on a 512MB “slice” (Cloud server) from Rackspace Cloud. And now you know yet another “Plone secret”. Now go make your Plone sites faster, and let me know how it goes in the comments section below.

Notes

[0] Using the latest distribution, “oneric”.

[1] I first saw this technique at NASA when NASA Science was powered by Plone; I found it odd at the time but years later it makes perfect sense.

[2] Ideally you’d configure this in p.a.caching, but I’ve not been able to stomach this yet.

For almost as long as I have been involved in the Plone project, I’ve been interested in Plone’s role in the Python ecosystem. Today as I look across the current landscape, I’m proud to announce a new milestone the Plone community can take pride in. But the change didn’t happen overnight, so first let’s take a quick look back.

PyPI support in PSC

Circa 2008, Tarek Ziadé and others finished the PyPI integration branch of PloneSoftwareCenter started by Sidnei Da Silva 2 years before.

This was a big step towards the “Pythonification” of Plone, mainly because Plone packages could now be published on plone.org in the same way Python packages are published to the Python Package Index: via “python setup.py upload”. In fact if you are publishing Plone packages, it is recommended that you release to both PyPI and plone.org simultaneously. This ensures your package is visible to the most number of potential users. And you can use mkrelease to automate the process.

Collective docs at RTD

Just when you thought things couldn’t get more exciting in Python land, along came readthedocs.org (a Django Dash production, IIRC). And to join the fun in July 2011, I moved Mikko Ohtamaa’s awesome Plone Community Managed Developer Manual to the Github collective and configured the RTD service hook.

This means that whenever anyone commits a change to the developer manual, within a few minutes a new Sphinx build gets published. It was so easy to set this up, I wish I had done it sooner.

So you can that see over the years, we have been working ourselves into a frenzy of Python goodness! And last week, it got even better.

Introducing: Plone packages!

Thanks to the good folks at Cartwheel Web, makers of the fine Open Comparison service, Plone now has its very own grid comparison website! (Djangopackages.com was first, followed by pyramid.opencomparison.org.)

This is a site where folks can add packages hosted elsewhere (e.g. svn, github, pypi), vote on them, and add grid comparisons to compare similarly-featured add-ons, frameworks, and other related software. Grids are the killer feature of this site, and personally I’ve been waiting to use them to answer questions like:

Q: What’s the best blogging add-on for Plone?

A: http://plone.opencomparison.org/grids/g/weblogs/

The content is entirely user driven (TTW only) so please head over to plone.opencomparison.org, login with your github ID, and start adding packages! And while you are at it, please report any issues you find here: https://github.com/opencomparison/opencomparison/issues. Lastly, let’s all tweet a big thanks to @pydanny and @audreyr for their hard work and generosity!

Warm and fluffy history

Pillow 1.0 was released on 2010-07-31, a little over a year later came Pillow 1.7.5 which features:

• Many packaging fixes (mostly to support dependency libraries)
• Windows binaries
• Over 13K downloads

Pillow is used in Plone’s UnifiedInstaller, and I know folks from the Django community are using it.

Future creature comforts

All in all, a mild success. So what’s next? More of the same. To quote myself from a recent Stack Overflow answer:

If PIL ever does exactly what Pillow does, then the fork will die. Until that happens, we have Pillow.

Bottom line accommodations

This may never happen of course, which is part of the reason for the fork. You can’t force someone to maintain software for you. But in the open source world you can grab the bull by the horns (take the lead) and try to make things better. Even better: if your project and community are healthy and functioning properly, your success or failure will be determined by the folks in the community (as it should be). In this case, the Plone and Python communities have spoken clearly.