Sunday, November 24, 2013

No more pdb.set_trace() committed: git pre-commit hooks

After 1, 2, ... 5 times it happens, you must find a way to solve the problem.
I'm talking of committing to your git repository a pdb.set_trace() you forgot to remove.

What is really nice of git (something missing in SVN for what I know) is the support to two type of hooks: client side and server side.
While server side hooks are complex and triggered when you push to the repository, client side hooks are simpler and under your control.

So, I can use client side git hook to solve the pdb problem?

Git hooks are executable file (of any kind) inside the .git/hooks directory of your repository. Those files must have a precise name, that match the action they capture (you can find a complete list in the Git Hooks reference).
In our case: we need to use an executable named pre-commit.

As you probably noticed, hooks are inside the repository. But can I have a centralized database of hooks that will be replied in every repository?
Yes, you must use Git templates.
To quick set you global templates, see this nice article: "Create a global git commit hook".
For impatiens, in two lines of code:

$ git config --global init.templatedir '~/.git-templates'
$ mkdir -p ~/.git-templates/hooks

Now put the executable file named pre-commit inside this new directory. After that, when you create a new repository, hooks inside the template directory will be replicated inside the repository.

The pdb commit hook is a well know problem, already solved. The best reference I found is in the article "Tips for using a git pre-commit hook", that use a simple bash script.

I simple changed the author idea a little, because I don't want to block commit when a commented pdb is found in the code (bugged! See next section!):

git diff --cached --name-only | \
    grep -E $FILES_PATTERN | \
    GREP_COLOR='4;5;37;41' xargs grep --color --with-filename -n \
    -e $FORBIDDEN_PATTERN && echo 'COMMIT REJECTED Found "pdb.set_trace()" references. Please remove them before commiting' && exit 1

There are some other info you'd like to know about client side hooks:
  • You can ignore the hook for a single commit (git commit --no-verify)
  • Only a single hook can exits in a repository. This is a limit, however you can find workarounds.

EDIT (21 December)

Normally I don't modify old articles, but I found a bug in the solution above. The problem: grep command return non-zero status code when it found no match.

Here another working solution (I'm not a Linux bash ninja... any other cleaner suggestion will be accepted!):

git diff --cached --name-only | \
    grep -E $FILES_PATTERN | \
    GREP_COLOR='4;5;37;41' xargs grep --color --with-filename -n \
    -e $FORBIDDEN_PATTERN && echo 'COMMIT REJECTED Found "pdb.set_trace()" references. Please remove them before commiting'
[ $RETVAL -eq 1 ] && exit 0

Saturday, November 16, 2013

Dive into HTML5 Canvas

In last weeks I read a book about the new canvas element of HTML 5: HTML 5 Canvas, by Steve and Jeff Fulton
I don't want to review the book itself (just two word: is ok) but reading it lead me think about "how Canvas can change the Web in future".

First of all: I found all the HTML 5 Canvas feature interesting, but while reading the book I felt a sort of deja-vu. Every time I run one of the given example about drawing and painting, it was like I had already see it on a Web browser...
No, I'm not talking of Flash (I never had any experience with that) but about Java Applet! Yes, I really said "Java Applet"!
When I started Web development 10 years ago applet was "cool" because you were able to do a lot of stuff impossible to to with pure HTML + JavaScript. But you already know that applet failed.
Now using canvas you can do (again) a lot of drawing work, but still: how can this be useful to Web users?

HTML 5 Canvas: The Good Part

I was not able to find every answers myself, so I asked to Twitter and I get back some useful feedback..
  • Videogames development. A lot of this. In my opinion this is great! I don't have time for developing videogames anymore, but is still a discipline I like to follow. A lot of new JavaScript frameworks for game development are coming out, thanks to canvas.
  • Graphs and Histograms. We have a lot of powerful graph generator for server side languages, that can create images on the fly, but we can now stop query a remote server for that (remember that HTML 5 means also "bein offline").
  • File handling and preview. As HTML 5 is able to manage files, and canvas can also manage some type of media like images, audio and video (see below), plugins like jQuery File Upload are now possible.
I get some more good examples, but in the meantime I continued reading the book, and I found new super-cool answers myself! So: keep reading.

HTML 5 Canvas: The Super Fancy Cool Part

First of all: video!
I don't want to simply talk about HTML 5 video support or how you can control video with JavaScript, but what you can do with video and canvas.
This can be sum in a simple sentence: canvas can handle a video like it were an image. What does it means? You can draw the current video frame (taken from a standard HTML video element) inside a canvas; write the current video frame every 20 milliseconds and you are really playing the video inside it.

Why is this cool? I can already see a video element inside my HTML... so?
The great part is that you can draw a video frame inside the canvas then draw other stuff on the top of it! You can add comments and images on the running video! Wow!

And now the best part: HTML Media Capture API.
A lot of browser already support this new technology that make possible from JavaScript to access native webcam (and microphone) of the user's device. And how JavaScript can use this privileged access is not putting it into canvas?
And after that? Can I upload my work to a server? Yes!

Playing with this new toy I spent some times developing a new Plone add-on: collective.takeaportrait. This sum all the stuff I learned about media capture and video manipulation:
  • If the browser support getUserMedia call, a new button appear
  • The button will open an overlay where the webcam output is displayed
  • A viewfinder with the standard Plone ratio for user's portrait and a countdown are drawn over the streaming video
  • Use can save a photo and send it to the server for replacing the current portrait

This is something that you already see somewhere, some social networks (for what I remember, Facebook for sure) already give that chance to users, but it's really amazing to see how few lines of JavaScript code can raise the usability of your site!

Now the bad news (also reported in the Fulton & Fulton book): there's no support for those features inside mobile devices right now. Really sad.

Saturday, October 19, 2013

Reusable jQuery plugins with Bower

Inspired by the new article of Maurizio Lupo named "Reusable javascript modules with Bower" and by a recent discussion we had at work about modern front-end development (mainly focused on the Plone world), today I took some minuted to test bower.

To quickly explain bower to a Python developer I can say that it's "the Distutils of the JavaScript world".
I'm not a front-end developer because I think that "knowing how to do some JavaScript" doesn't mean being a front-end guy, however I like the direction where JavaScript is going. But note: take the rest of this article like it is: a "Bower for dummies" note!

In my last article I quickly introduced the jQuery Plugin site way of keeping update it's database: a simple JSON file. Bower is doing the same for populating its component registry site.
So an "official" jQuery plugin can be also a bower component.

Step by step

First of all you need to install the node package manager (npn), and for a MacOS user is really simple (same for Linux guys):

    $ sudo port install npm

After that you can install bower.

    $ npm install -g bower

No we'll go back to our jQuery plugin.
First of all you need the bower.json file, but instead of writing it manually, let's simply type...

    $ bower init

... and answer to the questions. After that you can go inside the file and add some other missing stuff by looking and the available syntax.

This is a possible result:
  "name": "waria-checkbox",
  "version": "0.2.2",
  "homepage": "",
  "description": "jQuery WAI ARIA Compatible Checkbox Plugin",
  "main": "jquery.waria-checkbox.js",
  "keywords": [
  "authors": [
    "Luca Fabbri <>"
  "license": "MIT",
  "ignore": [
  "dependencies": {
    "jquery": ">=1.7"

The bower.json file is really similar to the file needed by the jQuery plugin site, but it's not the same (a boring task: you must keep updated both files).
As we are focused on jQuery plugin, note that jQuery (that can be installed using bower) is defined in the "dependencies" section.

Finally you need to create a new git tag:

    $ git commit -am "Now is possible to install the plugin by using bower"
    $ git tag -a 0.2.2 -m "Tagging version 0.2.2"
    $ git push --all
    $ git push --tags

Last step: register the plugin onto the bower registry:

$ bower register waria-checkbox


    $ bower install waria-checkbox
    bower cached         git://
    bower validate       0.2.2 against git://
    bower cached         git://
    bower validate       2.0.3 against git://>=1.7
    bower install        waria-checkbox#0.2.2
    bower install        jquery#2.0.3
    waria-checkbox#0.2.2 bower_components/waria-checkbox
    └── jquery#2.0.3
    jquery#2.0.3 bower_components/jquery

Now inside the bower_components folder we have both the plugin and jQuery.

Sunday, September 1, 2013

Extending jQuery selectors and facing conflicts with querySelectorAll

You know that is possible to extend the jQuery selector capabilities?
Just type...

$.expr[':'].foo = function(element) {
... and this function will be called when :foo selector is used.
Nothing new on this side.

Recently I started working on a new jQuery plugin and to make things simpler I needed to find a way to override an existing jQuery selector. Again: nothing new; some years ago I found that the method described in the article above can be also use for override, and not only for extend.
To make things more testable I decided to move this behavior into another (separated) jQuery plugin, which is the argument of this post.

When I used this method again nowadays I found unexpected results.
I was looking a way to change the way :checked and :checkbox selectors work, so I defined...

$.expr[':'].checkbox = function(element) {
$.expr[':'].checked = function(element) {
This is what I found:
  • :checkbox was working as expected
  • :checked was not working as expected
In facts, the :checked selector was only working when using it inside a .filter() call.
I'm not a jQuery core expert, I never looked at it's code very much, but this time I needed to investigate my problem. Also: I need to make this work on jQuery 1.7 and more modern 1.10 version, and codes are quite different.

Here what I found: both jQuery versions contains a method for capturing the :checked selector, but it's only called when you call filter() (so here my override attempt works as expected). For normal selectors jQuery now heavily relies on native querySelectorAll API for every browser that is supporting it.

This is the core of the problem: the :checkbox selector is a non-standard ones (not defined by any CSS specifications) while :checked is a know CSS selector. So browsers that support the :checked selector for querySelectorAll are calling this native API.

This is someway hilarious! While querySelectorAll are making our browser (and jQuery usage) faster, it's lowering jQuery extensions capabilities.

I found no smart way to change how querySelectorAll works (and probably there's no way at all, I think we are at C compiled code level here).
The trick I used is to disable the native querySelectorAll when the selector contains :checked, and in that case call my jQuery version instead.

    var pattern = /\:checked/;
    if (document.querySelectorAll !== 'undefined') {
        // need to partially disable querySelectorAll for :checked
        document.nativeQuerySelectorAll = document.querySelectorAll;
        document.querySelectorAll = function(selector) {
            if (pattern.test(selector)) {
                throw('Native ":checked" selector disabled')
            return this.nativeQuerySelectorAll(selector);

The first step is to disable (keeping a "backup") querySelectorAll, changing it with a custom function. Then all I need to do is check if the :checked selector is used somewhere in the query. If not: just call the backed up querySelectorAll, but if it's called somewhere I simply need to raise an exception.

This is the interesting part I found looking at the jQuery source: jQuery core try to use querySelectorAll every time is possible, switching to internal JavaScript code only when it's not supported. In this way I'm simulating the fact that my modern browser is not supporting :checked selector for querySelectorAll.


I found this experiment interesting, but there's some things I don't like:
  • I'm disabling the native querySelectorAll usage also when it's basic features are enough (i.e: if I really need to load only checked checkboxes and not other fancy stuff)
  • I'm wrapping every querySelectorAll calls inside the selector check, making all calls to querySelectorAll slower
  • I want only to extend jQuery here, but I'm also disabling all native querySelectorAll calls if the query containes ":checked"
  • I'm changing how JavaScript works. This is calling me back to times when I used prototype.js (that I never liked)
Any suggestion for a better code way are welcome!
This is the result of the experiment: jQuery WAI ARIA Compatible Checkbox Plugin.


Apart the problem described in this article, I let's spend some words on the "new" jQuery Plugin Site. Last time I published a jQuery plugin (lot of time ago) this site was a total mess: you need to authenticate, upload source tarball, write documentation on a wiki-like page, ...

I really liked how it works now: if you want to publish a plugin, just put it on GitHub and configure a commit hook! Simple and amazing!

Wednesday, May 1, 2013

How to make your Plone add-on products uninstall cleanly

This article contains practical informations, but also some personal opinions, about Plone install and uninstall tools. You are warned!


Recently I worked on lot of Plone migration from Plone 3.x to versions 4.2.

Is Plone migration to newer version a difficult task? Not on its own. The Plone CMS is distributed with an internal migration procedure that simply works! What can make the difference are add-ons products, that sometimes transform this one-click operation something difficult.

When you have migrate a Plone site you'll probably find some annoying issues:
  • add-ons that must be removed because never used by customer
  • add-ons that must be removed because newer products are giving the same features
  • add-ons that must be removed because they are totally unmaintained now, and you must migrate to something else
Whatever is your motivation, sometimes Plone product removal can be a complex task and not for a Plone internal technological problem but because of programmers laziness.

Once Upon a Time: before Generic Setup and ZCA

When I started working with Plone making a Plone product "installable" was a Python task: add-ons contained an Extension folder with an (or, uppercase) file and an install function inside.
This function did all the magic stuff (registering types, new properties, adding configuration stuff, ...).
The problem with that approach was that every Plone programmer was using a "personal style" for doing same things: obviously low-level APIs used were the same so, more or less, different product's install procedures were similar, but not exactly the same.

If you are curios, for historical reason take a look at an example: the oldest version of Poi install code.

Generic setup

Generic Setup was introduced with Plone 2.5. For the first time a (initially limited) set of common operation were translated to an XML format: for example you are now able to describe the new-type configuration using an XML file instead of writing Python.

Another example: a newer (still very old) version of Poi code.

Nowadays the main problem of Generic Setup is the lack of a central documentation (however the Plone Developer Manual is going in the right direction), but looking at the Plone source you can learn a lot.
Personally, I always look at the Products.CMFPlone source when I don't know/remember something but this can't help you for learning uninstallation.


Meanwhile Plone 2.5 introduced also new Zope 3 features (everyone were talking of Five).
This gave us a lot of really cool things we have today (browser views, viewlets, events, utility, ...)... and new "persistent problems" to face (portlets, persistent utilities, marker interfaces, ...).

This was an important moment for Plone features, but at the same time the always-present ZMI started losing it's superpowers. What I mean: before Plone 2.5/Plone 3 the Zope Management Interface was a very powerful tool where you was able to fix things and clean garbage; nowadays ZMI contains more or less the same features of years ago.

Please note that using ZMI had never been a good way of fixing things or configuring your Plone site: a good Plone add-ons must configure itself when you install it, and fully remove itself on uninstall.
But usage of ZMI helped armies of administrators to fix garbage left behind by non-polite add-ons, or quick fix stuff directly on production environment. Sadly this is not always possible today.

Why uninstall is important?

If you are an add-ons developer and you want to release your code to the community you must live with the possibility that your product will not fit the user's need and he will uninstall it. You must be sure that it will be a simple operation that don't leave problems behind.
If you don't care about this, you will became quickly the Bad Guy/Company and no-one will ever use your work in future. And you will also lose all your friends. And wife. And Cat.

Another fact: a lot of people still "try" products on production sites.
This is happening more frequently of what you can imagine; maybe they started adding stuff to a test or staging site but later this became a production site just because it's now filled by a lot of production data.
We can't simply say "Hey! You didn't read how to evaluate a Plone product guide"! This is a puerile excuse.

Tools for make products uninstall cleanly are around us: just use them.

Today: Generic Setup age


Before starts talking about product's removal let's ask ourself "how installation works nowadays"?

The portal_quickinstaller (the Plone tool that manage installations) is now Generic Setup compatible: this mean we don't need the magical file anymore.
How? Let me say that there's a well know standard for creating installation Generic Setup profiles: a profile named "default". Probably 99% of add-ons today use this common patter (you can unluckily find some exception, but I think is really uncommon).

So you can think that portal_quickinstaller is looking for a profile named "default", but this is false. The algorithm is more... "simple": it looks for the first profile found (alphabetically sorted) and use this an "installation profile".
When relying on this automatic choice, this will trigger a log warning message:
INFO CMFQuickInstallerTool Multiple extension profiles found for product Used profile:

This lead to some considerations:
  • you need to be sure that the first profile found is really the installation profile (so if you want to keep "default" as its name, and I suggest you to do so, you must handle carefully other profiles names).
  • you can't install two profiles from the portal_quickinstaller tool
About the second limitation: Plone spent a lot of energy in last years to free users from the need to access ZMI. When I talk of "using portal_quickinstaller" I mean the Plone UI access to the portal_quickinstaller: the "Add-ons" control panel.

The Add-ons control panel

What is not possible today is load from Plone UI additional setup profiles. This is OK for most products (commonly you simply need an installation profile) but there's some add-ons that bring additional configuration profiles, and for using them they explicitly document the need of still going to ZMI, in the portal_setup tool... not really user friendly.

Trick for having multiple installation profiles for the same product

Ok... I want to be honest.
There's a dirty way for having multiple profiles registered by the same product. We discovered this "feature" recently but I'm not sure if this is something good to teach, so use it at your own risk!

It seems that the portal_quickinstaller tool is loading the first-found profile defined in a Python module, not in a whole product.
The trick: define another Generic Setup profile in a different Python module.

Commonly profiles are defined in the Plone project root, directly inside the configure.zcml file or inside a profiles.zcml file. Let say this file is inside the module.

So you can include another Python module submodule:

      description="Installs the package"

  <include package=".foo" />
Then inside the submodule you can register the new profile (with another configure.zcml file).

      title=" foo"
      description="Foo profile"
In this way you'll see another "product" in your portal_quickinstaller tool.


When you start facing uninstallation, things are more confused.
Plone automatically supports uninstallation of products (this was true also in the pre-Generic-Setup era) performing automatically a lot of stuff we will see later.

The Add-ons control panel

But often, for performing a really good cleanup, you'll need an uninstallation profile.
The Plone standard for this is translated to distribute with your products a profile named "uninstall".
You can expect that portal_quickinstaller tool is looking for some specific profile name (or choosing it with some other fancy algorithm). No!
For being able to perform an explicit uninstallation you must still rely onto the old-friend file.

So: a good Plone add-on today must still be distributed with an Extension folder, with an file.
Inside this file we can skip the "install" function (tip: you can still use it for installing some Generic Setup profile that isn't the first found, or checking the Plone version and keep old Plone compatibility alternative profile) but you must provide the "uninstall" function.
This is how an file can looks like:
def uninstall(portal):
    setup_tool = portal.portal_setup

A lost old friend: "reinstall" button

What you can do from the Plone "Add ons" control panel are mainly the same operations you can do from the ZMI, explicitly accessing the portal_quickinstaller tool. But with some differences.

From Plone UI you can simply install (activate) an add-on and uninstall (deactivate) it, but you can also easily perform a product upgrade to newer version when this is needed.

The Add-ons control panel

Products upgrade are a great tools and, when properly used, they helps your users to update your configurations to a newer version.
However sometimes they can be "fragile":
  • when you upgrade from a very old version of the product.
  • when the product's developer didn't work well and he's not providing you an upgrade step.
The Add-ons control panel
In those situations a well-know patter is to uninstall then reinstall the product: this will remove the product's configuration (remember that, if an uninstall profiles is not provided, only some configuration are automatically removed), then a new configuration is done from scratch.

The ZMI view of portal_quickinstaller is different. It only knows about product install and uninstall, totally ignoring the product's upgrade (for this, if you really want to use ZMI, you need to rely on portal_setup tool).
But when a product in installed, you can find another option that the Plone UI is not providing: the "reinstall" button.

The Add-ons control panel
What "reinstall" can do?
Well... if you don't plan something different it's only a shortcut for uninstall+install in sequence, but it can be more.
The install and uninstall functions inside the file can catch "reinstall" action if you change the code like below:
def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        setup_tool.runAllImportStepsFromProfile('')"Uninstall done")
So: you can perform different things if the user pressed the "Reinstall" button (commonly: you can perform the uninstallation profile only if don't want to reinstall).

Why this is important? And why IMHO the reinstall button can be something good also in Plone control panel?

When an uninstallation is well done (that's the scope of this article), it's destructive as you really want that everything related to your product is removed from the site: you must be 100% sure that the user can safely remove your code from his buildout without giving him problems.

However sometimes:
  • users are upgrading from a very old version of your product, and the internal migration procedure is not working
  • users want to install again your product, means that they want to fully restore original configuration. User's actions, or other products, can mess up you product configuration.
The only thing that a user can do to restore is... reinstall. But doing a reinstall from Plone UI means performing all the super-clueaning-stuff you have prepared, probably destroying all configuration done by the users. Simply repeating the install profile can be better. 

Let's conclude: sometimes I still like to provide a "reinstall" support.

Handle uninstallation with Generic Setup

Keeping in mind that, the main argument here is products uninstall, let me describe all good things (steps) you can do with Generic Setup and how you must handle those steps when you need to uninstall the product.
For this task, I will describe every know Generic Setup import step and what you need to do (if you need to do something) for uninstall.

All the code below is also accessible in an example product on Plone uninstall.


The metadata.xml file keeps care of registering the current profile version of the product (from Plone 3.3 and above this is not the product version) but it's doing more: it can run Generic Setup profile dependencies (that are not necessarily egg dependencies).
Roughly speaking: it install products that are dependencies of the add-on you are installing.

<?xml version="1.0"?>

This is a double-edged weapon and you can't simply put there all your dependencies: it's better to look at the format of the dependency add-ons.
Why? Sometimes Generic Setup steps are destructive (yes, not only the uninstall profile can damage your configuration): they can reset a piece of configuration, ignoring if the configuration was already present in the site and customized by users.

Image handling settingsSome examples.
  1. If you develop an add-on that is compatible with Plone 3 and Plone 4 and you rely on, you must know that this package is not already present on Plone 3: you must add it to your egg dependencies list.
    But adding it also to the Generic Setup dependencies list is not a good idea because running this step will reset the image configuration of the portal (both on Plone 3 and Plone 4).
  2. TinyMCE. if you are developing a TinyMCE & Plone-compatible add-on you can't trigger the TinyMCE Generic Setup configuration on install because this will reset the TinyMCE status of a fresh Plone site.
It this a general plague? No. Some Plone components or add-ons can be putted in the dependencies list safely, but you need to know them (looking at their code).

Two good examples follows.
  1., that you can manually install on Plone 3 (with some version pinning work, but still you can do it). Reinstalling it's Generic Setup profile multiple times will do nothing that can be damage your configuration.
  2. A famous Plone add-ons: PloneFormGen.
    This add-on can be enhanced by other Plone add-ons that could add new fields or adapter. This features enhancement is commonly done through adding new types, registered as FormFolder possible subtypes. This is not done relying only onto the types.xml  Generic Setup step (see below), but also through Python scripting.
    In this way you can safely add PloneFormGen to your dependencies list, and reinstall it multiple times without loosing anything special.
    This is a good example of an add-on that want to keep things simple for integrators.

What is missing?

I already discussed this a couple of times in the Plone-Users mailing list but I didn't find other plonistas that share my idea.
Let see what I don't like of this approach: IMHO the dependency check of Generic Setup is lacking of a way to know if the add-ons is already installed (formally speaking: if the Generic Setup configuration has been already executed).
Sometimes (let me say "very often") what you really want is: install a dependency Generic Setup profile only if the product is not installed.
For doing this today: you must rely on Python.


You don't need to do nothing special to revert changed done by metadata.xml.
After all I said above you can ask yourself "isn't a good attitude to uninstall add-on's dependencies when I remove it?".
Commonly, no. You can't be sure that the one of the dependencies wouldn't be a dependencies of another products.


The actions.xml Generic Setup step is reserved to the configuration of the portal_actions tool you can explore or modify through ZMI. It's a container of CMF Action Categories and CMF Action items, that are used by Plone to draw a lot of interface elements, like links an buttons.

Multiple installations

Repeating the installation of the profile will restore all the actions, as defined in the XML file.


You must do nothing about actions.xml when you uninstall: Plone will automatically remove all your defined actions.
At first glance this seems a good thing ("wow, I don't need any uninstall step for actions!"), but to be honest it isn't: an explicitly requirements of an uninstall actions.xml on uninstall would be a safer solution.
Right now, if you want too keep an action you defined through Generic Setup when you remove a product, you need to manually re-add it after uninstall.

Managing portal_actions through ZMI is very simple, so I don't find any need of such kind of automatism. Also, most of times, orphan portal_actions elements are safe and simple to be removed.

Personal experience: sometimes customers ask to be able to manually handle portal tabs, as portal tabs are seen by the custom as something content related, not theme related.
If you started adding portal tabs using Generic Setup, the you teach to the customer how to manually change portal tabs using ZMI and later you'll provide to the customer a product version, you need to be sure that theme upgrade will not destroy customer's work.

For this reason, for some kinds of actions categories (like portal tabs), I prefer to not define additional actions using actions.xml but I define them manually using ZMI (or some other add-ons).
For this strange behavior I never liked and for take care of this kind of requests, we developed the collective.portaltabs add-on.


This is the step able to register new catalog index and metadata:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Plone Catalog Tool"> 
 <index name="getFoo" meta_type="FieldIndex">
  <indexed_attr value="getFoo"/>
 <column value="getFoo"/>
Until you use it at install time, it works perfectly.
Problems arise when you re-install a product or if you have already an index registered with the same name in your Plone site: in that case use of index will clean your index values.

For this reason I commonly prefer to register only metadata (column), and use Python to register indexes:
    catalog = getToolByName(portal, 'portal_catalog')
    indexes = catalog.indexes()
    # Specify the indexes you want, with ('index_name', 'index_type')
    wanted = (('getFoo', 'FieldIndex'),
              ('works', 'KeywordIndex'),

    indexables = []
    for name, meta_type in wanted:
        if name not in indexes:
            catalog.addIndex(name, meta_type)
  "Added %s for field %s.", meta_type, name)
    # lines below if you want also reindex items when executing it
    if len(indexables) > 0:"Indexing new indexes %s.", ', '.join(indexables))

Multiple installations

Repeating the Generic Setup configuration will register again the index configuration. A side effect of this is the index cleanup. This will force you to perform a reindex.

Instead, metadata columns are kept without loosing any data.


When you uninstall a product, no changes are done to the Plone catalog: you must handle the indexes and columns uninstallation yourself:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Plone Catalog Tool">
 <index name="Creator" meta_type="FieldIndex" remove="True">
  <indexed_attr value="Creator"/>
 <column value="Creator" remove="True"/>
Indexes and metadata left behind will not give you any trouble, however cleaning the catalog from useless elements will quick-up the indexing operations.


This is the place where sometimes Plone can gives big headache.
Component registry is a powerful tool that you can use to register local utilities, adapters and subscribers. This section will continue talking of utilities (that are most common components used there)
After the ZCA utility registration, you can register the utility in the component registry of your site on product install, using Generic Setup and the componentregistry.xml file:
<?xml version="1.0"?>
Syntax above can change if the utility is a named utility and this is often used with a Plone tool registration adding the object attribute (see toolset.xml below for Plone tool info).


What can happen if you don't provide an uninstallation profile?
Things may differ: sometimes you'll get immediately an ugly ZMI error as soon as you remove the package from the buildout:
TypeError: ('object.__new__(FooUtility) is not safe, use Persistence.Persistent.__new__()',
<function _dt_reconstructor at 0x110df1a28>,
(<class ''>, <type 'object'>, None))
Other times the problem stay under the hood for weeks, months... I started this article talking of Plone site migrations to new a version: this is exactly the perfect moment where such kind of problem arises and you find that a very old utility, of a product you badly removed ages ago, now gives you problems.

Can we do something from ZMI? Do you remember when I said that ZCA is a recent technology?
Well... You have a ZMI tab ("Components") on every Plone site, where you can see registered stuff.
Site Components tab
The UI seems editable, but I never had luck using this: on my past tests it never really cleanup things and now (Plone 4.2.4) I get a traceback error.
So: use it for seeing what you have registered, but nothing more.

So: how to uninstall?
First of all a good uninstall step:
<?xml version="1.0"?>
After running this, the "Components" tab view above seems cleaned but still, if you remove the egg from your buildout you get same problems.
This is orrible. Nowadays there's no way of really cleaning components using Generic Setup or ZMI.

So: you need to switch to Python!
There's a great article about how to manually removing local persistent utilities. This document is also a good starting point for removing other persistent stuff like adapters and subscribers.
The core of the approach is usage of getSiteManager() method:
>>> sm = app.Plone.getSiteManager()
>>> sm.utilities
<zope.component.persistentregistry.PersistentAdapterRegistry object at 0x1096b9c80>
>>> sm.adapters
<zope.component.persistentregistry.PersistentAdapterRegistry object at 0x1096bd1b8>
>>> sm.subscribers
<bound method PersistentComponents.subscribers of <PersistentComponents /Plone>>
Now, let's see how to remove our dummy utility registered above:
def _removePersistentUtility(portal):
    sm = portal.getSiteManager()
    util = sm.getUtility(IFooUtility)
    del util
    sm.utilities.unsubscribe((), IFooUtility)
    del sm.utilities.__dict__['_provided'][IFooUtility]"Removed utility")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        _removePersistentUtility(portal)"Uninstall done")
But how can you clean this if you are facing this kind of issue with a product that you haven't developed yourself (the Evil Prone Developer)?
There's a Plone add-ons that is often used in such kind of situation: it is wildcard.fixpersistentutilities. It's performing tasks above giving you a good Zope user interface, making simple finding broken persistent garbage.
Use it you you really need it, but please... don't pretend that your users will use it for uninstalling your product: provide an uninstall procedure yourself!


This is the step where you can register new Plone control panel link to your product's configuration section.
<?xml version="1.0"?>
<object name="portal_controlpanel"
        xmlns:i18n="" i18n:domain="">
    <configlet title="Example configuration"
        action_id="Example" appId=""
        <permission>Manage portal</permission>


The story of control panel uninstallation is full of products that leave broken link behind. The broken link is not something that can brake your site, but it's really annoying to see them around.

Uninstall is simple:
<?xml version="1.0"?>
<object name="portal_controlpanel"
        xmlns:i18n="" i18n:domain="">
    <configlet title="Example configuration"
        action_id="Example" appId=""
        <permission>Manage portal</permission>
Recently (not sure from which Plone version) the control panel can be cleaned also from ZMI, using directly the portal_controlpanel tool.
portal_controlpanel tool But as you seen, providing an automatic uninstall of the control panel link is very easy. Do it!


The tool that takes care of managing all Plone CSS, merging and minimizing them: portal_css.
Every products that want to register a CSS in the tool must provide one or more snippet like this:
<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet title=""
    media="all" rel="stylesheet" rendering="import"
    cacheable="True" compression="safe" cookable="True"
    enabled="1" expression=""/>

 <!-- more follow -->
From now, the portal_css tool will try to load the example.css recourse.


What's happen if the resource is not found, because you removed the product but you didn't clean the tool?
The portal_css tool is configurable manually, and it's take care of warning the user about missing resources, so you can clean manually.
portal_css tool
Automatic cleaning on uninstall is simple:
<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet id="++resource++example.css"
    remove="True" />

factorytool.xml and types.xml

I'll talk about those two kind of Generic Setup steps because they are used at the same time and they have same behaviours. They manage Plone content types.
The factorytool.xml file is configuring the ZMI portal_factory tool and is used only for Archetypes based content types.
The types.xml file (and the types directory and XML sub-elements) are used for registering and managing the content type itself (for both Archetypes and Dexterity based content types) through the portal_types tool.

I don't find useful to provide here some XML snippet because all this stuff is commonly generated automatically for you using paster (or ZopeSkel, or plone.templter, or mr.bob... whatever tool you are using).

Multiple installations

Repeating a type installation will totally restore the starting configuration, cleaning all the administrator manual changes.

Sometimes additional products would try to touch only some part of a type configuration. For example: you need to add a new available view to the Folder type (views are controlled by the view_methods property).
You can use three ways:
  • The Bad Ones. You will register the Folder type from scratch, providing the modified view_methods.
    Not good: you will clear all configuration changes done by the site administrator.
  • The Still Bad Ones. You can provide an empty type definition with just the modified view_methods.
    Not too bad, but you will clean all configuration done by others to the list of views.
  • The Good Ones. Provide only the view_methods (like above) with the single new entry, and adding the 'purge="False"' attribute.
    This will only add your new view.


You don't need to do anything! Sometimes I found  Plone add-ons that provide an uninstall XML for those two steps, but they are ignored.
Those two tools are of the type that automatically handle uninstallation: without doing anything it remove configuration when uninstalling.
Unlike the same magical behavior we saw for the portal_action tool, this time I like this kind of automation (and I've never had problems with it).

What you must note is that you removed a type definition. What happens to existing content of the type that lives in the site?
Well... as soon as you'll remove the type code from the buildout, the will be broken.

So: can be a good idea to automatically crawl the site and remove all content of this type?
I think no... and I don't know any product that is doing this. If your product user uninstalled the product by mistake you can waste a lot of work (well... you can hope that undo support will work)!


This import step manage the portal_javascript tool, for Plone JavaScript resources.
There's nothing I can say about this tool that I didn't reported about the cssregistry.xml import step, so I'll be quick.

The installation process is similar to the ones about style sheets:
<?xml version="1.0"?>
<object name="portal_javascripts" meta_type="JavaScripts Registry">
 <javascript cacheable="True"

 <!-- more follow -->


The ZMI tool can be used for manually removing old registrations:
portal_css tool But clean uninstallation is always simple:
<?xml version="1.0"?>
<object name="portal_javascripts" meta_type="JavaScripts Registry">
 <javascript id="++resource++example.js"


It's not so common, but sometimes additional products need to add new Plone users data in the portal_memberdata tool. This tool can also be accessed and modified manually through ZMI.
Recent Plone versions switched to module, so the importance of this tool is decreased.

Registering new properties is simple:
<?xml version="1.0"?>
<object name="portal_memberdata">
 <property name="foo" type="string"></property>
 <!-- other can follow -->


There's no Generic Setup support, so you need to use Python.
def _removeUserProperty(portal):
    portal.portal_memberdata.manage_delProperties(['foo'])"User property removed")

def uninstall(portal, reinstall=False):
    if not reinstall:


This Generic Setup import step is used to configure the portal_atct (ATContentTypes) tool, related to some features of the current Plone content types family.
It can be splitted in two kind of configurations.

The first section in related to collections criteria definitions but only for old Plone collection (that are disabled starting from Plone 4.2, which now rely onto

The other section configure some general property like image_types, folder_types, album_batch_size, album_image_scale and single_image_scale. To be honest I'm not sure if they are still used nowadays.

So, let's focus on collections only.

The strange thing about this tool is that you can't manage collections configuration using ZMI but only from Plone UI, so you can manually configure (changing properties, or disable) collection criteria or columns (used for the "collection view") calling the "/portal_atct/atct_manageTopicIndex" view.
portal_css tool What you can't do is to add new new criteria or columns, or delete old ones.

To add new criteria or columns you need a simple Generic Setup XML:
<?xml version="1.0"?>
<atcttool xmlns:i18n="">
  <index name="getFoo"
     description="A really dumb field" enabled="True"
  <!-- more can follow -->
  <metadata name="getFoo"
     description="A really dumb field"
     enabled="True" friendlyName="Foo" />
  <!-- more can follow -->


Old style collection criteria are based on catalog indexes. If the catalog index used by the criteria disappear, the "/criterion_edit_form" view will be broken.
Sadly, as sayd above, you can't cleanup things and delete the criteria from the configuration form. Wow! Now all your old-style collections are broken!
What you can do is to disable the criteria; in this way collections would work again, but for a real cleanup, you must provide an uninstall step:
<?xml version="1.0"?>
<atcttool xmlns:i18n="">
  <index name="getFoo"
  <!-- more can follow -->
  <metadata name="getFoo"
  <!-- more can follow -->


There's a lot of things we can say about portlets in Plone but I already dealt with this argument in another article:  "Plone portlets are not enemies... just rude friends". However the time of this old article something's changed.

Using the portlets.xml file you can register new portlet manager and portlet.
Registering new portlet manager is not common, while registering new portlet is a natural part of Plone add-ons world.
<?xml version="1.0"?>
<portlets xmlns:i18n="">
     title="Example portlet"
     description="An example portlet"
Apart the portlet registration (so: seeing the new portlet available in the "add new portlet menu") there's also the persistent object you will create on the database (the portlet assignment).
This is, in fact, the most tricky part.

Last note: again, portlets are a recent Plone addition so you can't do absolutely anything from the ZMI for using/fixing/cleaning them.


As already told in the other article, uninstalling the product that register the new portlet is enough for preventing Plone to raise errors and making new portlet to disappear from the add menu, but the real problem are the persistent instance created, that must be deleted before.
Good news: there are recent changes about it: ticket #7375 has been closed, so sooner will be possible deleting broken portlets from Plone (EDIT: seem already possible in Plone 4.2.5). There are also other experimental products.

While writing this article, I'm also looking at all importexport modules of Generic Setup world. Looking at portlet code you can see that an uninstall procedure is available (see the exportimport/ source).
A docstring says:
Remove a portlet definition using the 'remove' attribute so that it can no longer be added via @@manage-portlets. This does not remove any assignments:
<portlet remove="True" addview="portlets.Calendar"/>
To be honest, I never used a this remove attribute and I didn't get any errors in years, so probably this can be used to removed/unregister existing portlets you don't want to show to your users (not bad if you plan to substitute an existing portlet with a new ones).

Looking at the same code, there's also a purge feature. The docstring of the corresponding method says:
Unregister all portlet managers and portlet types, and remove portlets assigned to the site root
Great! This will introduce the last problem of uninstalling products with portlets: although is now possible to delete broken portlets, I think that a good uninstall must take care of cleaning portlets assignments left behind in the site.

The purge feature above seems a good starting point, however is not enough; removing portlets in the site root can be only the edge of the iceberg: portlets in Plone can be added everywhere, on every content and there's not simple way to know where.
The only thing you can for cleaning all contents is to traverse all your site tree and check every content... this is not something you can do when uninstalling: it can take hours and slow down your site.
What is missing? The problem is that Plone is not memoizing where you added new portlets (probably a marker interface would be enough), so you can't quickly identify those contents.


The Plone portal_properties tool is a well-know, robust component of the Plone world. It's a composition of "Plone Property Sheet" objects, that store a set of configuration (of primitive types, like strings, integers float, boolean, lists, ...). portal_css tool In the past this was the only place where you could add configuration for your products, and the Plone core itself was using this. Also today, a lot of configuration are still there.

What I always liked of properties sheets is that you easily managing them from ZMI, and cleaning garbage is simple. Also, APIs for using them are really simple. portal_css tool However I need to say that Plone configuration are now quickly moving to the Plone registry (see the registry.xml section below).

Registering new properties or a while new property sheets is really simple:
<?xml version="1.0"?>
<object name="portal_properties">
 <object name="navtree_properties" meta_type="Plone Property Sheet">
  <property name="metaTypesNotToList" type="lines" purge="False">
   <element value="ExampleType"/>
 <object name="example_properties">
  <property name="title">totally useless properties</property>
  <property name="foo" type="boolean">True</property>
  <property name="bar" type="string">xxx</property>
You can use this tool for changing default Plone configuration (note the use of purge option above, for not cleaning the whole metaTypesNotToList property) and registering your new property sheet.

Property sheets are doing nothing by their own: your product must use them fro something.

Multiple installations

Multiple installation of a propertiestool.xml profile will restore every time the configuration. If the site administrator had manually changed something in the configuration, or a 3rd part product did it, you are probably deleting those configurations.

For this reason I commonly prefer using Generic Setup for adding new property sheets, but still using Python for adding properties.
In this way you can check if the property already exist, and skip it:
    dict(name='foo', type_='boolean', value=True),
    dict(name='bar', type_='string', value='xxx'),

def registerProperties(context):
    ptool = getToolByName(context, 'portal_properties')
    props = ptool.example_properties
    for prop in _PROPERTIES:
        if not props.hasProperty(prop['name']):
            props.manage_addProperty(prop['name'], prop['value'], prop['type_'])
  "Added missing %s property" % prop['name'])
  "Property %s already present. Skipping" % prop['name'])

def setupVarious(context):
    if context.readDataFile('example.gs_various.txt') is None:

    portal = context.getSite()
The problem example used at the beginning of this article is, in facts, related to property sheets registration. 


There's no Generic Setup support for cleaning portal_properties tool, you must provide your Python code. However property sheets left behind are really simple to clean using ZMI (and they don't create any issues if you don't remove them).

Real question is: what to clean?
If you added a totally new Plone Property Sheet, answer is simple: let's remove it.
def _removeProperties(portal):
    portal.portal_properties.manage_delObjects(['example_properties'])"Property sheet removed")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        _removeProperties(portal)"Uninstall done")
If you changes some other Plone default properties you must be careful and be sure of what you are doing.


The rolemap.xml file can be used to customize new permissions defined by your product, or change some default site permissions.


There's no Generic Setup uninstall support for this, and in both cases introduced above, this is probably good.

If you are removing a product that add a new custom permission, the new defined permission will not be used anymore (and probably the real question will be "how to delete a Plone custom permission?").

In the case you have customized a 3rd party permission, you could provide a rolemap.xml file that will try to restore the old permission configuration.
An example: you create a policy product that gave the "Review portal content" permission also to Editor role. On uninstall you can provide a new permission map that restore the starting Plone configuration.
However this is not a 100% secure choice and the reason is always the same: can you be sure that no 3dr party Plone products or administrator manual changes touched this permission? You can't.
Unluckily the rolemap.xml file doesn't provide a 'purge="False"' feature (for editing only the permission for a single role).

Again: evaluate what to do.


Probably 50% of garbage you can see in a Plone site after removing a product came from this import step!
The portal_skins tool is the home for old style CMF technology and old-style Plone themes but is still used today for a lot of Plone feature and add-on products.

From this tool you can define new plone skins and modify layers set of existing skins.

To register a new layer and new skin (common style for Plone theme add-ons):
<?xml version="1.0"?>
<object name="portal_skins">

 <object name="example-gs"
    meta_type="Filesystem Directory View"
 <skin-path name="example-theme" based-on="Sunburst Theme">
  <layer name="example-gs"
For register only a new layer for all existing skins (common style for classic add-ons that need CMF skins resources):
<?xml version="1.0"?>
<object name="portal_skins">
 <object name="example-gs"
    meta_type="Filesystem Directory View"
 <skin-path name="*">
  <layer name="example-gs"
Both registration are splitted in two section: the first one is registering a filesystem directory as a possible additional layer, but this will not be used without the second section.


Items inside portal_skin tool are easy to manage and clean through ZMI and broken stuff left behind by lazy products will not give you any problem.
However layers and skins removal is simple.

First of all: you only need to care about unregister the skin-path. My tests reported that the filesystem directory registration is automatically removed when you uninstall the product.

For totally remove a theme:
<?xml version="1.0"?>
<object name="portal_skins">
 <skin-path name="example-theme" based-on="Sunburst Theme" remove="True">
  <layer name="example-gs"
For remove only a layer (from all skins where it has been registered):
<?xml version="1.0"?>
<object name="portal_skins">
 <skin-path name="*">
  <layer name="example-gs" remove="True" />


Portal tools are singleton, persistent items stored in the site root, visible from ZMI and (commonly) invisible from Plone interface. For a long time, portal tools has been the only place where Plone developers were able to store data and code together.

When ZCA and persistent utility were integrated this kind of items has been partially deprecated. Today there's still a lot of Plone tools, but commonly they are also register as persistent component.

Tools are very simple items, and the Generic Setup support is limited only to the tool registration:
<?xml version="1.0"?>
  <required tool_id="portal_foo"


There's not Generic Setup supporto for deleting tools. If you remove the product from your buildout you'll see a broken object in the site root (because the code is no more available), but it's really simple to delete it using ZMI.

To automatically delete it, you can use Python, but one time again take care on users that are reinstalling your product: if you delete the tool you will lose all configuration stored inside.

Obviously, if the tool is also registered in the component registry, you need to take care of it (see the componentregistry.xml section above).
def _removeTool(portal):
    portal.manage_delObjects(['portal_foo'])"Portal tool removed")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        _removeTool(portal)"Uninstall done")


Not much to say about that, just because it is mainly used by Plone themes, to:
  • Move viewlets around (this is only for theme... for sure)
  • Hide existings viewlet (this can be done because you are providing a new viewlet that replace a default ones)
Note that all this can be done through the Plone @@manage-viewlets view.
I'm not an expert of this import step, but now you can find a good viewlet Generic Setup documentation in the code itself.


An uninstallation Generic Setup step for viewlet could be really similar to an installation one: you only move viewlets and probably un-hide hidden ones (that mean "remove the hidden state"...).

What you must do when somebody uninstall your theme? You must restore viewlet in the original order? And what is the original order? How can you know it?
Probably you can't do much (apart showing viewlets you hidden).


Workflow is a complex argument, and also the Generic Setup syntax can be complex.
It can be splitted in two section:
  • Defining workflow (state, transitions, ...)
  • Defining bindings of workflows to Plone types
Workflow can contains code, but most of the times the code is inside the workflow itself, so totally stored in the site database: this mean that you can remove the product who defined a workflow and still get it working.


Workflow are stored and manageable through the portal_workflow ZMI tool, so is quite easy to perform manual cleanup.

How embarrassing! I'm obviously focused on understand how to remove created workflow but this time I'm not able to find where Plone is managing Generic Setup for this (I'm missing the type bindings part).
So I can't be sure if you can remove workflow items using Generic Setup or not., but I suspect that there isn't any Generic Setup support for uninstallation.

What is important to know is if you need to uninstall (and what).
Is a good idea to remove a workflow that your product installed? Can you be sure that another products, or the site manager is not using it anymore for other types?
This is not so impossible: while defining new workflow is done using Generic Setup or ZMI, the workflow associations to types can be done (and is really recommended) using the Plone control panel ("Types configuration").

Be aware of that.


Historically some old version of paster and probably old documentation generated a lot of Plone add-ons where additional operations to be done with Generic Setup import where stored inside code executed using import_steps.xml:

The use of this Generic Setup step can be used to register a new possible step inside the portal_setup tool.
An import steps example
The new way of getting this kind of features is to use ZCML, and genericsetup:importStep:
      title=" miscellaneous import steps"
      description="Various import steps that are not handled by GS import/export handlers."
      <depends name="propertiestool"/>


When the registered code disappear, the import step will be broken.
An import steps example
This kind of errors (that only trigger some log warning messages) can be easily removed from the "Manage" tab of the portal_setup tool where you'll find a "Steps with invalid step handlers" section.

Like always, you can use Python to automatically fix things:
def _removeImportStep(portal):
    to_remove = ('',)
    registry = portal.portal_setup.getImportStepRegistry()
    for step in to_remove:
        if step in registry._registered:
  "Removing import_step %s" % step)

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        _removeImportStep(portal)"Uninstall done")

Plone 4 included a new component: the Plone registry. The idea behind this is really good: simply speaking this new component can replace the old portal_properties tool (see above) with something that is:
  • more powerful
  • data model can be defined using interfaces
  • usable from Plone UI (there's not ZMI support for it)
IMO the target has been partially reached: the old portal_properties tool is still more robust the Plone registry, and can be easily cleaned up (but not so powerful).

Somethings things goes worse: let's talk of new style collections.
If you are providing new collection criteria in one of you product add-on, you are also registering items in the registry using the prefix, that is not present on Plone 4.1 and below: so you are breaking Plone 4.1 compatibility.

Don't want to repeat myself: I wrote another article about Plone registry on RedTurtle Blog.


Again: don't want to repeat myself: in a second article about Plone registry I introduced how to uninstall properly and manage registry changes in the data model.

About new collection criteria: things are not perfect. Removing registered criteria from the registry partially works only on Plone 4.3...
Probably is possible to do something using Python, but right now I've no idea.


Browser layer is a really powerful Plone toy: it make possible to register views (or customize existings views) only when your product is installed.
Here how to do this with Generic Setup:
<?xml version="1.0"?>
    <layer name=""
           interface="" />


For a long time I didn't provided an uninstall step for browser layers, and you'll find a lot of products that are ignoring this. When you remove a product that registered a browser layer from your buildout, sometimes you don't get any problem.

I started getting a lot of problem with this on Plone version migration.

The truth: you must give an uninstall step for this, and Generic Setup is supporting it:
<?xml version="1.0"?>
    <layer name=""
           remove="True" />

Other less used Generic Setup steps

There are other Generic Setup import steps like diff_tool.xml, contentyperegistry.xml, kss_registry.xml (KSS has been removed from Plone core starting with Plone 4.3), ... it's quite uncommon to use them so I don't spent time here to provide examples, and they commonly don't provide uninstall procedure.

Also not that 3rd part product could provide additional Generic Setup support for other components: an example is ATVocabularyManager.


A single rule: for every thing you install with your product, provide an uninstall counterpart:
  • Your users will love it
  • You'll love it when you will need to remove or change your product in future, for example while migrating to new Plone versions.
Take a look at

Sunday, March 24, 2013

Codemotion 2013 (finalmente!)


Per la prima volta quest'anno ho deciso di partecipare al Codemotion. Codemotion è un'esperienza che già da qualche anno aveva in qualche modo attirato la mia attenzione ma poi non avevo mai approfondito.
Una motivazione tra tutte: sarebbe stata la mia prima conferenza "general purpose" (senza un focus specifico su un linguaggio o una tecnologia) e, come molti sanno, guardarsi intorno (AKA: che cosa stanno facendo i developer PHP, a parte essere presi in giro) non fa mai male.

La parte dell'evento comprensivo dei talk si è svolto in due giornate, il 22 e il 23 Marzo. Per vari motivi si è scelto di limitarsi ad una sola giornata (sabato 23) il che ha significato una toccata e fuga a Roma. Non ero solo, ma mal accompagnato dai colleghi Andrea e Nicola.

Due note sull'organizzazione dell'evento: una veloce iscrizione online (gratuita!) e l'unico costo è stato quello del viaggio in treno (che l'esperta di viaggi Trenitalia Federica, ha trasformato in un'esperienza interessante... vedere sotto).

Altra cosa: un gruppo What's Up per poter comunicare (nel caso ci fossimo trovati in sale separate).

Il viaggio

Sapete una cosa? Poter arrivare a Roma da Ferrara e tornare a casa in giornata è una bella sensazione. Non era la prima volta, ma sicuramente non avevo mai viaggiato in Bussiness! Niente di sconvolgente, ma ho apprezzato i biscotti dolci e il caffè (e il viso del controllore che non può fare a meno di chiedersi se hai sbagliato carrozza).

L'evento si è svolto alla facoltà di Ingegneria Roma 3, e raggiungere il posto con la metropolitana è stato piuttosto semplice e rapido.


Il primo impatto è stato quello di notare una buona partecipazione di persone (in realtà era molto buona, ma siamo arrivati a talk già in corso, quindi molte altre persone erano nelle aule) e come speravo anche molto eterogenee. Tra i miei colleghi io ero l'unico "verginello" di Codemotion e non mi ero fatto raccontare poi molto dalle loro esperienze precedenti, quindi per me è stato tutto nuovo.

Ho apprezzato particolarmente gli spazi dove aziende e startup si aprivamo alla presentazione del "cosa facciamo" e le classiche scatole "lascia qui il tuo CV".

Non male nemmeno l'angolo dei libri tecnici in vendita a prezzi scontati.

Vogliamo proprio trovare qualche lato negativo?

Tutti peccati veniali!

La rete wireless non era molto affidabile (ma va detto che dopo un primo tentativo ho abbandonato in favore di carta e penna).

La cosa un po' più fastidiosa è stata il non essere riusciti a ritirare il regalo, compreso nell'iscrizione (una maglietta azzurra estremamente nerd!) perché esaurite molto in fretta. Peccato...
Va beh... dopo tutto l'evento era gratuito, panino e caffé inclusi, quindi passiamo tranquillamente oltre.

Ultima cosa: ho trovato tutti gli speaker in difficoltà con i 40 minuti di tempo che erano loro dedicati (troppo pochi)... ma sinceramente non vedo altre soluzioni se non perdendo lo spazio per uno dei talk.

I talk

I talk erano distribuiti in varie aule della facoltà e suddivisi in categorie:
  • Enterprise
  • Code It!
  • Zero Code
  • Big & Fast
  • Scripting
  • Insert Coin (questa è difficile... in pratica videogames!)
  • Makers
Rapida analisi dei talk che ho seguito, in ordine di presa visione (o presunta tale).

HTML5 game and gamification design from the trenches

Questo talk di Pietro Polsinelli sarebbe dovuto essere il mio primo talk e sinceramente non vedevo l'ora... il mio passato da (presunto) volenteroso sviluppatore di videogame è stato soppresso dal mio lavoro di programmatore dedicato al Web e dagli impegni della vita, ma il mio interesse per il mondo dello sviluppo dei videogiochi è un po' come una dipendenza di cui non ti liberi mai completamente. Mi avrebbe un po' riportato ai tempi in cui io e il mitico MarcoMa ci occupavamo di Cheese Boys!
Mamma mia che bei tempi!

Ma tornando al talk... purtroppo siamo arrivati troppo tardi, alla fine del talk mancavano 10 minuti e non ho nemmeno tentato di trovare la sala.

Ero davvero curioso di vedere dal vivo cosa si può fare con HTML 5... magari il prossimo anno! Per ora mi accontento di visitarmi il blog dell'autore: the game design experience.

An Introduction to AngularJS

Questo talk è stata una presentazione delle funzionalità del framework JavaScript AngularJS, prodotto open source fortemente sponsorizzato da Google (e nato tra le sue fila). A presentarlo Joel Hooks.

Il mare dei framework JavaScript è fortemente popolato e a volte la troppa scelta può portare all'immobilità per il terrore di investire sul cavallo sbagliato. Il fatto che AngularJS sia spinto da Google può essere uno dei motivi che mette tranquillità ad alcuni (a meno di qualche altro fenomeno Google Wave) ma ancora di più lo è il fatto che il progetto sia open.

Il framework si occupa della manipolazione del DOM con approccio MVC, dove l'utente non deve direttamente occuparsi della manipolazione della pagina se non prima definendo un modello a cui l'elemento della pagina deve essere collegato e modificarsi di conseguenza al cambio dei valori del modello.

Dalla presentazione (come poi non poteva essere altrimenti) non sono riuscito a comprendere tutte le potenzialità dello strumento, se non per poi scoprire che la presentazione stessa (le "slide") erano scritte con AngularJS stesso... e fino alla fine non me ne ero sinceramente accorto.

Tra le conclusioni finale dell'autore, AngularJS pare porsi tra l'uso di JavaScript puro (qui inteso come: uso di jQuery... dimentichiamoci del Vanilla JavaScript please) e un più complesso Backbone.js.
Probabilmente la scrittura di applicazioni JavaScript fortemente spinte (dove il modello è estremamente importante e complesso) Backbone.js è la scelta giusta, ma in altre situazioni AngularJS potrebbe prevalere.

HTML5, CSS3 e JavaScript: Web app per tutti gli schermi

Sala piena fino al soffocamento (se non fosse stata la stessa del talk precedente sarei quasi sicuramente rimasto in piedi) per il bel talk sul frontend design di Marco Casario.

Argomenti vari, principalmente l'ennesimo tentativo di far capire alla gente che il Web è cambiato e che l'uso degli smartphone e tablet è ormai più frequente che gli accessi desktop.
Che cosa fare quindi coi nostri siti? Siti dedicati ad ogni possibile dispositivo, pur sapendo che probabilmente il numero dei modelli non farà altro che aumentare?

La risposta è il responsive design, ma non solo.

Casario ci mostra quindi come nella sua azienda viene affrontato il problema dello sviluppo di design
Anche qui la sorpresa più grande che ancora molti non comprendono a fondo: la prima cosa è un'analisi degli utenti, per capire come useranno il sito. Non si parte quindi con un sito, poi lo si rende responsive a lavoro finito, ma la versione mobile deve essere prevista da subito (e senza URL alternativo!).
C'è poco da fare per i front-end designer: le media query sono una realtà che va imparata e assimilata.

Ma il talk non si è limitato a dare uno scossone a project manager e front-end designer, ce n'è anche per gli sviluppatori di backend, con argomenti quali la riduzione delle chiamate HTTP, la compressione, l'uso di HTML 5 non solo come markup, ma con le sue nuove feature, ...

PS: pare proprio che jQuery Mobile sia tutto tranne che un framework "lite"... meglio buttarsi su qualcosa di alternativo come Zepto.js.

Hacking@School: Realtà Aumentata per la didattica con Software Libero

Prima di tutto: avere uno speaker romano è un vero spasso e Massimo Avvisati ne riassume il concetto!
Talk fuori dalle righe ma sono rimasto un po' fedele alla mia idea di "guardarmi attorno".

Argomento: Augmented Reality, di cui non sapevo praticamente nulla se non "che cosa è" e l'aver visto cosa fa con qualche prodotto (commerciale).

Eppure questo talk mi ha mostrato come ci sia il modo di approcciarsi a questo mondo anche usando potenti strumenti open source. Bellissimi ed interessanti esempi direttamente in demo live.

Particolarmente apprezzato conoscere come lo speaker si occupi di insegnare ai bambini/ragazzini come sviluppare videogiochi... averlo avuto quando giovinavo!

[Pausa pranzo]

...con panino alla mortadella!

Let’s test!

Un talk-staffetta di David Funaro e Andrea Giuliano il cui argomento è FATE I TEST!
Nulla di realmente nuovo, ma magari qualcuno sarà stato evangelizzato.

E' stato sottolineato come iniziare a fare i test cambi la vita ("ma come facevo prima di iniziare a fare i test a fare questo lavoro?") e poi è stata introdotta una prova pratica del TDD, con linguaggio PHP.


Bellissimo talk di Simone Bordet, che ci ha mostrato come il web sia enormemente cambiato, i contenuti sono cambiati, la banda è aumentata... ma il protocollo che lo ha portato alla luce sia ancora qui: HTTP con tutti i suoi limiti, dovuto alla visione che il mondo aveva del Web ormai 20 anni fa.

E allora come sopperire a questi problemi senza dover aspettare HTTP 2.0?

I browser ad oggi supportano tutti (pare, finalmente, anche Internet Explorer con la versione 10) l'uso di WebSocket e ad oggi ci sono vari strumenti molto interessanti come CometD... ma la maggior parte delle risorse del talk è stata riservata al "nuovo" protocollo ideato da Google: SPDY.

L'autore ha più volte posto l'attenzione sul come Google e i suoi potenti mezzi siano arrivati a SPDY in un modo non convenzionale nella storia dei protocolli: provando ed implementando applicazioni reali (GMail e c.), poi consolidando i risultati ottenuti in base all'enorme mole di statistiche che il colosso possiede.

I dati mostrati relativi all'aumento della velocità, dovuto all'eliminazione dei byte inutili che il protocollo HTTP si porta dietro ad ogni richiesta, è stato impressionante... e questo prima di arrivare alla spiegazione della tecnica di push.

La cosa che maggiormente dovrebbe far pensare è che SPDY è già tra noi, i metodi per utilizzarlo partendo dall'attuale infrastruttura della rete ci sono (per ora: pare principalmente uno strumento facile per il mondo Java con Jetty).

PS: HTTP 2.0 sarà molto simile a SPDY
Un talk non tecnico, presentato dal simpaticissimo Fabio Fabbrucci (in pratica, "il mio designer preferito"!!!).

Una carrellata di varie situazioni che nella vita possono capitare e di cui bisogna fare tesoro, di situazioni che possono capitare in un ambiente di lavoro (fenomeno del Burnout) e di come spesso sia importante sapersi mettere in discussione e prendersi le proprie colpe.

Poi è stato messo in evidenza l'importanza di come uno sviluppatore si pone verso l'esterno. Partecipi a conferenze? Hai un tuo blog personale? Contribuisci a prodotti open source? Tutte cose molto importanti... ma per un colloquio non vanno mai scartati neanche i requisiti umani.

E per ultimo: l'importanza di "mettersi in merda" (AKA: mettersi da soli in situazioni che ci possono creare difficoltà, che ci fanno esporre)!


La fase finale dell'evento è stata la presentazione dei vincitori di un hackathon partito il primo giorno di conferenza e terminato il secondo. Leitmotiv: l'uso degli open data per migliorare la vita.

Alla prossima!

Saturday, January 12, 2013

Python libraries I want to use (AKA: my Python Wishlist)

Are you one of the guys that sometimes open the Python Package Index homepage simply because like to see what's new?
I am.

Often I see some new Python packages release about a library I never heard before and I say "Cool! I will use this library for sure one of those days". When this happen you add this library to your delicious bookmark... and you probably forget about it.

Unluckily my every-day-job is 95% focused on Plone where is not so common to introduce new Python packages, but recently I needed to develop something not Plone-related for a... let say "personal need", and I was able to use three of those libraries in the same project!

Let review them now.


The requests library is not new or unknown, and a lot of other Python libraries use it. Why? Because it's simply amazing.

The first time I tried to connect to remote services using Python for something that isn't a simply operation like read from a remote URL, I found that making a POST request is not so simple (...curl is not simple at all).
Then I found the requests library and managing HTTP requests become very easy... and pythonic!
You need to...
  • perform a form POST request?
  • send a file?
  • perform a request with authentication?
  • need to send also cookies?
  • ...
 This is not a problem!


If you ever used the popular JavaScript jQuery library to manipulate the DOM and after that you needed to check an HTML source with other programming languages, you missed all of the jQuery selector syntax.

pyquery is nothing more that this: you will be able to use a jQuery-like object in your Python environment, and use a lot of the CSS 3 selectors, or other pseudo selector you loved.

This is so quick and simple to use that I completely changed the way I perform Zope functional testing in Plone. Ok, is not so common to use doctests nowadays because the are often complex to be maintained, but when it happen and you need to check something in your generated HTML, using pyquery is a lot simpler.


Tests in Python are easy and wheel documented, however is not so simple to create tests when you need to test a feature thats talk with remote services using the standard Python libraries for open remote URLs.
An example: can be easy to write tests when you are using the requests library introduced above?

The Python httpretty library solve exactly this need; you can read this definition in the project's home page: "HTTPretty is a HTTP client mock library for Python".
It mock standards HTTP Python call, giving you a way to simulate what the call must return you back and perform testing totally online.

Another good news? Documentation explicitly use requests library for giving examples!

Other libraries (not yet tested)

As say above: I was able to use all those three libraries together in a single project (Allanon), but my delicious list contains other packages that I've not explored yet:
  • say: a "super-simple templated printing and formatting" library.
  • premailer: "Turns CSS blocks into style attributes".
    This is something that Plone Newsletters products must evaluate!
  • pyzmail: "pyzmail is a high level mail library for Python".
  • nikola: "a Static Site and Blog Generator".
    Sometimes I think about drop site and redo it (and this blog) with this tool.
  • ...