Showing posts with label release. Show all posts
Showing posts with label release. Show all posts

Thursday, May 8, 2014

Loading JavaScript configuration and i18n for Plone add-ons

collective.jsconfiguration is a Plone package that want to give a possible approach to the way of including JavaScript configuration or 18n data inside add-ons.
Although it's heavily targeted to add-ons that contains JavaScript components it will not provide any JavaScript at all, but only three possible ways to get configuration from the server.

Motivations

In last years I saw a lot of different approach to the problem, looking at the source of many different add-ons; every approach could have some advantage or side effects. I've only one golder rule: the worst you can do is to generate your JavaScript dynamically from the server:
class JavaScriptView(BrowserView):

    def __call__(self, *args, **kwargs):

        registry = queryUtility(IRegistry)
        settings = registry.forInterface(IMyPackage)

        # ...omissis...

        self.request.response.setHeader('Content-Type', 'text/javascript')
        return """function something() {
   var settings_1 = %(settings_1)s;
   var settings_2 = %(settings_2)s;

   // do something

}
""" % {"settings_1": settings.settings_1,
       "settings_2": settings.settings_2}
This approach "works", but writing something in a programming language through the usage of another is bad and ugly:
  • A developer that's look to your JavaScript code can be astonished when he get that the code in inside a Python source.
  • The more JavaScript code became complex, the more will became unreadable/unmaintainable.
  • You can't use your favorite IDE JavaScript features because he can't understand you are (partially) write a JavaScript source
  • ... (another 10 "because is bad" can be probably found) ...
 So it's clear that the JavaScript must be in a pure .js file.

A (little) step forward can be to split the code above in a "configuration part" and keep the real code (that will read configuration) in another .js file.
This is better, but still you have some JavaScript code (even if simpler) in a Python/.zpt file... and you have 2 JavaScript instead of ones.

AJAX?

Yeah, AJAX is a right answer to the problem: you write a pure JavaScript code that ask to the server, that's normally reply with a JSON, all configuration needed. You have one .js source file and nothing more.

Well... to be honest you have an additional call to the server for getting the configuration, but how this can be a bad thing? It will probably be a very small piece of data, isn't it? So very very fast...
Not exactly.
The Web is full of articles about mobile/front-end development and how to keep high performance. If you want to read something interesting about this argument take a look to the recent "Is jQuery Too Big For Mobile?".
The real enemy is not how big is your data (hey, I'm not talking of a 10Mb HTML file!) but the latency: the network (even the mobile ones) it's quite fast nowadays but the latency is always pretty high (even more on mobile). We must reduce at the minimum the loading of external resources, especially if they are resources that can't be loaded asynchronously.

Going back to our dummy example above: it's clear that the configuration is a required resource for running the something JavaScript function. We can't start using it without having server side configuration (maybe we can do a little better with some advanced approach... I find this way to call a JavaSript library before it's loaded really interesting).

It will be the same also for i18n: we can't draw a user interface before we have the internationalized strings available.
I've some experience with jarn.jsi18n, and I used it in at least a couple of "pure JavaScript" Plone projects (projects where you don't have a template where add data in other ways I'll introduce later).
In one of them I found that the user interface were loaded in english at first access; after the first attempt translated messages were used normally.
This happened because strings were loaded through an AJAX call to the server but if you use the string "too early" (before AJAX response) they were still in primer language. Luckily after the first usage the library cache translations in the browser local storage, a very smart approach indeed.
The library gives no "onload" callback, so I was planning to provide a pull request for this but... you really like the idea? Delay the execution because we need the i18n strings?!

So AJAX is bad for i18n/configuration load?

No! If you are developing a pure JavaScript application and you don't know what backend technology will be used (if any) AJAX in probably the only choice.

But I'm focused on Plone add-ons here; we know what kind of backend we have: why not load configuration directly from the Plone pages, where your JavaScript will be executed?

Load configuration and i18n from templates

When you are facing the i18n problem with JavaScript and you have a template available (so you are developing an add-on with a view, or a viewlet) you can put translation in HTML 5 Data attributes.
This way you don't need any AJAX call and (even better) you can rely on i18ndude (I tried to use i18ndude also with jarn.jsi18n but it's not fully compatible, there's no support for the "default" value of a translation msgid) and zope.i18n machinery. Cool!

The same will be for general configuration stuff: you can put you server side data inside templates, again in HTML data 5 attributes. This is not a new technique, Plone 5 added some some information inside data attributes (and for what I saw also some small encoded JSON data).

But while using template and data attributes is amazing for translation, I don't like too much using this approach for other data like server side configuration: you will probably find yourself convert a lot of strings in other data types or, if the configuration is large, flood the page of HTML 5 data attributes.

Still use JSON

What is the best way to give data to a JavaScript developer? In my opinion the simplest and most direct way is still JavaScript itself, so providing a JavaScript object.
For example: Plone 4 (and also Plone 5) gives to JavaScript developers the portal_url variable that always gives you the URL of the site. Yes, this is ugly because this variable pollute the global namespace, if another piece of code define a global portal_url var one of them will be overwritten, ... I'm also sure that will change in future, probably it will became a new data attribute, but it's still the quicker way to read that information from JavaScript.

There is an old and well know convention on how not spawn global vars all around the global namespace: put them in a data structure: instead of calling "portal_url" a "plone.portal_url" could be a lot better.
However a lot of purists neither like this approach; in facts you are still polluting the global namespace and the possibility that another JavaScript code don't define a "plone" var is near to zero... but not zero (however I still like this, and I kept it in collective.jsconfiguration).

What's left is still a JSON data source, but we don't want to use AJAX. Uhmm...

The last chance: a lot of super-power JavaScript framework started to use client side templates.
The technique is really simple: define a new script tag of a type that the browser will not know and so it will not try to execute, and put inside it what you want.

You can use this script to store demi-HTML code, or other type of data... as a plain JSON

collective.jsconfiguration

This is the way used by collective.jsconfiguration: it simply register a new viewlet (in the page head) and wait for you registration of additional configuration (from 3rd party products).

Add-ons can register three different types of them:
  • type=text/collective.jsconfiguration.json
    It will store a JSON data inside the script tag. The source will be available to be executed by JSON.parse.
  • text/collective.jsconfiguration.xml
    It will store any kind of data, but it's designed to be used with demi-HTML, as a view/page template output. As the HTML is inside a script tag you are not really forced to use an XHTML or HTML 5 data attributes but you can simply provide an XML.
  • text/javascript
    This is exactly like the JSON case, but it's for guys that like the idea to provide their data in a pure JavaScript plain object.
It is especially useful if you are developing an add-on with JavaScript but without any server side rendering element (no template, no view, no viewlet, ...) because the JavaScript will find the configuration and translation you defined in the HTML head of your page: you only need to configure what to put inside.

plone.app.registry integration

When talking of configuration and user preferences you will probably store your ones inside the Plone registry. Using collective.jsconfiguration and collective.regjsonify you can store inside your JSON or plain JavaScript object the same data stored in your add-on registry sheet.

Example application

For better understanding how collective.jsconfiguration (and collective.regjsonify) works, you can check the example application.

OT: goodbye Plone (for a while)

This was the really-last product in my list of "Plone add-ons that I think could be cool or useful" so I decided to stop spending time on Plone development for a while.
To keep myself busy I will probably start learning some new technology, or I will try to finish my Plone Workflow book...
Who knows?

Saturday, April 12, 2014

"General Type": a new criterion for Plone Collections

A new 1.2 version of plone.app.querystring has been released.
There are some improvement and bugfix but I'm particularly interested in one new feature: customizable parsed query. Why?

Some times ago I started developing a new product for providing some usability enhancement in type categorization using Collection but it was a dead end: it wont work without patching Plone. But the accepted pull request changed everything, so here it is: collective.typecriterion.

The product want to replace the Collection's "Types" search term (adding a new search term called "General type").

The scope of the add-on is to fix some usability issues with Collections:
  • Users don't always understand all of the content types installed in the site
  • User don't always get the difference from a type to another (classical examples: Page and File, or File and Image)
Plone type collection criteria Also there are some missing features:
  • There's not way to quickly define a new type alias or exclude types from the list
  • There's no way to group types under a general (but more user friendly) new type
Some of the point above could be reached searching types using interfaces (through object_provides index) instead of portal_type (the attribute that commonly store primitive type name of every content, but:
  • although search by interface is the suggested way to search by types, it's not used anywhere by Plone UI
  • using interfaces lead to inheritance behavior (which is great... until you really want it)
  • sometimes you don't have the right interface to use. For example, there's an ITextContent interface in ATContentTypes, but it's implemented only by Page and News, not by Event. And generating new interfaces is a developer task
The idea is to keep using portal_type but give administrators a way to group and organize them in a more friendly form.

After installation the new control panel entry "Type criterion settings" will be available.
Plone general type control panelThe target of the configuration panel is simple: is possible to group a set of types under the cloak of a new descriptive type name. In the example given in the image we take again definition of a "textual" content (a content that contains rich text data), grouping all the know types.

After the configuration you can start using the new search term.
Plone type collection criteria Usability apart there's also another advantage in this approach, that is the integration with 3rd party products.

Let say that you defined a new general type called "Multimedia" and you configure it as a set that contains Image and Video, and let say that Video went from the installation of redturtle.video product.
After a while you plan a switch from redturtle.video to wildcard.media. What you need to do is simply to change the configuration of the general type, not all the collections in the site.

Finally an interesting note: the code inside the collective.typecriterion is really small. All the magic (one time again) came from Plone.

Thursday, February 20, 2014

Plone, HTML 5 Canvas and Face Detection with Webcam

One of my latests articles was about HTML 5 Canvas and Webcam integration and in the same article I put all together in a Plone add-on for integrating portrait changes with Webcam.

Recently, following a retweet of one of the cool guys I follow on Twitter, I randomly hit an article that talk about a game that integrate user's webcam as a controller (unluckily I lost the original link, neither I remember the programming language used. The article were also generally introducing face recognition, and this captured my attention.
I asked myself how cool could be getting a face recognition feature with Python. It this something possible to do? Probably not so easily...

Introducing OpenCV

...or not? If you look for "face recognition" and "Python" on Google you'll always get references to OpenCV, the Open Source Computer Vision Python library.
I simply scratched the surface of this huge piece of technology as I'm totally a newbie about computer vision. What I get is that the library can really do a lot of stuff, and it's well documented.

Let me introduce some very-general information.

To use OpenCV for detecting faces on an image you must understand the difference between "face recognition" (find a know face on image or video) and "face detection" (find a face, in general). Obviously the second is a simpler task. Why? Because whatever is your task, OpenCV must be trained to find your target. For not simple task like face recognition you can (for example) train the software by providing a set of images where the object face can be found, and the result of the train is an XML file. After that you can you this file to implement something like Picasa, iPhoto or Facebook are doing when you submit new photos.

With face detection things are simpler because you can find one of those XML file online, already generated for you.

Another important informations about the library: recently a deep API changes has been performed so a lot of examples you can find online are broken (or must be fixed).

Finally, when you are able to do so, prefer the use of cv2 library instead of cv. They are more or less the same library but (for what I understand) cv2 is faster because is based on numpy, so C-compiled code.

Going back to what i did: I focused on face detection. Let see how.

Applying face detection to Plone (yes, I said it)

Meanwhile I were also fixing some minor issues in collective.takeaportrait. A new minor feature is the possibility to move the viewfinder by using mouse drag&drop (because must be forced to be in the middle of the screen is not comfortable).
Here came The Idea: how about a viewfinder that automatically center onto the face captured by the webcam?

Here my wish list:
  • JavaScript check for server side availability of face detection feature (just because OpenCV is not a simple library to be installed... and let me be honest: this is a cool feature, but not really useful)
  • With a not-so-long delay the whole Webcam image take by the canvas is sent to the server for a face detection view
  • OpenCV on the server perform the face detection
  • If a face is found, a rect is sent back to the JavaScript callback
  • The viewfinder is centered on the face using the feature already implemented for drag&drop
The use of Plone here it's a bit unnatural but It was the simplest environment for my experiment because of the work already done with the webcam in the last article. Apart Plone itself I hope you'll the general idea: how simple can be the browser/webcam/face-detection integration, whatever will be your back-end Python framework.

As you can suppose, the experiment was a success!

Introducing collective.takeaportrait face detection feature

I don't think I can add more useful details. Let's see the video!

Saturday, September 8, 2012

Plone and microdata: adding support to microdata to Plone

My last article was about adding microdata support to Plone event content type.
The article also introduced what changes needs to be done to Plone for getting microdata support (in general) and the resulting product (collective.microdata.event 0.1) applied all those changes.

However microdata inside a framework like Plone is not only something you can add, is also something you can support; so I did my best for creating a product that help people to support easily schema.org vocabulary in Plone.

Introducing collective.microdata.core

The resulting experiment is collective.microdata.core, a base package that provide a set of minimal features I already introduced with collective.microdata.event, but this time without relying on any microdata vocabulary.

The product is not for final users, but for developers and integrators. It simply give those features:
  • Provide the definition for the Thing schema.org type (the most basic ones) for all Plone content types (because every content can be a "thing")
  • Provide a rude adapter for obtaining a Thing definition from a content (a very little set of informations)
  • Provide a catalog indexer for saving into brains the most specific microdata type
  • Provide the catalog indexer implementation for Thing.
The package is very small because there isn't a lot of work to do. Unluckily the most part of the work is inside the content's view (like I said in the previous post) and this is still something not very easy to do right now.

Testing you microdata (directly in Plone)

I already wrote about how can be difficult testing microdata today with online tools and what is the JavaScript Microdata Tool. The collective.microdata.core product can adding this little JavaScript library to Plone, just for testing purpose.

This can help you "seeing" microdata inside Plone pages.
Microdata tool with Events

Microdata information inside folder content listing views

Having microdata in content views is great, however you must know that you can provide more that one microdata snippet inside a page. Going back to events example: you can provide a list of events, and search engines can index them all.

The optional package collective.microdata.contentlisting is doing this, but know that the task is not very easy.

Once again: we need to customize Plone views (in that case I talk about folder listings views). The product (right now) is limiting itself to customize the "standard" and "summary" views.

The problem in this task is related to the different type of information we need to put inside those views, when we met different microdata types.
Plone itself put some custom logic inside views, for displaying better information when the listed item is an Event (for displaying start and end date) or a News Item (for displaying the creator; I think this is a new feature, never saw before). This can't be a general purpose approach.

So: this experimental package simplify a lot views, delegating what to be displayed to other tiny-views that 3rd party product can provide. Obviously, default tiny-views are supported by the product so you didn't see any difference.

Once again, collective.microdata.event (new 0.2 version) is supporting this extension.

Microdata tool with a folder listing view

The new version of collective.microdata.event

Version 0.2 is only a refactoring: all logic has been moved to the core package, and this has been transformed to a working implementation of the other two packages.

Moving on

There's still the main issue: when provide microdata implementation we need a deep customization of content's view (if you remember my previous article: customizing with the use of old-way "main" macro). This are an issue that an add-on product can't fix without customizing the Plone main_template (and this is something I really don't like).

It's impossible to have a microdata support in Plone without putting some new features in the Plone core. Is not possible to be sure how important microdata can be in the near future, but I'm quite sure that can be really useful. I think that Plone need to directly support microdata (at least, for the content's main view).

What we can do then? What about a PLIP?

Saturday, June 30, 2012

Approaching automated images optimization in Plone

State of a naked Plone
Is not a news that images inside your site commonly cover a big percentage of the total size of the page (and probably also a good part of the information provided).
You can obviously optimize your Web server for using browser cache and take care of a lot of additional trick, however this probably only solve some kind of problems (I mean: you must do it... it's important!).

There are sites where images are changed (or added) frequently, maybe every day or hour. You can still think about use only small sized image, but this is not always simple to do (for a lot of users) or applicable.

Let think about a Plone site where your main page is a collection that show news.

How a Plone collection of news looks like The screenshot above is taken from a basic Plone 4.2 site where I simply used a collection of news with a modified version of the folder_summary_view template (where I show the 400x400 resized format called image_preview).

The original sizes of images above are not giant but not too small: first news is not giving you any image, 168KB for the second news, 266KB for the third news and 328Kb for the fourth news, for a total size of 762KB).

Now: don't forget one of the most useful Plone feature: the integration with PIL.
Using the PIL (or Pillow ;-)) library Plone is automatically (server side) resizing your image when you ask for a resized version of it (offtopic: when I explain Plone features to users, this is still one of the preferred ones).

So PIL is doing a lot of good job here: we are not downloading 762KB that browser will simply resize after download, but we download a resized version of the image istead.

You can see this in the "Document Size" report taken from the Web Developer Toolbar:
Document size: thanks to PIL Now you can ask: are not the 226KB of JavaScript the real size problem of the page?
Not exactly:
  • JavaScript source can be gzip compressed by Web server (like Apache in front of Plone)
  • JavaScript source are probably also cached in browser; although you update your site's contents frequently (like said above: you add news every hour) JavaScript is always the same. So: it simpler to use the browser cache with JavaScript than with images.
About the last note above: excluding layout images (logo, icons, ...) you must not forget that in Plone images are contents.
About image optimization
I remember a very interesting chapter about image optimization in one of the last book I read (Even Faster Web Sites, from Steve Souders): I learn a lot of information about different images format and problems using them.

The main argument of the chapter is about lossless image optimization.

When you use images for the Web you are often wasting bytes that you can save instead.
I'm not talking of compressing the images while loosing information and give to your users uglier images just for save some kilobytes: I'm talking of saving bytes while keeping the same level of visual information.

The book above talk of a lot of command lines tool that do the trick:
What they do is: optimizing the image when possible and removing image metadata.

Some weeks before reading the book, Denys Mishunov show us a cool tool for Mac front-end developers: ImageOptim. What this tool does is nothing more that try to run all of the tools above (and some other) on images that the user provide.

So: front-end developer must take care of providing the best image compression and optimization they can. Tools like YSlow or Google PageSpeed can easily help you to find images that you need to optimize.
This also should help your site with search engines optimization.

Let's go back to the example page above: I will probably need to run image optimization on all my Plone theme images once but, as you can see above, layout images are a minimal part of the total size of the page.

What I really can't do is: force my users to optimize images before loading them!

What we can do in Plone
My first idea:
As all tools above are command line tools, why don't use theme inside Plone? Why don't call theme as external processes before storing data in Plone?
I'm not the first that think about this task: Jon Stahl wrote a couple of articles about Plone and images optimization two year ago.
Inside the article you can read a sentence that say "Doing a mediocre job on this would probably be pretty easy, but it will take some focused effort to really nail the details that will make this sing"...
I quickly understand that my idea was exactly the kind of mediocre job Jon is talking about :-), and he's right.
Let's move on to understand why.

I put my idea in an alpha product for Plone: collective.optimage.
Do not use it in production until you read carefully the documentation and know what it's doing.
What the product will give you is:
  • react when a new image file is provided to IATBlobImage contents
  • take the mimetype of the file and run all registered optimization handler configured onto the blob file
  • substitute the original blob with the optimized ones
The product can be also configured for using some kind of image optimization tool while ignore others. Indeed the product will do nothing and force you to register manually additional ZCML, one for every external tool.
You can also easily provide your own.

Let's repeat the test with the same page. I added collective.optimage to the buildout then uploaded again all images inside news:
Document size: thanks to collective.optimage We saved 7 kilobytes, but a lot more if we refer to full size images:
  • 147 Kb instead of 168 Kb for the image of the second news
  • 94 Kb instead of 266 Kb for the image of the third news
  • 299 Kb instead of 328 Kb for the image of the fourth news
Total size now: 540 Kb instead of 762.

Problems of the approach
Apart some technological choices I did (just to be not forced to monkey-patch Plone code), the main problem is low performance.

When you use collective.optimage, your Zope thread is running an external process (if you configure more than a provider for the same kind of image, you'll run all of theme) and it's waiting for the execution to end.
Depends on image, format and tool used, this can be a long task: saving the image can became 2/5/10 seconds slower.
Some big images require a lot of time, while some tool are slower than other (for example: I provided an optimization handler for pngout, but it's disabled because I found it really slow).
This is not what you need if your Plone site can host a lot of concurrent editors.

Other approaches
Why don't run this task as scheduled job during the night? Why simply try to optimize all image blob file when the server is not working at 100% (like you'll probably do with a static HTML site)?
This can also be done offline (after all: with blob support images are simply file on the filesystem).

This is a task possible solution but you will provide to your users the unoptimized version of your images when they are "new":
  • editor save the news item
  • first-day visitors will download the unoptimized version
  • night job optimize the image
  • other visitors will get the optimized image from here to end of times
This can be enough if your Plone site is an image archive but not if your main need is optimize news, or other images that expire quickly. A news item image in a productive site can be downloaded thousands of time until another news take over it. Who care about old news?

An approach in the middle can be delay the job for some seconds or minutes, putting the task in a queue of "image to be optimized". This can probably be reached quickly using plone.app.async (so I bet on this as "best solution") but note that this didn't solve totally the delay from having the optimized image available to visitors (but this delay can be really short, like some seconds, in most cases).

Stop! Why collective.optimage is not working on my News Item?
Because I found (and I was stuck when I discover this) that right now images of News Item content type in Plone are not stored in blob. The plone.app.blob product is only working for File and Image content types. I hope this will change in future...

Instead of supporting also non-blob-image, I preferred to test those features on a patched plone.app.blob version that support News Items also.
You can try my fork of plone.app.blob.

Saturday, May 19, 2012

Documentation for Plone products: some bad and good attitude

Times ago I wrote some notes about how to release Plone products. I think I can add some other information to that article, mainly about documentation.
The "documentation argument" is already in the old post (section "Deprecated README") but I want to talk about something different.

One time again: all those minor issues are what I find difficult to explain when I give training to customers that want to learn Plone development.

Let's talk about how documentation change when you release a new version of a Plone product!
HISTORY.txt / CHANGES.txt
Sometimes I surf the Cheeseshop because I need some non-Plone-Zope library and the first thing I miss of the Zope world is the lack of a changes history directly given on the Pypi page.

I've not a bad behavior warning about this obviously! I think that every other framework in the universe must follow the Zope and Plone community example (what I mean is: the history file being also part of the long_description egg option)!

Commonly when history file is well done can also help you to find new features or dependency version changes... this great (but: keep reading)!
README: uncomplete
There are good products (often, sadly, developer targeted product) that can give you great things but have very poor documentation. What I can say about them? This is wrong for me... nothing new. I never liked a simple description of the product, maybe with a sentence like "look at the code and tests".

One example: http://pypi.python.org/pypi/Products.contentmigration. A really good product with a poor documentation.
README: good at the first release, deprecated later
Another bad attitude is to create a good (great) README file at the first release, then never changing it again later (yeah, I know... I've already touched this argument but let's going deep).

One of the thing I don't like is when the HISTORY.txt file also became the source of documentation. Sometimes reading the HISTORY/CHANGES section of a product you can find additional information or features.
I mean: this is good, but not only in the HISTORY section!

When you learn this about Plone, you'll start to read also at the history section of the product you like, but this is sad. Why don't update also the README?

Why this happen? I don't know. Maybe for laziness, maybe because you think that this new little feature is not important... who knows?
Sometime I think that the problem is also consequence of the work of multiple developer. Maybe the Guru Plone Developer did the product first, the another Little Plone Developer added a tiny feature and he someway fear to update also the README file.

So: if you are a product developer, please update the documentation. If you like to use 3rd party products: look at the HISTORY.txt: you could find good news there.

Example: did you know that, when using LinguaPlone you can rely onto a Plone site root view (@@language-switcher) that will redirect you to the proper language folder? This can be found in the history of version 3.1a1.
Package numeration changes
Have you ever experienced this? You installed a product version x.y.z, and it worked. Then a x.y.z+1 is released. No documentation changes, few lines of code in the history, that simple says "we fixed this issue..."... but your upgrade was not so easy, or probably failed.
Looking at the source, you discovered that this release is really different from the one before.

You will suffer of this problems commonly only when you need to keep compatibility with older Plone releases. Another example (still related to Products.contentmigration): what is the last version you could use on Plone 3?
For what I know is the 2.0.1 version. The is also a 2.0.2 or 2.0.3 version (and you can think: this is only a minor release, it's better to keep it) but this will not work on Plone 3 anymore.

This is only a problem of how numbers are used in the release version: given x.y.z I like to:
  • keep x for a deep technological change (for example: now your product will require PostgreSQL to work)
  • keep y for a feature change or addition
  • keep z only for bugfix
You see it? It's the same logic that Plone use!

What about a version that drop support of an old version of Plone?
Probably the most user friendly choice is to change the x number (Like LinguaPlone did: starting from version 4 and above you are warned that is only a Plone 4 compatible version... unluckily you need to go to the history section to read this! :-)),  maybe changing only y can be acceptable... but in my opinion not z!
Please, keep it for bugfix only!

Friday, December 30, 2011

Related contents: keeping compatibility with Plone 3

Some days before the Christmas break we found a little compatibility problem with collective.portletpage: when used on Plone 4, and a user add a related item to a Portlet Page content, the "Related content" area were doubled, shown twice.
One in the Plone 3 style (a ul/li structure inside a fieldset element) and another in the new Plone 4 style (that now is a dl/dt+dd code).

The problem? Starting with Plone 4, the related content section has been removed from content view of default content types and has been registered as a viewlet.
This change is great if speaking of design of Web pages. Now a user can move the viewlet in the content view.
Before this (in Plone 3) the related contents area was always at the bottom of the content's view template, called explicitly as a macro... that still exists in Plone 4.

Technically speaking, the new Plone 4 viewlet is registered for every content type, but still the Plone 3.3 related contents macro exists, and contains the same old code (by the way... I think that keeping it as an empty macro could be a better choice).
So: the problem I'm describing is probably quite frequent. It strikes every content type product updated to Plone 4 but were the view has not been updated recently. I meditate about this problem two minutes and I found other two products we own that right now suffer the same problems: Smart Link and RedTurtle Video. We already planned to fix them sooner.

Quick fix
If you don't need anymore to keep Plone 3 compatibility, simply remove the old call to the macro from your content's view. So: take a loot to your content types now!

Less quick (but fair) fix
What if you want to keep Plone 4 compatibility? Removing the macro call make your product no more perfectly working on Plone 3.3: explicit related contents area will be removed from the Plone version where the related item viewlet is not present.

For  Plone newcomers this can seem a waste of time, but for us isn't. Even if Plone 4 is quickly growing, we still have a lot of old Plone 3 customers that ask for new features. I personally don't like too much the "branch way" of keeping compatiblity (continue releasing an "old family" product and a "new generation" ones).

So you can find many workaround for this, but the new Plone 4 feature (a viewlet for related items) in my opinion is also a great enhancement.
For this reason I take some time to develop a very simple product: collective.relateditems.
With this you simply need to update your content view to Plone 4, then add this product to your buildout when you are in a Plone 3 environment (see the documentation to know how to make this dependency automatically fulfilled).

The product simply register a related items viewlet exactly how is done for Plone 4, but keeping Plone 3.3 CSS styles.
What this product also does is to register this viewlet only for content's that implements a new provided interface. So you are the one who choose if display related items viewlet or not (thanks to ZCA, there is a very simple way to make Python classes implementing an interface even without touching 3rd party code).

Conclusion
I have no real conclusion, just a suggestion: go and try your old product, adding to it at least one related item!

Saturday, June 25, 2011

Inno alla Collective

Qualche tempo fa mi è capitata una richiesta da parte di un cliente che al 70% si è rivelata già svolta grazie al lavoro di un'altra azienda (straniera) che produce prodotti Plone e che ha rilasciato il codice.
Fantastico!

Premessa: dato che il discorso che vorrei affrontare qui non è relativo a quel prodotto o a quell'azienda ma è più un discorso di attitudine generale, non scendo nei particolari. Chiamiamoli quindi Prodotto X e Azienda X (e visto che ci siamo, anche Cliente X).

Come innumerevoli altre volte, con soddisfazione ho potuto dire al Cliente X "quello che mi chiedi, Plone (la comunità) te lo offre praticamente già fatto". Soddisfazione sua (che ancora una volta ha capito quanto abbia fatto bene a scegliere Plone nel panorama dei CMS Open Source), soddisfazione mia (che lavoro meno e non devo per l'ennesima volta reinventare la ruota).

Rimane fuori quel piccolo 30%. Il Prodotto X era carente di:
  • traduzione italiana (questo capita ancora troppo spesso)
  • qualche funzionalità aggiuntiva, specifica del Cliente X, che poteva però essere resa una funzionalità generica e riutilizzabile per tutto il Prodotto X.
Aggiungiamo anche un piccolo particolare, ossia che il prodotto era testato (funzionante) solo per Plone 4. Il Cliente X ha una solida installazione Plone 3.3, sono certo che prima o poi passerà a Plone 4 (4.1?) ma senza fretta.
Quindi alla lista sopra, aggiungiamo anche il backport del codice a compatibilità con Plone 3.3.

La Collective
La comunità Plone è fantastica, tra le altre cose, per la sua adesione quasi totale all'uso del repository di codice comune: la Collective.
Raramente lo sviluppatore medio si prende il tempo di andare a cercare nella pagina della documentazione di un prodotto se c'è un link al repository del codice. E' certamente buona norma documentarlo esplicitamente (ne ho già parlato) ma i Ponisti sanno dove andare a trovare il codice.
E' come entrare in una casa di un amico, e questo ti chiede "prenditi pure una birra". La cercheresti fuori dal frigo?

Capita mai che il codice non sia dove me lo aspetto? Certo... è capitato, ma credetemi se vi dico che è un evento talmente raro da potermi dimenticare che alle volte succede!

Quale è stata la sorpresa questa volta? Ovviamente il Prodotto X (che tra le altre cose utilizza il namespace collective.xxx) non era sulla collective.

Digressione
Mi immagino già un certo numero di lettori che a queste prime parole stanno già affilando le armi pensando che questa sia una difesa all'uso di SVN. E' come se già leggessi commenti dire "la collective è roba vecchia, ora è meglio passare a Git".
Non lo nascondo, c'è una recente tendenza di spostare la gestione del codice da Subversion a Git e soprattutto Github.
Se persino il core dello sviluppo Plone si sta muovendo verso questa nuova tecnologia, sono certo al 100% che ci siano ottimi motivi ma qui non si sta parlando di quella collective.

Non deve importare la collective come "scelta tecnologia" ma come "idea": un posto dove lo sviluppatore Plone sappia di poter trovare tutto il codice che gli serve. Un posto dove tutti i plonisti del mondo possano collaborare!

Se poi la collective è un reposiroty di codice fatto con SVN, CVS, Git o come copia manuale del codice... non importa! Se c'è uno standard de-facto, usiamolo. Ad oggi avere il codice Plone sulla collective o su Github, non fa grande differenza!

Torniamo al Prodotto X dell'Azienda X
La mia idea, fin dal principio (e credo sia sempre un'ottima idea) è stata naturalmente quella di:
  • tradurre/far tradurre il Prodotto X in italiano
  • aggiungere la funzionalità richiesta dal Cliente X in modo tale che fosse generica (la nuova funzionalità stessa ovviamente andava ben tradotta)
  • far funzionare il prodotto con Plone 3.3 e superiori (attività banale questa volta)
Come si comincia in questi casi? Ecco un semplice consiglio:
Se possibile, non limitiamoci a sistemare in 10 minuti le cose che ci interessano del Prodotto X, magari sul nostro repository di codice privato o tramite patch, ma forniamo agli sviluppatori dell'Azienda X un branch del loro stesso prodotto.

Dopo questo chiediamo all'Azienda X di integrare le nostre modifiche!
Questo consiglio paga? A mio parere sempre. Un branch o una patch applicata ad un prodotto di terze parti che tenete per voi e per il vostro cliente non porterà a nulla.
La sensazione iniziale è di aver velocemente risolto il problema, ma è un'attitudine che guarda molto avanti nel tempo.

Se avessi preso quella strada avrei impiegato meno tempo (non nascondiamolo), ma prima o poi il Prodotto X avrebbe rilasciato una nuova versione (corretto un bug? Aggiornato il codice ad una nuova versione di Plone? Aggiunto una funzionalità?) e io sarei stato costretto ad "inseguire", a riapplicare la patch alla versione successiva del prodotto.
Ecco che quell'ora o due risparmiate oggi diventano quattro ore domani.

La posizione del codice
L'Azienda X esce però dagli standard, commette quindi un piccolo peccato. Il Prodotto X è open source ma il codice non si trovava sulla collective.

Scopro infine che il prodotto è ben documentato e nella documentazione c'è un link al repository del codice: scopro con perplessità che l'Azienda X ha scelto di usare Mercurial e un servizio simile a Github: Bitbucket.

Non so usare Mercurial e non avevo troppo tempo per imparare quei comandi minimi per usarlo tanto da poter creare un branch.
Quello che alla fine ho fatto è stato quindi mandare all'Azienda X un file di patch con le modifiche.
Per arrivare a quel file, quel prodotto finito, ho dovuto fare qualche tentativo in varie direzioni, che alla fine si sono tradotti in... una copia del prodotto nel nostro repository aziendale (questo perché avere il codice sotto controllo di versione permette di tornare sui propri passi con eleganza e velocità).

L'ironia sta tutta qui: la scelta dell'Azienda X di non usare la collective (e la mia incapacità nell'uso di Mercurial ovviamente!) mi ha obbligato ad una serie di operazioni che mi hanno fatto perdere più tempo del dovuto.
Sono convinto che lo sforzo sia stato ben ripagato: la patch è diventata parte del Prodotto X e questo a mio avviso è sempre un buon risultato.

Ecco quindi un nuovo consiglio:
Se il vostro codice è open source ed è nei vostri obbiettivi rilasciarlo, il posto giusto è la collective (+ Github).
L'Azienda Y preferisce un repository interno. Perché?
Sia ben chiaro: lavorare con l'Azienda X è stato molto fruttuoso. La scelta di usare Bitbucket è, come dicevo, un peccato veniale. Il repository era ad ogni modo pubblico e la patch è stata discussa e integrata con estrema velocità.

Esistono motivi per cui un prodotto rilasciato non debba invece rendere disponibile il codice sulla collective?

Non ne trovo. Nemmeno uno!

Non sto dicendo ovviamente che tutto il vostro codice debba essere rilasciato. Sono convinto che una parte di questo (che sia piccola e grande dipende dal vostro tipo di cliente medio) stia bene in posti riservati perché conterrebbe informazioni che riguardano il Vostro Cliente.
La scelta in questi casi è ovviamente il vostro repository, o quello del vostro cliente (nel caso ne abbia uno).

Ma se rilasciate il prodotto, perché non dovreste usare la collective?

Come dicevo, non trovo grandi motivi, ma posso mostrarvi i motivi per cui invece dovreste rilasciare il codice dove tutti possano contribuire.

Fix di bug
Un plonista usa il vostro prodotto e ha trovato un baco. Potrebbe averlo fissato per voi. Una cosa in meno che dovrete correggere voi.
Aggiornamento ad una versione successiva di Plone
Un plonista vuole usare il vostro prodotto e ne crea un branch che lo rende compatibile con una versione di Plone più recente. Quando uno dei vostri clienti, magari gli stessi per cui già avete sviluppato il prodotto, vi chiederanno "funziona con il nuovo Plone", voi potrete dire di sì... e sopratutto dire che qualcun altro ha pagato per quell'aggiornamento. La comunità funziona.
Aggiunta di una funzionalità
Il plonista, di solito tramite un branch, aggiunge al vostro prodotto una nuova funzionalità. Vi pare poco?!
Nuova lingua
Il plonista straniero ama il vostro prodotto e lo traduce nella sua lingua. Il bacino di utenza del vostro prodotto aumenta e con esso la vostra visibilità, o quella della vostra azienda.

Quanto descritto nei casi qui sopra non è qualcosa che ho letto sul "Manuale Etico del Plonista". Sono tutte cose che ho visto succedere... e non una sola volta!

Siate generici, e riciclate
Ecco forse il consiglio più importante.

Il Cliente Y vi chiederà qualcosa di nuovo. La prima cosa da fare è guardarsi intorno. Capire se c'è giù un prodotto che fa al caso vostro. Se lo trovate, Plone (e voi con lui) farete un figurone.

Se questo non accade ma trovate qualcosa di simile, vale quanto detto fin'ora... se la vostra necessità si avvicina al 60/70% o più a quanto fornito da un altro prodotto, valutate se è possibile estendere quel prodotto, o migliorarlo.

Anche in questo caso Plone (e voi) farete una bella figura.

Capitano invece casi in cui dovete praticamente per forza tornare a sviluppare qualcosa di completamente nuovo.

Prima di iniziare a risolvere il problema del Cliente Y a testa bassa pensate se quanto state sviluppando non possa essere reso più generale, e possa quindi essere rilasciato.

Vi è mai capitato, dopo una nuova richiesta, di dirvi "questa richiesta mi è già stata fatta"? Se sì, e non avete reso la soluzione del problema generale, avete perso un'occasione. Il copia/incolla del codice da un prodotto ad un altro non è una soluzione molto mantenibile...

Non inserite un nuovo pezzo di codice "a caso" dentro ad un altro prodotto. Non conta se questa nuova funzionalità vi sembra minima. Spendendo poco in più potreste ottenere qualcosa di prezioso, qualcosa che in futuro potrebbero usare altre persone, ed essere migliorato da altri.

Come già detto: quello che spendete oggi vi ritornerà domani. Investimento!

Prendo un esempio tra i tanti che potrei citare: collective.portaltabs. Questo ricalca perfettamente quando descritto sopra.
Abbiamo incontrato vari utenti che:
  • volevano il controllo dei link in testata del sito
  • avevano paura (o non avevano i poteri) per accedere alla ZMI
Senza un prodotto apposito l'unica soluzione è accedere alla ZMI ed imparare come fare. E' facile? Lo è certamente per molti di voi, ma l'utente inesperto non ama la ZMI. ZMI è dove nulla è in italiano, dove si possono fare "cose pericolose" e dove le cose non sono per nulla chiare.

Quel prodotto non fa nulla di speciale, ma fa qualcosa che vari nostri clienti hanno richiesto.

Traduzioni
Un altro grande ostacolo al rilascio al pubblico sono le traduzioni del prodotto. La tentazione di partire in quarta scrivendo le varie label e descrizioni della vostra interfaccia grafica direttamente in italiano può essere forte.

Molto spesso quello che si ottiene è quindi un buon prodotto, probabilmente anche riutilizzabile e riutilizzato tra vari clienti e installazioni, ma che non può essere rilasciato pubblicamente.

Le prime volte che vi scontrate con il sistema di traduzioni di Plone può sembrare una procedimento molto lento. Ancora una volta: è un investimento. Ad oggi per me è diventato normale scrivere tutto in inglese e tradurre tutto velocemente alla fine.
Fare le traduzioni dei vostri prodotti in Plone è facile, davvero facile (ho scoperto di recente che con la nuova versione 4.1 non ci sono alcuni problemi che ho riscontrato in questi anni) e richiede un tempo aggiuntivo piuttosto misero.

Se fare questo, il vostro prodotto potrà diventare qualcosa che tutta la comunità potrà riutilizzare ed aiutarvi a mantenerlo.

Plonista avvisato...

Thursday, March 17, 2011

Release products for Plone: some bad and good attitude

Time passed from age when plone.org was the only place where a Plone user was able to find and download Plone products, and also where a Plone product author (starting from now: a releaser) could publish and distribute his own projects using directly the PloneSoftwareCenter installation.

Plone 3.1 arrived, and the Plone products begin to be found as Python eggs. This was great! Also with buildout adoption, this new way to make and release products change our world.

In facts this wasn't a real "new" way to do Python stuff. Python world already use eggs, simply Plone update its status.

Automation make release process faster
So, Plone products became Python eggs? Well... they can now be released also on the cheeseshop. This (informally know as pypi) is the main Python collector for eggs in the world, and more important the process to release additional eggs on the pypi repository is simple and fast... a lot faster that use the TTW procedure from the Products section on plone.org.

You need simply to type:
    python setup.py register sdist upload
...and the current egg will be registered and released. Very simple!

Plone.org as a egg collector
What about Plone.org? Modern Plone advanced users know that you can look for Plone add-ons onto the cheeseshop, but the great part of users community (and new also users that wanna try the CMS) will user still plone.org Products area.

Again using PloneSoftwareCenter, the plone.org became a pypi server for Plone stuff. Fantastic! Amazing! Releasing a product there became as fast as on pypi!
    python setup.py register sdist upload -r plone.org
This will work starting from Python 2.6, but there's a Python module for being able to use this also on Python 2.4 (see collective.dist).

All those great pieces of information are taken from the tutorial "How to upload your package to Plone.org". Every Plone releaser must read this at least one time.

So, where are the bad news?
The main problem there is in the last sentence of previous part. I fear that:
  • not all of the Plone releaser know this tutorial, or
  • they ignore it, or
  • they think is not so important.
To be honest, skipping the "release on Plone.org" part don't give problem to developers: the software is still on pypi and so installing it on customer servers is possible. Also, pypi is the first choice for downloading eggs with buildout (plone.org is used only as mirror, in the case that pypi is not responding).

So why bother about continuing to perform Plone.org releases?

Because I'm pretty sure that we need to always think about new Plone users and non-developers users.

Some days ago I was creating material for a Plone training (for Plone administrators, not Plone developers) and I wanted to insert a reference to a specific Plone product (a new portlet). I was not sure of the project name so searched for it.
My "developer attitude" made me go onto pypi.python.org, but then said: "This is not good and simple to understand for non-developer... let use plone.org, I need to teach to those people how performing such tasks".
Epilogue: the portlet product is not registered on Plone.org.

This give me the idea for this article. Plone newcomers will never find this portlet! They only know that there is a Plone website (plone.org) with a Product addons section (plone.org/products). What the hell is "pypi"?

You can test this every day: try to monitor how often a new Plone product release is registered on pypi, simply looking at its homepage... I think that every day no less that 10, maybe 15 new releases are done.

Now try to find how many new release are putted on Plone.org products section... you know the answer.

So: this his an invitation to Plone developers to register they work also on Plone.org, and read the tutorial given above. This is important for the community.

Maybe the tutorial above needs to be highlight someway on the Plone website.

Any other bad attitude?
Taking the release process also on Plone.org as a standard is a great goal, but we can do better.

Let's not imagine that a collective.foo project has been successfully released on pypi and plone.org. PloneSoftwareCenter is a fully ready-to-use pypi server implementation... but it offer a lot of additional features.
Those features are not so simple and quick to be executed as the release process itself, but they are very important for users.

The problem that I'm going to explain has become a little worst now that the Products section on plone.org is changed, becoming a lot cooler with additional features.

After performing a plone.org egg release, the product description data of our collective.foo is not fully completed. This mean that releasers must go to the plone.org project created and edit it.

EDIT: to perform such link actions on plone.org you will have problems with the current TinyMCE version that the site use. You need to disable TinyMCE while you are editing reST data.
To do this, use Firefox with Web developer toolbar and using the "Disable" menu, disable all Javascript. Enable it again only when finished.

Let's explorer what is not managed from Python release process, starting from the "Default" schemata:
  • Categories. This is a very important field for users that want to search for Plone products from plone.org. The Products section has a specific drop down, for example to filter only "portlets" or "content type" products. Sadly if we don't compile this our product can't be found when filtering. This is one of the worst error you can do there.
  • Contact address. This is compiled directly from the Python egg, but I want to highlight also this field because sometimes the e-mail there is only a default paster template created, that came from the ZopeSkel defaults. Sometime you read "plone-developers@lists.sourceforge.net", but this is only a default, not customized.
  • Home page. This field is often compiled with the SVN repository URL... not a very big problem, but the good field for SVN is some lines below this. Also this trivial error came from ZopeSkel default.
  • Screenshot. For product that give something to the Plone UI I think is a very good attitude to take 3 minutes and make a screenshot. This help users to understand what you have done, and make plone.org a better website... (let me say "less boring").
  • All other fields. All other field in this first schemata are less important, but take you time to think if you have something to put there.


Now the "Metadata" schemata:

  • The only field there that must be noted is the Classifiers field. You can compile this directly from the Python egg, and is also used on the cheeseshop... so take some time to add all classification that match from the classifier list. This make possible to search categories from pypi website. The most important (but don't limit yourself to it) is the "Framework :: Plone" value. Don't forget this.


Skipping the third schemata, let's go to the "Review" ones.

  • Self-Certification Checklist. This is a self-certification, so the developer review himself and his work. The only good attitude there is to be honest... and take 5 seconds to fill this data. Dont' be unpolite!


Some last thing to do? Create a documentation section if you have something to give to users.

Finished? Well... no... also the Release content on plone.org need to be checked:

  • Release Summary. Compile this with a few lines of text, like "bugfix release", "securify fix to previous version", "added some simple features", "a complete rewrite of previous versions", "first public release", ...
  • Full Release Description. A complete text of the new features, not a changes list of features (that can be added below).
  • Changelog. Changelog (given there or wherever you think is better) is a great information for your product, and I really like this attitude, used widely in Plone and Zope projects.
  • Compatibility. Very important, and now a much more, as plone.org Products section filter automatically projects for the "Plone 4" release.
    You developed a great project for Plone 4? Well... if you don't say that this is Plone 4 compatible filling this field, probably a lot of users will not find your product (they need to change the combo box to "Any version").
All the field above, excepted "Compatibility" that is a very important field, can be skipped and ignored if you write a good README.txt file in you egg (see below) that will automatically fill the main text field of the Product content and also the pypi page.
I find a good attitude take some time to compile those fields, but I can forgive you if skip it!

Is this enough?
Going back to the egg structure, the two most important field (also used in the project "Default" schemata are the description and long-description.

First of all sometimes (not very often) eggs are released without a "description" information, or a stupid ones. For finding those projects Zopyx released the nice zopyx.trashfinder. This is used for the project description on plone.org and also on pypi! Write a good (but also short) summary there!

Another bad attitude is not compile at all, or compile with a few futile lines of text the README.txt file, that will fill the "long description" text of your egg.
Take you time to write there a good README and documentation. Give there all the information you have.
What syntax I need to know and write there? Use reST! So you can write text headers, format text and images! Images are important (so why limit yourself to the plone.org screenshot image?).
   .. image:: http://youhost/youpath/youhelpfulimagename.jpg
:alt: A good alternative image
Deprecated README
When you write a good README, take some time to read it from scratch at every release. It's quite common to find Plone products (also well-know products) that say something wrong, just because a sentence wasn't true again. A set of examples:
  • "Compatible with Plone 3.0"... but the last release is only for Plone 4
  • "Installing without buildout" instructions.
    I've never truly understand such kind of instructions. Buildout is the only official way to install Plone. You can't install Plone with "easy_install Plone" so why teach to install something other with "easy_install collective.myamazingproduct"?
    More bad: maybe the last release has added the "Plone" dependency in the egg... so one time again, easy_install it system wide will give you only headache.
  • ...(I'm sure you can find some other example)...
A README with bad instructions can be worst than don't having a guide at all. Such kind of bad documentation bring users to say "Plone is difficult" or "Plone is undocumented".

Last product version must not be the only version available
Final bad attitude.
When a new release is registered on pypi, it automatically hide the previous versions. If you type (as example) http://pypi.python.org/pypi/Plone you are seeing the last version directly, that is (at the time I'm writing this post) Plone 4.1b1.

I had a discussion about this recently on Product-Developers mailing list: as far as the new product release brake compatibility with previous Plone version (I mean: version that are still maintained) the guy that register the release on pypi must also think about don't hide the previous releases. For what I know, there isn't an automated way to do this, you need to login on the cheeseshop.

A good example? http://pypi.python.org/pypi/Products.Poi
As you can see, the new 2.x (only for Plone 4) is not hiding the 1.2.x branch for Plone 3. Also, documentation is clear. If a user read the page of the 2.x release you can read explicitly "Poi required: Plone 4; this version of Poi will not work with Plone 3".

No more to say, now go and check how good are your Plone releases!