Views hits counter for videos in Plone

Some years ago we developed a Plone site that mainly provided video and multimedia contents. One of requirements was to obtain a views counter for videos in a classic YouTube style, but this feature at the end was cut off.
However the idea to obtain this in Plone for collective.flowplayer persisted in my mind (another product in the set of "products that I want to develop someday and probably I will never do because I don't have time").

Recently I traveled from Ferrara to Rome by train, and back forgetting the book I was reading at home, so... why don't take some time to investigate again this old feature?

Literature
First of all: what are the know ways of performing this task?
I took some time to look at the Web; unluckily I don't find some general rules or patterns. I though this not as a "visitors counter" feature: if a user visit the page where we are displaying the video player, it doesn't mean that the user will see the movie.

Now the problem: even if I don't want a perfect system and I don't care if after 100 views the counter will show 90 or 110 (the error will be the same for all views in the site, so statistically speaking, useful), I'd like to create a system where:

• an evil visitor can spent time to raise manually it's video
• an evil bot can't raise my counter to 10.000 in few seconds!

Let's move on, step by step on what we can do.

Monitoring the Play of my clip
Flowplayer JavaScript APIs are well done and already integrated in collective.flowplayer, so in Plone. The simplest thing we can do is to rely on the onStart clip's event. After this event has been captured by our callback we can think about sending a call to the server, that mean "the user is seeing the video".

The question: is this enough? If I provided a 5 minutes clip and my lazy visitor only see few seconds after moving away, can I count this as a new video view? Even if my super-fast Internet connection already downloaded and buffered it all?

My answer is "no" (however this can't be taken as "the right answer"). Let's try to think about a system that monitor only if the video has been totally view!

Monitor the end of my clip
The next step is to use also the onFinish event. We can monitor if the user starts to see the clip but also we can check if the clip is completed. In this way we can send to the server our message only when the video has been finished.

This is a step forward, but we can't be sure again that the user really watch the video. He could start to watch at the clip and then move the slider some seconds before the end.

Monitor cuepoints
The two JavaScript events above were already know by me, I used them while developing collective.flowplayer_toolbar.
What I learned looking at the documentation is that Flowplayer APIs also support cuepoints. Cuepoints are a set of events callback that are automatically called every n seconds (let me use 5 as n for our example).

Now we send to the server the final message only if the onStart event has been executed and if all cuepoints also are also executed. To do this we simply use a counter that is increased after every cuepoint execution.
We still rely on the onFinish event, but we can use JavaScript to be sure that the message is sent only if the counter reached an certain amount. This amount must be a value that depends on clip duration but again: Flowplayer APIs contains method to obtain video duration.

Starting to think at the Evil Guy
In a perfect world, where all are Good Guys and no one will ever try to break rules, the general description of the code given above can be enough. But we have Evil Guy.
Who is the Evil Guy? Is a technically-low-level visitor that don't know how to write code or hack JavaScript, but simply try to raise a views counter of a clip. Is a cheater.
How he can do? First of all he can spend a lot of time clicking the "Play" button again and again, every time the clip reach the end (Evil Guy always has a lot of free time).

To protect our site from this type of attack we simply need to monitor that this clip has been already view by that user. When the video starts we can send a call to the server and get from it a generated random token that we keep secretly in the JavaScript environment and on the server itself.
We will not use cookie for this, because Evil Guy can quickly learn how to manipulate them, so we choose to keep this information in the Zope session (this can lead to problems with multiple Zope instances, in that case we probably need some complex RAM cache). We store also the video path, because we want that users still able to look at other site's clip (and raise counters normally on them).

When the video reach the end we still send to the server our message but this time we also send the secret token. Only if the token match we raise the counter.
Another click of the Play button will call again the server, but this time we can see that there is already a token for that clip stored in the session. This mean that the visitor already saw that clip. This time we will stop immediately any other operation.

Even if the Evil Guy reload the browser page, he can't do anything else until the session expires.

The collective.flowplayerclipviews product
Travel from Ferrara to Rome is long, but not enough! All I described right now is more or less what you will find in the first version of collective.flowplayerclipviews. As you can imagine, I'm really far from the target!

Evil Guy gets smarter
Evil Guy can also start learning programming, and so know that JavaScript is a client side language and he can cheat, stopping the boring time he need wait for press the Play button again.
First of all: destroying the browser session is simple as close the browser or erase cookies.
After that he can simply take the secret token from the server and call the server again few milliseconds after, pretend to see the clip!

So now we can start to think about video duration server side. As we said, Flowplayer can give to you the video duration inside JavaScript environment but what about server side? There collective.flowplayer rely on the hachoir suite: inside all multimedia content Plone will store some annotations about video information (mainly: width and height).
Unluckily for us, right not collective.flowplayer is not storing also the video duration (that hachoir supports, but Plone doesn't need) so for now let simply think that collective.flowplayer will do this in the future... and we are using this future version in the rest of this article.

Having the video duration server side can be used to stop Evil Guy from raising the clip counters very quickly. Even if he write a program that take the token (simulating the Play button) and immediately call the server (simulating the clip end), we can also check that the latter request that mean "video terminated", arrives a certain amount of time later the "video started": this amount of time is the video duration!

This will be a short victory. Evil Guy can then start running hundreds of cheating Play operation contemporaneously, wait for the video duration, then send the finish message. In this way he must wait for the video duration, but he can raise the counter of hundreds/thousands anyway.

Can this be avoided? The only way is memorizing the address of the Evil Guy and keeping it in memory for some time. However ZODB is not the place for this kind of temporary data: we can think about storing this information outside, again in a RAM cache environment, or an external database, but we need to keep the write operation on ZODB at minimum.

ConclusionsAfter all those changes we can say that we have a good system... but we need to keep in mind that we can't be sure that the visitor really see our video! He can press the Play button, the go to take a shower! World is not a perfect place.

The collective.flowplayerclipviews right now is only a proof of concept, not to be used in production. If you look at the source, you can see that the _getClipDuration method is not implemented. After that all Plone feature will be there (we still need the external IP address storing structure).

During the time I spent writing this article (commonly it requires me some sessions), I tried again the search of general documentation about my argument. This time I added "plone" to the set of keyword... then I discover that we already have a Plone product that does this task (my fault: I don't googled well)!
I'm talking of collective.piwik.flowplayer! This product is part of the Plumi suite. I already checked Plumi before starting this article, but I miss that feature. Also know that this product is right now deprecated in favor of collective.piwik.mediaelement.

As many other Plumi internal submodules, those products are usable outside a whole Plumi site. Both modules are not implementing the counter feature in Plone but smartly rely on an external software: Piwik.

Piwik is an analytic software, we can say it's a competitor on Google Analytics, based on a JavaScript snippet that you must put in the page.
Apart all other analytics features, looking at the source seems that it is simply checking the Play button pressure... (let me say that I tend to complicate my own life and this is probably the good way). If this is enough for you, I strongly suggest to rely onto this service.

Conclusion (this time, really)
I hope I shown you that having this feature in Plone is possible.

Honey I Blew Up My Plone Product!

This article talk about Plone products but I think this is a very general argument that hit every programming language: the lack of modularity in products.
Know that:

• I'm not talking of the Plone core, that is obviously going in the right direction (the plone.* and plone.app.* universe is one of the clearest example) but only of products add-ons.
• I'm not also talking of bad code (no "Spaghetti code"). Maybe your final result is great and will change the world! I don't care.

Your first time
If you are not a genius, probably the first Plone product you did was crappy. Apart of the features in it, its structure was probably something like a single product called "Products.MyProjectXy".
What this mean is: all Plone features you need in a single product (a real monolith!).
Maybe you were already an experienced programmer (so your product was well designed... inside) but the final result was not the best.

After some times, when you started looking at other product's code (or if you bough the Martin's book), you probably quickly learned that splitting the product is good: the Separation of concerns is really important.

What is the level of modularity you need to reach? In how many different eggs/packages you need to split your project? I've no a real response about it... we have a Third Normal Form about software design? I don't remember too much of my lessons about program engineering...

Thinking modular
Let me say: splitting a Plone project in more than one egg is "boring". Sometimes you have the sinister feeling that you are loosing your time (and I also remember times when ZopeSkel wasn't there to help us) and the temptation to ignore this practice. Maybe a Demon of Bad Software is talking at your soul in that moment.

An example: you are in a very productive state, lines and lines of wonderful code and new features are flourishing under you fingers (Daniel Pink call this the "Flow State")... then you develop something you know it's better to be moved away from the module you are developing because you know that this choice, one day, will pay you back. But to do that, you need to stop your "Python Frenzy" for some minutes.
Don't know you, but this sometimes hurts me. However I know this is good and must be done so (bleeding) I move the code away. Not later! Immediately! Why? Because the path to "Products.MyProjectXy" is easy... when you are there, it's too late!

Every time a customer ask you for a new feature:

• think if this feature can be made more "general purpose"...
• ... or if another customer can like it ...
• ... or if someone else already asked for something similar.

If at least one of those answer is true, probably you must develop something "more general".

Too famous Plone features (sometime more that the product itself)
Some concrete examples.

Let me say that sometimes, when you create a Plone product, you don't know how useful to the community it can became in future. Worst, sometimes a feature inside you product can be useful to other...

... is not always easy to "think at future": the list of projects that follow talk about great Plone products I used a lot of times and the saved my time. So probably done by great developers... however inside them I can still found sometimes the problem I'm described above.

The Kupu link administration
We have all moved away from WYSIWYG Editor Kupu, preferring the famous TinyMCE.
However Kupu still have a useful feature inside it (I still remember the first time I see it at one of Plone Conference... woah!): an administrative tool that can transform your links inside documents from "normal" ones to the "Use resolve UID" form, or get back to the normal form.

Unluckily this feature is shipped with the Kupu product (and: using AJAX stuff, it need that Kupu si really installed to work properly).

The Maps location field and widget
The product Maps give us some simple but direct features of integration with Google Maps. Also, it contains a new type of field and widget: the LocationField (that can contains 2 coordinated values) and the LocationWidget (that help filling the LocationField using Google Maps interface).

This new field/widget pair can be very useful, but this feature is inside the Maps project, not given as external dependency. Also you are forced to install the product in your Plone site if you need it (having it only in your buildout is not enough).

Flowplayer metadata extractions
The well know collective.flowplayer is giving simple Plone Video features to Plone. Instead of simply giving you the video player integration, it contains a simple piece of code that extract video metadata from uploaded files using the 3rd party library hachoir.

We have two problem there: if you need the metadata extraction feature you need to add Flowplayer to your buildout (minor issue: this time you don't need to install it) but also you are forced to use hachoir, that is not a perfect library (first of all: it brakes you pdb!).

Other examples?
I'm sure there are other good products that give you additional features you can like to have also outside the products itself.

Conclusion
Keep you products compact but always think about software reuse. Experience done with your own customers and users can help you to know when a piece of technology need to be moved away and maintained alone. Also: small products are simple to be maintained.

So: don't listen at the Demon that say "Enlarge you Egg".

Tabindex: handle focus and JavaScript events on every elements

My last reading (still in progress) is a book about JavaScript: "JavaScript Cookbook" (Shelley Powers - O'Reilly). It's full of very interesting chapters (probably it will feed many future article here).

Today I will talk about a JavaScript behavior learned in one of the examples found in the book, related to JavaScript events inside Web pages.

Types of usable event
Using pure JavaScript, not all existing events can be used on every page elements. For example: one of the first things I liked when I started using jQuery year ago was the possibility to use onclick event for whatever DOM page element, also in Internet Explorer (that commonly dosn't allow this event to be fired for all types of element).

The example I found in the book talks about another event: onkeypress.
What kind of elements can react to this kind of event? Commonly only elements able to take the focus (like links, but more common are form elements like textarea of input).
The example shows how using another uncommon JavaScript attribute we can change this behavior, enabling all other page elements. I'm talking of tabindex attribute.

How to use tabindex
As I'm mainly a Plone programmer, when tabindex was removed from all templates of the CMS (don't remember exactly, but I think it was when Plone 3.0 was released) I was happy. Why? Because tabindex is commonly not needed in a well-designed Web page.

The tabindex attribute define the order in which a user navigate the page using the keyboard (the TAB /SHIFT+TAB key) but if you define the order of your DOM elements in a logical manner you will simply not need this attribute. However note that it is not deprecated:

• Some non-common pages can need this attribute to force the navigation first to a page section that isn't at the top (to be honest, I can't find a simple example of this)
• A value of tabindex of "-1" make an element, commonly navigable using keyboard, not accessible anymore.

See this good tabindex reference for more details.

Now the great tabindex magic I learned: when tabindex is used on a DOM element (with a value of 0 or more), it magically gain the power of obtain focus.

Let see some examples.
Please note:

• I tested examples with Firefox, Opera, Chrome and Safari, on MacOS. With older Internet Explorer versions you probably need some fixes in event handling.
• If you are on MacOS, use Chrome because the keyboard navigation with TAB is a mess. See below for more on this.

The first example is a simple HTML page with 2 links, one paragraph, and where onclick and onkeypress on those elements raise simply an alert message (the text inside the node).

<html>
<head>
<script type="text/javascript">
<!--
window.onload=function() {

    var handleAction = function(event) {
        alert(this.innerHTML);
    }

    var p = document.getElementById('foo');
    p.onclick = p.onkeypress = handleAction;

    var links = document.getElementsByTagName('a');
    for (var i=0;i<links.length;i++) {
        links[i].onclick = links[i].onkeypress = handleAction;
    }

}
//-->
</script>
</head>

<body>
    <a href="javascript:;">Link 1</a>
    <p id="foo">
        Test here!
    </p>
    <a href="javascript:;">Link 2</a>
</body>
</html>

First of all the mouse event: you will be able to click on both links and the paragraph (not sure of this in all IE versions as I'm not using jQuery there).

Now let's try the keyboard navigation. Using the TAB you will be able to move only onto link elements. When one element get the focus, clicking a key you will get the same message you get for a mouse click.

If you look at the code you'll see that I'm trying to register the onkeypress event also on the HTML paragraph, but it is ignored.
This is right: a world where every DOM element can get the focus will make keyboard navigation painful. But: how can I do if I really need that a single DOM element (commonly not focusable) could take focus and keyboard events?

This is where tabindex help. If you look at the second example you will see that the only difference is the use of tabindex on the P node.

<html>
<head>
<script type="text/javascript">
<!--
window.onload=function() {

    var handleAction = function(event) {
        alert(this.innerHTML);
    }

    var p = document.getElementById('foo');
    p.onclick = p.onkeypress = handleAction;

    var links = document.getElementsByTagName('a');
    for (var i=0;i<links.length;i++) {
        links[i].onclick = links[i].onkeypress = handleAction;
    }

}
//-->
</script>
</head>

<body>
    <a href="javascript:;">Link 1</a>
    <p id="foo" tabindex="0">
        Test here!
    </p>
    <a href="javascript:;">Link 2</a>
</body>
</html>

However, the behavior difference is big.
You can try again the page with keyboard navigation with all browsers used before (again: I suggest to use Chrome if on MacOS).

Now you are able to give the focus to the paragraph! Also, keyboard event is now raised on the paragraph. You probably get the point...

Tabindex attribute can make focusable whatever element of the page and you are able to use keyboard events on them.

Isn't this great? I think that could be helpful sometimes.

Browsers differences (AKA: MacOS hate TAB)
Let's change the article focus.
Testing example one and two with all defined browsers shown some different behavior.
The tabindex definition says that element with tabindex are the first that take the focus (from the lower value to the upper), keeping the DOM order when values are equals; only then, all other focus capable elements are traversed (still, using the top-down DOM order).

For a reason I never deepen, on MacOS (but workarounds can be found on the Web) only form elements seems able to take focus (links not). Also with Firefox, that on other Linux/Windows environment works like a charm, I can't put the focus on links.
The only tested browser on MacOS that works as expected is Chrome, however also Chrome does something unexpected: if you test the second example you will see that the DOM order are kept (while the tabindexed ones must be the first).

But example 2 works also on other browsers on MacOS: you are able to put the focus using TAB key onto the paragraph.
So you can think "I can try to put a tabindex on every DOM page element I want to be accessible using keyboard to fix this MacOS problem".

If you test example 3 (that put "tabindex=0" also on A elements) you'll that this is false. Still, links are not accessible using keyboard. Strange...

Related contents: keeping compatibility with Plone 3

Some days before the Christmas break we found a little compatibility problem with collective.portletpage: when used on Plone 4, and a user add a related item to a Portlet Page content, the "Related content" area were doubled, shown twice.
One in the Plone 3 style (a ul/li structure inside a fieldset element) and another in the new Plone 4 style (that now is a dl/dt+dd code).

The problem? Starting with Plone 4, the related content section has been removed from content view of default content types and has been registered as a viewlet.
This change is great if speaking of design of Web pages. Now a user can move the viewlet in the content view.
Before this (in Plone 3) the related contents area was always at the bottom of the content's view template, called explicitly as a macro... that still exists in Plone 4.

Technically speaking, the new Plone 4 viewlet is registered for every content type, but still the Plone 3.3 related contents macro exists, and contains the same old code (by the way... I think that keeping it as an empty macro could be a better choice).
So: the problem I'm describing is probably quite frequent. It strikes every content type product updated to Plone 4 but were the view has not been updated recently. I meditate about this problem two minutes and I found other two products we own that right now suffer the same problems: Smart Link and RedTurtle Video. We already planned to fix them sooner.

Quick fix
If you don't need anymore to keep Plone 3 compatibility, simply remove the old call to the macro from your content's view. So: take a loot to your content types now!

Less quick (but fair) fix
What if you want to keep Plone 4 compatibility? Removing the macro call make your product no more perfectly working on Plone 3.3: explicit related contents area will be removed from the Plone version where the related item viewlet is not present.

For  Plone newcomers this can seem a waste of time, but for us isn't. Even if Plone 4 is quickly growing, we still have a lot of old Plone 3 customers that ask for new features. I personally don't like too much the "branch way" of keeping compatiblity (continue releasing an "old family" product and a "new generation" ones).

So you can find many workaround for this, but the new Plone 4 feature (a viewlet for related items) in my opinion is also a great enhancement.
For this reason I take some time to develop a very simple product: collective.relateditems.
With this you simply need to update your content view to Plone 4, then add this product to your buildout when you are in a Plone 3 environment (see the documentation to know how to make this dependency automatically fulfilled).

The product simply register a related items viewlet exactly how is done for Plone 4, but keeping Plone 3.3 CSS styles.
What this product also does is to register this viewlet only for content's that implements a new provided interface. So you are the one who choose if display related items viewlet or not (thanks to ZCA, there is a very simple way to make Python classes implementing an interface even without touching 3rd party code).

Conclusion
I have no real conclusion, just a suggestion: go and try your old product, adding to it at least one related item!

5 minutes with tal:attributes

Let's take a quick read about tal:attributes, an important Template Attribute Language statement of the Zope and Plone development.
About static attributes
You already know that when using tal:attributes you can mix dynamic attributes with static ones:

<h1 class="main"
    tal:attributes="id python:'main-%s' % (25+25)">
Hello World
</h1>

...that produce...

<h1 class="main" id="main-50">
Hello World
</h1>

Obviously, when you use the same attributes in the static and dynamic part, the dynamic ones take precedence:

<h1 class="main"
    id="main-999"
    tal:attributes="id python:'main-%s' % (25+25)">
Hello Worldd
</h1>

...that produce...

<h1 class="main" id="main-50">
Hello World
</h1>

When I learned this behavior, years ago, I started as attitude to remove all static attributes that later would be overridden by dynamic ones (like in the first example). Why? Because in this way is impossible for a developer to do something stupid like don't realize that the attribute is dynamic and not static.

Today I changed my idea because when you begin to look at someone else code, you will like documentation.

Sometimes when you look at a page template made by someone else you don't need (want) to understand the meaning of every single attribute, but a general idea can be enough:

<a href="http://myhost/site/folder/document-to-be-updated"
   tal:attributes="href view/getSomething">
Update me!
</a>

In the example above, knowing that "view/getSomething" return a valid URL can be understood because the value is used for an href attribute, but what kind of URL? Is an internal URL? External ones? An URL to a specific view?
To know this we need to investigate the getSomething code, or test the application.
This can be avoided if the original template designer documented an example URL using a static href URL that document more or less the general meaning, exactly like done above.

Playing with attributes order
Let's look at this TAL code that generate XML:

<foo attb="value2"
     tal:attributes="attc string:this is C;
                     atta string:this is A;" />

... that produce...

<foo attb="value2"
     attc="this is C"
     atta="this is A" />

When mixing static and dynamic attributes, static ones are rendered first keeping the original order, then all dynamic ones again keeping the order.

In the example above I'd like to see three attributes in the right logic order, but two of them are dynamic:

<foo atta="value1"
     attb="value2"
     attc="value3"
     tal:attributes="attc string:this is C;
                     atta string:this is A;" />

... that produce...

<foo atta="this is A"
     attb="value2"
     attc="this is C" />

So with static attributes you can control also order of dynamic ones.

OK, this is not really interesting (commonly who cares of order of attributes inside generated code?) but sometimes you need to controls exactly the generated code. For example, when you wrote functional Python tests, then you check output.
Omissis
Another important thing to learn is omit an attribute dynamically, based on expression. Yes, because with tal:attributes you can also want to control if an attribute must be added to a node or not.
This is very popular when you want to work with an HTML select element or checkbox:

<select tal:define="values python:[1,2,3,4]">
    <option name="foo:list" value="1"
            tal:repeat="val values"
            tal:attributes="value val;
                            selected python:val==2 and True or False" >
        <span tal:replace="val" /></option>
</select>

... that produce ...

<select>
    <option name="foo:list" value="1">
        1</option>
    <option name="foo:list" value="2" selected="selected">
        2</option>
    <option name="foo:list" value="3">
        3</option>
    <option name="foo:list" value="4">
        4</option>
</select>

As you can note, we simply return True or False, without specifying what kind of value we want for the attribute. This is the way for obtaining the standard XHTML value equals to the attribute name.

Another example is for attributes where we want to controls also the attribute value:

<div tal:define="foo python:2">
    <span tal:attributes="class python:foo==1 and 'fooClass' or None">First</span>
    <span tal:attributes="class python:foo==2 and 'fooClass' or None">Second</span>
</div>

... that produce ...

<div>
    <span>First</span>
    <span class="fooClass">Second</span>
</div>

There we are also controlling the class attribute value: we need to specify its value or return None to omit the attribute.
Using static default
The last tip! I learned this one recently, looking at 3rd party code.

Sometimes you put in the static node attribute the very-common value of an attribute, for example a class:

<ul>
    <li class="veryCommonClass1 veryCommonClass2"
        tal:repeat="foo python:[1,2,3]"
        tal:attributes="class python: foo==2 and 'veryCommonClass1 veryCommonClass2 selected' or 'veryCommonClass1 veryCommonClass2'"
        tal:content="foo" />
</ul>

... that produce ...

<ul>
    <li class="veryCommonClass1 veryCommonClass2">1</li>
    <li class="veryCommonClass1 veryCommonClass2 selected">2</li>
    <li class="veryCommonClass1 veryCommonClass2">3</li>
</ul>

In the example above we used a real static HTML attribute for our nodes (and this is good, also for documentation) but it's annoying to be forced to repeat the whole default value also in the tal:attributes expression.

This can be simplified a bit, using default!

<ul>
    <li class="veryCommonClass1 veryCommonClass2"
        tal:repeat="foo python:[1,2,3]"
        tal:attributes="class python: foo==2 and 'veryCommonClass1 veryCommonClass2 selected' or default"
        tal:content="foo" />
</ul>

... that produce ... the same output!

Seems that the variable default can be used in the tal:attributes expression to get the static value we used.
Unluckily default is not a string but a Python object, so we can't use it also for compose the class value when we need to add also the "selected" class: trying in the expression a default + 'selected' will raise an error.

Thats all! TAL can be a language sometimes tricky! Hopefully we will use all Chameleon in the near future and things will be a little cleaner!

Debugging a slow site using Apache response time and TinyLogAnalyzer

We met recently a new customer, a company that need help with a slow application composed by:

• a Plone CMS
• a set of external processes (a daemon, an e-mail scan system, ...)
• an complex integration with an external database engine
• an integration with two kind of Visual Basic based client desktop application

With the grow of the company bussines they suffer in years a gradually decay of performance.
Some architectural changes were needed, but the company also asked for some quick fix, to get back a faster application while major changes are planned.

Tune up this application was not simple because the whole system is Windows based, so a lot of well-know software we commonly use are not applicable. The company contacted us because the core of the application seems the Plone CMS and the underlying Zope, but we met a customized version of a very old Plone installation. More or less we were facing something that was not Plone.

We decided to start to look at the code (everyone always hope to find some app-go-slow=True flag activated!), but meanwhile we enabled the response time log on the Apache front-end (and it works also on the Windows version!)

How long does it take to serve a request?
You can change the default Apache HTTP log configuration from something like this:

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

...to this:

    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T/%D" combined

Suddently you log format change to this:

    [31/Jan/2008:14:19:07 +0000] "GET / HTTP/1.1" 200 7918 "" "..." 0/95491

...adding the time (in seconds and microseconds) needed to handle the request.

Please note that in a complex installation this will measure the total time needed for the request (if you put a reverse-cache proxy and a load balancer behind your Apache, this will raise the total time) but their Windows installation is very simple: Apache and Zope behind.

How to analyze results?
After a month of collected log data in the new format we wanted to see some statistics. The problem now was: how to read this log to learn something?

After some Google search I was a little worried. There are some real-time monitor tool for log analysis, but we are really scared to try them on a production Windows environment.

The only log analyzer found was apache-response-time: the projects description is really promising, however I was not able to use it: I installed it on a Linux and MacOS system but both I get a segmentation fault error.
Neither a stackoverflow question helped me to find something more. 

So I started to write some few Python lines preparing a custom analyzer. This job became interesting, so I spent a pair of week-ends for code refactoring and to make initial foo.py script is something reusable.

The result was TinyLogAnalyzer, a command line HTTP log analyzer that take a log file with response time enabled and extract two kind of statistics: the top total time and the top average time.

The top total time give you an idea of what is the most time consuming part of the application. In out case we found that more than 20% of the total time was spent performing searches and a special call performed by an external daemon.
Some optimization and changes to this code make the customer a little happier.
Also we find that a lot of requests were done for download external JavaScript resources, never cached.

The top average time instead give you an idea of the slower part of the application. With this I found that object creation and traversing in a special part of the application was really slow due to use of basic (not BTree) Plone folders.

All other release of the log analyzer product you can find are bug-fixing and new features still done for the same customer's problem (meanwhile we collected two more months of log).
For example, I found that some of the external tasks run also during the weekend, so collect data during Saturday and Sunday was wrong. The same for the night time.

Conclusion
The script is all but perfect. For example it keep all data in memory (so it's not the good tool for handle gigabytes-long-file) and read the log from the first line to the last (so if you need to skip some months it is still really slow).

However: it helped me a lot, giving some time to us and some breath to the customer, making things going better while major refactoring are planned.

Apart of the script itself, I think that enabling the Apache response time can be a great source of information when things goes wrong.

Plone, security and workflows: learn how to use "Acquire" option

This article is a new episode of my security and workflow common mistake using Plone.

In the first part, "Plone, security and workflows: when rely on Owner role is bad" I introduced a potential workflow problem, that however can be a real problem only in some kind of organizations.

Today I don't want to focus onto roles, but on the use (or missing use) of the "Acquire" check when designing workflows.
Don't know how to use this checkbox can lead to security problem, especially in conjunction with the View permission.

Is also important to be noted that this problem can be inside current workflows distributed using Plone. This depends on how you and your users think that Plone security must work to fill your needs.

What is the "Aquire" meaning?
You can see the "Acquire" check ("Acquire permission settings?") while defining the workflow permissions page, for every permission managed by the workflow.


The simple meaning is "roles that behave this permission are the ones listed there, plus ones defined on the container of this content".

If the parent itself use a workflow and manage same permissions, it can also use Acquire check again, so acquiring other permissions for roles from it's own parent... and so on.
This continue until the Plone site (or sometimes, up again to Zope root object) where all permissions are defined in the "Security" tab.

Use the Acquire settings without flagging additional roles for a permission is ok: the meaning is "this workflow state don't want to manage this permission". Instead, another state of the same workflow will probably add or change something to the permission.
If every state of a workflow simply use the Acquire check without change something, the customization of this permission in the workflow is useless and can probably be removed.

Also: don't manage a permission for a workflow is still ok. You will never find a workflow that handle all permissions. Only manage the ones you care about (so, for workflow for custom content types, please remove the "Change portal events" permission!).

Out of the box security
More than one Plone standard workflows suffer of what we can define "a not perfect configuration". This do not mean that this is a Plone architectural problem: with Plone you can create great, perfectly working and safe, workflows! Just: the provided ones could be not perfect in some situations and for being able to fix them or create new ones, you need to understand why!
Also, if the workflow is ok or not, depends on how you and your customer think that Plone must work.

The problem is commonly associated with the View permission, however all analysis that follow are the same for every other permissions. We will refer to "Simple Publication Workflow", while will be the same for other like "Intranet/Extranet Workflow".
The "Simple Publication Workflow" is used from all Plone default contents, also by folders.

Let's take a look at the published state permissions setting.


Look in detail at the Anonymous column. The View permission is given at the Anonymous role. And this seems good: you are explicitly saying "every content in this state will be visible by the anonymous". Always. Everywhere.

How can be this bad? The content is public, I want that Anonymous see it!

Let think this applied to folders: users (99% of times, in my experience) see the private and the published state like a room with a door:

• A published folder is like a room with an open door. You can see inside the room itself, and other rooms that follow (if those rooms are open)
• A private folder is like a closed door. You can't see inside it, and you don't care if other doors that follow are open or closed. You don't even know if there are other doors!

Users that move from published to private a main folder that contains a deep tree of subfolder and contents, commonly think that this action will hide all the tree!

You probably already get the point. An explicit check of the View permission for a role is like a magic crystal ball. You don't care of the door: you have the power too see inside.

Going back to Plone world: think about having a Plone folder (folder-1) with a document inside (document-1). If the document is published you can see it using your crystal ball. The review state (even if private) of the containing folder doesn't matter.
This is what is not always clear to users.

• Users can reach http://yourcompany.com/
• Users can't reach http://yourcompany.com/folder-1 (he will get Unauthorized)
• Users can reach http://yourcompany.com/folder-1/document-1

During developer training that customers ask us, one of the argument is always how to customize workflows. Sometimes those customers have already installed a Plone site or intranet by their own so when I explain this, is not uncommon to see surprised faces.

O__o

Another bad attitude: customers never tested the workflow deeply and they think that their contents are is a secure review state just because a main folder is private.

How to fix this
As said above: this is not a Plone technological error. There's a way to get exactly what we want: a set of rooms with doors, with no crystal ball, secret passage or teleport bridge that can transport unwanted users inside the room.

The trick is: don't say that the document is visible explicitly, leave this responsibility to the container (to the room's door).

This is traduced in a workflow configuration that remove explicit permission for Anonymous but use the Acquire power.


Note also an important thing: we don't give the check to Anonymous, but we now check permissions for all other roles that matter. We mean there: all roles that must see the content ignoring if the access door is open or closed.

In this way an anonymous user is able to see the content if he can enter the room (published container folder) and will be no more able to see anything when the door is closed (private container folder).

This is really simple!

When not talking of anonymous access
Somewhere above I said that also the "Intranet/Extranet Workflow" suffer the same configuration problem. Can we fix also this?

This kind of workflow is designed for intranets, but also provide some review state for public contents: it add the "internally published" concept to our Plone site.

Note that this workflow is designed to be used with "Intranet Workflow for Folders" associated with folderish content types.

When using those workflows, we are sometimes referring to the Member roles as we did for Anonymous above.

The default configuration of the "internally_published" state is as follow:


The concept behind this state is "having a published state for intranet users" (so excluding Anonymous).
So we have the same problem for the explicit check of the View permission, this time for Member role.

We can fix this in the same way:


We removed the explicit View permission for Member role while enabling the "Acquire".

This time the cure has some limit.

First problem: if we put an internally_published content in the site root, in our "fixed" version of the workflow we are exposing the content to anonymous access, while the original version protects it (know that you will never remove the View permission for Anonymous role from the site root or Plone will work in unpredictable way).

Another problem: take a look at states defined for "Intranet Workflow for Folders". It's composed by two states: private and internal. If we look at the internal description we read "Visible to all intranet users", so it want to be a "published folder for intranet users".

The private state has no View permission for Member role, while the internal state explicitly check it for site members.
As explained above, this mean that if we put an internal folder inside a private folder, site members will be able to skip the private state of the main container folder and access directly the inner ones and all contents inside.

If we fix this removing the explicit check for Member while activating the "Acquire" check we will have a similar problem to the first ones: anonymous users will be able to access the internal folder in the site root...

There is no perfect solution for this problem using only a good workflow configuration. We need to make some default configuration to the site structure, helping the workflow:

• We fix all "internal published like" states of both workflows as described (using "Acquire")
• We put all internal folder in one (or more) main private folder(s) in the site root
• We give to all users (we can also use the "authenticated users" virtual group) the "can read" check (Reader role) from the sharing tab of this private folder
• We remove from all contents/subfolders inside master private folder(s) the roles inheritance from the sharing tab, removing Reader power to all members
• Inner subfolder and contents will live with normal worfklow security settings

With a configuration like this we can be sure that Anonymous users can't access our contents (as they all live in a private folder) while still be able to control the visibility of a subtree changing the main folder review state (relying on "Acquire").

Think about having a dedicated workflow for folders
Designing a workflow can often be design a set of workflows. This is common; the "Intranet Workflow for Folders" is a perfect example. To use it right, it's better to use also the workflow for folders.

I find often useful to create two different workflow for real contents and folders.
Here a common request I get: Reader role must be able to enter inside private area, but inside this being able to only published contents.

Obviously the workflow I will design will be a workflow that rely onto "Acquire".
While the private state for folders will give explicit View permission to Reader, the private state for contents won't give such power. Done!

Conclusion
Before load your new workflow in a production site or intranet, be sure of what workflow you give to contents... and test it.

A simple way for performing those tests is_ search contents using the site search feature.