Lazy loading on demand in CMS

Context:
Application is responsible for generating pages from templates. User can edit such templates to show data from storage. Data shown on the page are defined by template, each template can show separate set of data (for instance, news, tasks, articles etc.)

Problem:
How to render a user defined template correctly, so any provided by user or application data can be included into template in runtime and have a good performance of rendering?

Solution:
Lets view the problem from MVC point:
View is a template defined by user, and can use any data entered by user (news, blog articles, tasks etc.)
Model is data that should be used by a template engine to render a View.
Controller is a place where user can prepare the Model, that will be sent to a template engine to generate specified View.

As we can’t predict what kind of data are used in any particular View (it could be news, blog articles etc.), the simplest implementation is to fetch all data from storage and push them as a Model to the template engine. In this case template engine will render a View correctly, as all required data are available from the Model. The bad side of this approach is a very slow performance as we need to fetch all data and put them as Model again and again for each request. The more data and different types of data we have to use to populate a Model, the more time this phase will take.

Another approach is to analyze template on save or before render, to fetch all types of data we may need to render it right. For example, before saving a template, we found that it contains a references to blog articles and latest news. Although, we may need to run such operation only once and populate the model only with specified type of data, there is a large number of case not covered by this method: using predefined macros, including other templates and access to data that will be need in runtime, for instance: a single article with specified id, a list of latest 5 articles or anything else that need to be fetched dynamically.

If we think more about the last case, where we need to fetch data dynamically, we will find that it's much easier and better to populate Model not with data, but with data access objects. Of course, in this case we will need to create these data access objects on each request, to pass a context to theme and use them while rendering, it’s still much better (and faster), then populate Model with all data.

When I say a data access object, I don’t mean this object should get the data directly from database or any other storage. But I mean that this object can store request context and can access a restricted set of data, e.g. the list of news posted by current user only or the list of tasks that belongs to current user.

OK, hope your engine template supports methods call and you can get the list of 5 latest news just as simple as saying ${news.latest(5)} or get current user information just with ${user.profile()}.

But here I would like to propose another way to access those less dynamical data. For instance, we need to show user profile information or list of active tasks. Of course, this can be done with appropriate DAO from Model, but there is another way.

This another way helps to save templates clean, - without calling all those DAO methods, but also without prefetching data on the Controller side. This way is based upon a famous Lazy Loading technique.

Although, this may look a bit harder to implement, it brings us a few valuable benefits. First of all, as I said before, code looks clean and easier for user to work with. It's much pleasant to write ${news.latest} than ${news.latest()} and ${user.profile.firstName} than ${user.profile().firstName}.

Second, and it's related to the lazy loading too, we load data once and only when it's needed. The emphasis in on the word 'once'. When we hit our object from the Model first time, it's loading data, which then are available since and to the end of rendering phase. So no matter, from what part of template you access it, it will use already loaded data without touching a storage again. When I say "any part of template", I mean it can be anything available in template engine: macro, function or included template. In this case, you don't need to use a variable or local cache to store loaded data, as you may need to do with data access objects.

Third, lazy loaded data can be complex structures. For instance, we load a current user profile. Profile object is a structure that contains user names, characteristics, address, contact information and a list of followings profiles. Here, address and contact information are embedded objects, which are stored in separate tables/collections and list of followings is a reference to another a few profile objects. If you need to output user full name, you just don't want to load address information, contact information and list of followings. That's where we can also use lazy loading to save user time and server powers. We load address information when need, we load the list of followings when need, we load contact information of some following only when need.


In my next article I’m going to show a simple implementation of lazy loading technique for FreeMarker template engine, which is my favourite one for many years.

Assign <#nested> to variable in FreeMarker

FreeMarker is awesome template engine. I'm still wonder how much powerful it is and how flexible it can be when you need. Last week, when I was writing a simple macro to render tweets using user defined template, my though was that it may be good to define this template as macro nested content. For instance:

  <@tweets 'rkhmelyuk', 10>
      [picture] [username]: <br/>
      [message]<br/>
      <a href="[url]">[date]</a>
  </@tweets>

Pretty simple, but the only way for me was to load tweets using Twitter API through JSONP protocol. So my macro was need to generate a javascript code, that should correctly define callback and fetch tweets. Callback function simply applies received JSON data to the user defined template.

And here is the problem. How to get macro's nested content into javascript variable? Well, who ever used FreeMarker knows that nested content can be rendered with command <#nested/>. This could be a solution, but isn't. Why? Because I need to do extra preparing of template to put it as a javascript string. I need to remove newlines and replace quotes. So the best decision is to put nested content as Freemarker variable and do preparation over it first and than insert into javascript. And here is the power of Freemarker. This was possible and easy to do with <#assign/> or <#local/> command:

 <#local layout><#nested/></#local>
 <#-- Now layout variable contains a template defined as nested -->
 <script type="text/javascript">
    function callback_${id}(data) {
       var layout = "${layout?SomePreparationsGoHere}";
       renderTweets(layout, data);
    }
 </script>

In this code, the nested content is assigned to variable layout. Then we generates a javascript code and insert layout variable as string, but first do some preparations.

I also like flexibility of Freemarker. I hope that could find a time and describe some other tips about it soon.

Freemarker Tips

I like FreeMarker. It's simple, functional and pretty fast tool.
In project we are using Freemaker with SpringMVC.

To work with message bundle are used few macros, one of them is spring.messageArgs. So today I've spent some time on searching how to pass argument to it. There was only one argument and, obviously, I was expecting that this macros could be used in next way:

<@spring.messageArgs "some.message.key" "argumentValue"/>

Unfortunately it's not so easy :(. The macros is calling method that accepts arguments as an array of objects. And string "argumentValue" isn't an array. So I found next simple solution:

<#assign args = ["Ruslan"]/>
<@spring.messageArgs "hello.man" args/>

where

hello.man = Hello {0}!

Another interesting thing about Freemarker is the way you can print numbers. If you want to print number value in argument X as a regular number rather than formatted number, you may do it in next way ${X?c}. I always forget to add ?c suffix and later receive many errors. Fortunately, it's possible to use #{X} that has the same effect. I found that it saves my time.

Code Generation

We are writing the code and this is our job. That's the art that we make. We put our thoughts, our patient, our soul to make it ideal or near to ideal. Sometimes we just write the code asap as there is no time or code is simple or we wrote similar code hundreds of times. This small article is about writing similar code again and again.

When we are writing similar code second or fourth time it's not an art and it is tedious. But that's useful to polish previous code: we can find and fix bugs, add needed logging, rewrite with better performance, make the code clean and handy to read and change etc. Often we are changing previously written code to make it better as well.
But what if you need to write similar code tens or hundreds of times? Will you change previous 20 classes because now you find the best algorithm or fixed the annoying problem with performance? Will you be glad to fix 50 classes over the project because just now you find the bug? (so it must be fixed, right?)
That will take your time, that you could spend on coding something interesting or studying some new popular technology or have a beer with your friends. And what about customer for whom this software is a business.
That's why I'm writing about code generation, the thing we all know and use often. You remember, when you're creating new project or adding new class with your favorite IDE, you got initial code so you can write your important part rather that spend time on writing the same code again and again.
While reviewing code generation tools we can divide them into external and internal. External are provided by IDE or other providers (remember famous xdoclet toolkit?). Internal is written by your team and used withing some one or few projects.
Internal code generators could be simple enough to generate only basic template source code or UI that will be changed and extended. It also can be large and complex to generate layers (domain types, DAOs and repositories etc) of your software.

Lets review next types of code generators:
1. template code generators
2. partial code generators

Template code generators run once at the begin. They are responsible for generating source code that will be rewritten or extended by developer. It's important to have clean generated code with comments for generated code and for placements where to put your code. It's good to use such generators if you have already ideal code to be generated. Ideal means that you will not need ever to change template and regenerate the same code again and again. Regenerating code will remove code written by developers with hands.

Partial code generators could be run as much as you will need. They are separated from code written by developers. For C# that would be good to use partial keyword (as I remember partial class declaration was added to have distinct generated and written by developer code for WinForms and ASP.NET); for Java extending could be used as well. We use generator to generate only similar code and put in another file, e.g. partial class definition or base abstract class. The specific code is written in derived class. While we find new bugs and improvements we need just to change templates or generator configuration and regenerate the code again.

Although code is generated it also must be easy to read it. Document your code, as you need to do it only once - in templates, so don't be lazy and stingy! For partial code generator or template code generator that is twice as important, as you are working directly with generated code.
Don't forget to add comment with notion that code is generated automatically and can be regenerated again, so other developers will think twice before changing it by hands!

Also remember that it's okay to add code generation to project step-by-step. For example, I'm the only who is using the new generator for now. I need to do so to find and fix defects, find the parts of code that could and must be generated, improve configuration and way of use. In this case the generated base code (with partial code generator) is commited too.

Use the best tools to write templates. For example I'm using Freemarker template engine to describe templates and generate source code. There are few code generators that read configuration file and generate appropriate code. For such tools performance is not as important as flexibility of templates and configuration.

Document the generator. Share source code within team. Put your generator into source code repository near to the project, but not into it. Also make and tag latest stable version as runnable scripts or program, so other developers can update and use it without spending on it additional time.

Spring MVC + FreeMarker + Tiles 2

This is extended and translated into English version of my post Spring MVC + FreeMarker + Tiles 2. I've found that many are interested about the joint work of those frameworks.
Well, this article shows how to configure the Spring MVC application with Tiles 2 and FreeMarker. You may don't use the SpringMVC framework, and in this case you need to make by your hands all the configuration work that Spring MVC does for you.
As you may know, Freemarker is popular template engine write on Java and used by mostly by Java applications. Tiles 2 is also Java frameworks used to prepare document using one or more "tiles" - the parts that are defined separately but used to be generated at one document. For example, using tiles you can define site header/footer, news or post blocks at your site etc.
So, how did I get it working?
Simple!

In file web.xml add the freemarker configuration.

<servlet>
        <servlet-name>pages</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
</servlet>

 <servlet-mapping>
        <servlet-name>pages</servlet-name>
        <url-pattern>*.page</url-pattern>
 </servlet-mapping>

<servlet>
        <servlet-name>freemarker</servlet-name>
        <servlet-class>freemarker.ext.servlet.FreemarkerServlet</servlet-class>

        <init-param>
            <param-name>TemplatePath</param-name>
            <param-value>/</param-value>
        </init-param>
        <init-param>
            <param-name>NoCache</param-name>
            <param-value>true</param-value>
        </init-param>
        <init-paramv
            <param-name>ContentType</param-name>
            <param-value>text/html</param-value>
        </init-param>

        ....

        <load-on-startup>1</load-on-startup>
</servlet>

<servlet-mapping>
        <servlet-name>freemarker</servlet-name>
        <url-pattern>*.ftl</url-pattern>
</servlet-mapping>

And here is the important part *-servlet.xml configuration:

<!-- Tiles 2 configuration-->
<bean id="tilesConfigurer" class="org.springframework.web.servlet.view.tiles2.TilesConfigurer">
        <property name="definitions">
            <list>
                <!-- the tiles configuration files are declared here --> 
                <value>/WEB-INF/defs/general.xml</value>
                <value>/WEB-INF/defs/secure.xml</value>
            </list>
        </property>
</bean>

<bean id="tilesViewResolver" class="com.javaua.spring.integration.ext.ExtUrlBasedViewResolver">
    <property name="viewClass" value="com.javaua.spring.integration.ext.ExtTilesView"/>
    <property name="exposeSpringMacroModel" value="true"/>
</bean>

<bean id="viewResolver" class="org.springframework.web.servlet.view.freemarker.FreeMarkerViewResolver">
    <property name="cache" value="true"/>
    <property name="prefix" value=""/>
    <property name="suffix" value=".ftl"/>
    <!-- if you want to use the Spring FreeMarker macros, set this property to true -->
    <property name="exposeSpringMacroHelpers" value="true"/>
</bean>

<!-- Controller that returns view = /layout/header -->
<bean name="/layout/header.page" class="com.javaua.HeaderController"/>

Also show here the part of general.xml file where Tiles definitions are set up.:

<definition name="home" extends="default">
        <put-attribute name="title" value="Home Page"/>
        <put-attribute name="body" value="/templates/home.ftl"/>
</definition>
<definition name="default" template="/templates/main.ftl">
        <put-attribute name="header" value="/layout/header.page"/>
        <put-attribute name="footer" value="/templates/layout/footer.ftl"/>
</definition>
<definition name="/layout/header" template="/templates/layout/header.ftl"/>

Source code of com.javaua.spring.integration.ext.ExtUrlBasedViewResolver class:

import org.springframework.core.Ordered;
import org.springframework.web.servlet.View;
import org.springframework.web.servlet.view.AbstractUrlBasedView;
import org.springframework.web.servlet.view.UrlBasedViewResolver;

public class ExtUrlBasedViewResolver extends UrlBasedViewResolver implements Ordered {

    private boolean exposeSpringMacroModel = false;

    public void setExposeSpringMacroModel(boolean exposeSpringMacroModel) {
        this.exposeSpringMacroModel = exposeSpringMacroModel;
    }

    protected View loadView(String viewName, Locale locale) throws Exception {
        AbstractUrlBasedView view = buildView(viewName);
        View viewObj = (View) getApplicationContext().getAutowireCapableBeanFactory().initializeBean(view, viewName);

        if (viewObj instanceof ExtTilesView) {
            ExtTilesView tilesView = (ExtTilesView) viewObj;
            tilesView.setExposeSpringMacroModel(exposeSpringMacroModel);
        }

        return viewObj;
    }

Source code of com.javaua.spring.integration.ext.ExtTilesView class:

import org.springframework.web.servlet.view.tiles2.TilesView;
import org.springframework.web.servlet.view.AbstractTemplateView;
import org.springframework.web.servlet.support.RequestContext;

import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.ServletException;
import java.util.Map;


public class ExtTilesView extends TilesView {

    private boolean exposeSpringMacroModel = false;

    public void setExposeSpringMacroModel(boolean exposeSpringMacroModel) {
        this.exposeSpringMacroModel = exposeSpringMacroModel;
    }

    @SuppressWarnings("unchecked")
    protected final void renderMergedOutputModel(Map model, HttpServletRequest request,
                                                 HttpServletResponse response) throws Exception {

        if (exposeSpringMacroModel) {
            if (model.containsKey(AbstractTemplateView.SPRING_MACRO_REQUEST_CONTEXT_ATTRIBUTE)) {
                throw new ServletException(
                        "Cannot expose bind macro helper '" +
                        AbstractTemplateView.SPRING_MACRO_REQUEST_CONTEXT_ATTRIBUTE +
                        "' because of an existing model object of the same name");
            }
            model.put(AbstractTemplateView.SPRING_MACRO_REQUEST_CONTEXT_ATTRIBUTE,
                    new RequestContext(request, getServletContext(), model));
        }

        super.renderMergedOutputModel(model, request, response);
    }
}

Why we need those two classes? Simple enough. Class ExtTilesView exposes Spring macro model which is required when using standard macros used to support work with Freemarker and Velocity typelate frameworks. And the class ExtUrlBasedViewResolver helps us to prevent view-not-found-error. Because, standard UrlBasedViewResolver is used for getting tiles view, it will throw an error when view with specified name is not found. In my case, I have two view resolvers and each will throw an error if view can't be found.
And FreeMarkerViewResolver will request for the file, while tiles requests just for XML definition. That's why I've ran the tiles view resolver as first one.