Android Emulator Snapshot

Here I'm going to describe a simple way to decrease a start up time of the Android Emulator. This is possible after snapshot saving and loading option is enabled.

Open your terminal and enter the command:

$ android

Open Android Virtual Device Manager from menu Tools -> Manage AVDs... You'll see a window like this:


AVD Manager


Select your working AVD from the list and click on "Edit" button. In the opened dialog check on to enable Snapshot. Save changes. Then click on the "Start..." button and check "Launch from snapshot" and "Save to snapshot" options.


Launch options


If you haven't enabled "Saving to snapshot" option before, it may take a time to start the emulator first time. After it is started, you can close it and start again. This time emulator will be started much faster.

MOBI version of The Architecture of Open Source Applications book

The Architecture of Open Source Applications is a great book where one can find the description of the architecture of the 25 open source applications, like Eclipse, LLVM, Mercurial, HDFS and Berkley DB. This book is free to read in the HTML format online, but if you want to get a PDF or Kindle version, you'll need to buy it either at Lulu.com or at Amazon.com.

Of course the recommended way is to buy the book, as all royalties from these sales will be donated to Amnesty International. But you can download a MOBI version for Kindle of this book for free, just continue reading.

First, want to tell a short story. After I found a free SICP version for Kindle compiled from HTML files at github.com/twcamper/sicp-kindle, I was looking for a chance to create something like that myself. And then I found the AOSA book. It has an HTML version, and I wanted to have a MOBI version to read it in my Kindle.

And so I made a Kindle version of The Architecture of Open Source Applications book and currently it's available from github.com/rkhmelyuk/aosa-mobi.

Download a Kindle version of the AOSA

SyncTab

I've released today an updated version of a SyncTab Android application.

Well, a few words about what is a SyncTab. SyncTab is an application to send an interesting links and browser tabs from Android phone to desktop browser (current only Chrome is supported). To start using SyncTab, user need only to download and install Android application and Chrome extension. User will need to register an account using either Android application or Chrome extension and then he/she can easily send tabs from phone to desktop. There is more information about this in the Getting Started article.

So what's new in the version 0.6? Not much:
  • added an about page to show current user name, website link, link to submit issue or contact
  • get the correct page title and description with regards to the page charset

But I have more plans for version 0.7:
  • share tabs from browser to the phone
  • tag and filter shared tabs

So try SyncTab now, and wait for a new features.

Mobile website with jQuery Mobile

In my previous post I've described how to add support of mobile version of website using django. In this post I'm going to stop on front-end side of mobile website.

From the beginning, I wanted to create a simple mobile version from scratch and use a similar to original website look. But, after some investigation, I've changed my plans. I decided to use some framework to create a mobile website. So I found two popular such frameworks: Sencha Touch and jQuery Mobile. Whilst Sencha Touch looks awesome and production ready, while jQuery Mobile is still in release candidate stage, I stopped on the second one. This choice was very simple for me: jQuery products usually are qualitative, there is a strong community behind them and they introduce new versions very quick. Nevertheless Sencha Touch is a good product too.

I'm not going to describe how I used jQuery Mobile step by step, but only list of some facts and tips. First of all, I was using jQuery Mobile version 1.0 RC2, and that means, it wasn't a production-ready version yet. Even so, I found a few bugs or maybe just a things that need an improvement.

I just love such structures as lists. So will list a fact using a list =).
  • You should read a manual first and follow a standard layout mechanism, like declare header, content and footer sections. This will save you a lot of time.


    <div data-role="page">
               <div data-role="header">
                   <a href="{% url m_home %}" data-icon="home" class="ui-btn-left">Home</a>
                   <a href="{% url m_help %}" data-icon="help" class="ui-btn-right">Help</a>
                  {% block header %} <h1>[App Name]</h1>{% endblock %}
               </div>
    
               <div data-role="content">
                {% block content %}{% endblock %}
               </div>
    
               <div id="footer" data-role="footer">  Copyright © 2011 ... </div>
           </div>
    

    It's a good practice to add Home/Back and Help buttons to the header or footer panel. The navigation bar can be also added to the header, where it looks very natural:


    <div data-role="header">
       <a href="{% url m_home %}" data-icon="home" class="ui-btn-left">Home</a>
       <a href="{% url m_help %}" data-icon="help" class="ui-btn-right">Help</a>
       {% block header %}<h1>[AppName]</h1>{% endblock %}
    
       <div data-role="navbar">
          <ul>
             <li><a href="{% url m_download %}">Download</a></li>
             <li><a href="{% url m_contact %}">Contact</a></li>
             ...
          </ul>
       </div>
    </div>
    
  • Fixed position of header and footer isn't working as it's expected. Well, expected it to be fixed always, while content is scrolling. But currently it just disappears when user starts scrolling and shows back after a second or so scrolling is ended.
  • Hash anchor links (#something) aren't working yet.
  • Can't show an image as a page, instead shows an "undefined" text. If you have a links that reference images, you must specify rel="external" for them.
  • Use header to specify page title, rather than do it in the h1 element in the content section.
  • Use collapsible blocks for long pages. For instance, I've found them very helpful to organize a help page with many different topics.

    You may also want to embrace collapsible blocks using collapsible set.
  • For large image, don't show original version of it on page, but prefer thumbnail image as a link to the original version of image.
  • Use list view where possible. For instance, a main help page contains a list of articles which are references to the appropriate pages.
  • Buttons should be with icons where possible. You can easily add icon to the button with the data-icon="[icon]" attribute. The list of available icons is on jQuery Mobile website.
  • Make website as simple as possible: structure should be simple, navigation bar should contain only the most used links. Remember, that it's still hard to work fully with websites on mobiles.

Well, that's all as for now. Hope that would be helpful.

Mobile website with Django

Although I'm not a web guy, but more specialized on backend logic, I had a very interesting experience today creating a website version for mobiles. I had a simple website with a few pages, minimum server side logic and a slice of documentation there. This website perfectly works for desktop and tablet browsers, but is not that good for mobiles. I use to call this website as "normal" version throughout this post. It's important to have a working mobile version of website, as website is about Android application.

A little bit about technologies and task. Website is running on Django as well as mobile version should be too. Mobile version of website should be much simpler than normal version, not so beautiful, but must do own job well: allow users to know about the product, download and access to the documentation.

I decided to use the same django application for both normal and mobile website versions. I found a few different ways to add mobile website, for example: redirect to another django application, replace TEMPLATE_DIRS setting with a version that points to mobile templates if user is using mobile browser or just check and use different logic/templates in every view method. As I wanted to have a different structure for mobile website, I decided to go by my own way.

Mobile website path should be prefixed with /m/ rather than be the same or use a different domain name. So, if normal website download page is http://example.com/download/, then mobile download page should be http://example.com/m/download/. Django should check every user request whether it's from mobile browser or not using a User-Agent metadata, and if so, redirect to the mobile version of the page. In other words, if user opens a /download/ page from mobile browser, then she will be redirected to the /m/download/ page. But if she opens a /m/download page, then no redirects happen - just a mobile version of page is returned to user.

I've created different templates and views for mobile version of website. Normal version of website has references only to the normal pages and mobile version of website has references only to the mobile pages. This was done to avoid not necessary redirects that could slow down a work with website. Thus, I had a different version of URL patterns for website in project's urls.py:


urlpatterns = patterns('website.views',
    url(r'^$', 'home', name='home'),
    url(r'^download/$', 'download', name='download'),
    url(r'^screenshots/$', 'screenshots', name='screenshots'),
    url(r'^contact/$', 'contact', name='contact'),
    ... 
)

urlpatterns += patterns('website.mobile',
    url(r'^m/$', 'home', name='m_home'),
    url(r'^m/download/$', 'download', name='m_download'),
    url(r'^m/screenshots/$', 'screenshots', name='m_screenshots'),
    url(r'^m/contact/$', 'contact', name='m_contact'),
    ...
)
As you see, website.views are responsible for handling normal website pages, while website.mobile for handling mobile website pages.

Now need to redirect mobile browser users to the mobile version of website. For this purpose, I wrote a custom middleware called MobileWebsiteMiddleware. The task of this middleware is to check if user is from a popular mobile platform and needs to be redirected to the mobile version of website. The other responsibility is to redirect users from mobile to the normal version of website in case if request isn't from mobile browser. This is an extract option, that should be disabled while testing.


class MobileWebsiteMiddleware(object):

    MOBILE_PREFIX = '/m/'
    MOBI_REG = re.compile(
        '(iphone|windows ce|mobile|phone|symbian|mini|pda|ipod|mobi|blackberry|playbook|vodafone|kindle)',
        re.IGNORECASE)

    def process_request(self, request):

        if 'HTTP_USER_AGENT' in request.META:
            userAgent = request.META.get('HTTP_USER_AGENT')
            matches = self.MOBI_REG.search(userAgent)
            path = request.path_info

            if matches:
                # from mobile browser, check if need to redirect to mobile
                if not path.startswith(self.MOBILE_PREFIX):
                    # need to redirect to mobile version
                    return HttpResponseRedirect('/m' + path)
            elif path.startswith(self.MOBILE_PREFIX):
                    # need to redirect to normal version
                    return HttpResponseRedirect(path[2:])

        return None

If you'll check the list list of user agent patterns in MOBI_REG you wouldn't find neither iPad nor Android. I want to show a "normal" website for tablets. "Android" keyword is used in User Agent for both mobiles and tablets. While Mobile Android browsers pass a User Agent with a "mobile" keyword, but Tablet Android browsers don't do that. Thus, I will have a normal version for desktop and tablets browsers, but mobile version for mobiles and Kindle.

Now I have a simple website, that has two different version: one for desktop and tablet browsers and another for mobile browsers. User will be automatically redirected to the right version of website which depends on used device.

In my next post I'm going to tell more about using jQuery Mobile to build a mobile version of website.

Django: use global variables on rendering templates

Problem

Some settings or global value is used in every template or, for example, in base layout template. This value should be passed as context variable to each template. I call such value a global context variable in this post.

For instance, it can be a Google Analytics Code, which should be used in production, but not in staging or development environments. In this case, it’s fine to move GA code to the settings file (of course, if different settings used for different environments). Though need to find a way to get the value from the settings and pass it to the template context before rendering.

Solutions

There are a few simple solutions that solve the described problem:
  1. Add global variable to the template context in every view. But according to DRY and good sense this is a bad solution.
  2. Write a tag to fetch setting value by name and output it. Nice solution and flexible enough in case we need to show different settings in different pages. The only disadvantage is fetching setting value each time it’s accessed. But in most cases this is a minor.
  3. Write a context processor. In the documentation said that context processor is a method that takes a request object as argument and returns a dictionary of items to be merged into the context. In other words, context processor returns a dictionary of items that will be available in template on render phase. In our case, we can get setting value and return from our own context processor method.

    from django.conf import settings
    
    def global_vars(request):
        return {'GA_CODE': settings.GA_CODE}
    
    Also, need to declare our context processor in settings.TEMPLATE_CONTEXT_PROCESSORS:

    TEMPLATE_CONTEXT_PROCESSORS = (
       . . . 
       'synctab.website.context_processors.global_vars',
    )
    
    From now, GA_CODE context variable can be used in any template, and will be processed correctly.

In my opinion, the best decision for fixed list of globally accessed template context variables is a custom context provider. It's also a right way, if most or all templates use same global variables. Otherwise, it's good to write a few tags to cover access to the global state, like settings.

tmpfs: work with your data even faster

Currently improving IDEA performance by copying cache files into the RAM using tmpfs. Actually, tmpfs and ramfs are good ideas.

As described in Wikipedia: tmpfs is intended to appear as a mounted file system, but stored in volatile memory instead of a persistent storage device. Simply put, when you copy a file to such FS, this means that you copy a file to the RAM. When you create a file in this FS, this means you create a file in RAM. When you delete a file in this FS, this means you delete a file in a RAM. And so on.

The negative side of tmpfs is that it's not backed by any storage: on restart or system crash you'll lost your files and data stored in tmpfs. But on system start you can either copy files from the disk to the memory again and continue to work.

In case of IDEA cache, I will need to write a simple script that periodically copies the cache from tmpfs to the disk. So, on restart, I can simply restore cache, and don't need to wait while re-caching is done.

So, how to create a tmpfs storage? It's pretty easy to do with next commands. Here I create an empty directory tmp and mount it as tmpfs filesystem.
# mkdir ~/tmp
# mount -t tmpfs -o size=500m tmpfs ~/tmp
Option size=500m limits memory usage to 500m. tmpfs also supports flushing content to the swap when need. This is one of the main differences between tmpfs and ramfs. ramfs is not limited and can grow dynamically, and the used memory can't be freed or flush to swap. The negative side of such dynamic nature of ramfs is that system can hung when no free memory left.

To read from and write content to RAM is much much faster than to do the same operations with a file on disk. It's pretty good optimization if you need to support read-only content or content that can easily be restored when need.

Such type of content is a hosted website. You can decrease page or resource loading time by moving them from hard disk to the memory filesystem.
 # mkdir /var/www/www.example.com
 # mount -t tmpfs -o size=50M tmpfs /var/www/www.example.com
 # cp -R /home/www/www.example.com/* /var/www/www.example/com

To mount tmpfs automatically on system load, you will need to add another record to yours /etc/fstab configuration file. The only you need to do now is execute next command on system start:
 # cp -R /home/www/www.example.com/* /var/www/www.example/com

As a summary, tmpfs is a good way to work with large amount of files or data that need to be accessed or processed quickly. Such data also is either read-only or can be easily recreated. Samples of such data are static websites, website resources, temporary cache etc. When need to process large amount of data, you can also split it into small pieces and process each one by one. tmpfs is also takes a limited amount of RAM, and can increase over that amount.

Performance Optimization for Android

Here is the list of the performance optimization tips when developing Android applications:

  • Avoid creating objects if you don’t need them.
  • Avoid memory allocation if you can work without it.
  • Array of ints is preferred than array of Integers.
  • Two parallel arrays of ints are a lot more efficient than an array of (int,int) objects. Two parallel arrays of Foo and Bar are a lot more efficient than an array of (Foo,Bar) objects.
  • Avoid creating short-term temporary objects if you can.
  • Make your method static. Invocations will be about 15%-20% faster. It's also good practice, because you can tell from the method signature that calling the method can't alter the object's state.
  • It's reasonable to follow common object-oriented programming practices and have getters and setters in the public interface, but within a class you should always access fields directly.
  • Direct field access is about 7x faster than invoking a trivial getter.
  • Use static final for constants.
  • Prefer for-each loop.
  • Declare fields and methods accessed by inner classes to have package access, rather than private access.

Read Designing for Performance article from Android Dev Guide to find more details and samples.

List also available as gist.

Rename of GitHub account and Deploy Keys

Today I've fixed failed builds that were caused by renaming GitHub account. Bamboo couldn't pull the changes from GitHub master repository after rename.

So, I corrected the account name in the .git/config file for origin remote. But this didn't help - I had same the same problem to pull the changes.

That was a security problem: GitHub denied access to the repository, because I don't have access to the old repository (of course, that because account name was changed).

After some investigation, I decided to drop existing Deploy Key and create the same again.
And Voilà! This small manipulation fixed the problem with access to the GitHub repository and now builds are green again.

Hightlights: How Browsers Work

This is a short summary of great article How Browses Work by Tali Garsiel. My opinion that it is a must-read article not only for web developers, but for everyone who want to know how browsers work.

I’m not a web developer, but was interesting in this article to know more about how browsers work inside. This article can help to understand the process of rendering html page, loading and processing CSS and JavaScript files by browser.

Here I list of my highlights after reading How Browsers Read article:
browser-render-workflow

As you can see, browser needs to build a few different trees: DOM tree, rendering tree, style rules tree, style context tree etc.

The rendering engine will start getting the contents of the requested document from the networking layer. This will usually be done in 8K chunks.

To render a page faster, browser doesn’t wait while all page and css and scripts are loaded. Instead browser loads a chunk of html code, parse it and build a DOM and rendering trees. Then rendering tree is used to paint the page, so user can see it. If we know the size of chunk, we can predict how fast is html page processed. If the html page size is under 8K (or whatever specific browser limit), we should expect it to load much faster.

For better user experience, the rendering engine will try to display contents on the screen as soon as possible. It will not wait until all HTML is parsed before starting to build and layout the render tree.
The algorithm is expressed as a state machine. Each state consumes one or more characters of the input stream and updates the next state according to those characters. The decision is influenced by the current tokenization state and by the tree construction state. This means the same consumed character will yield different results for the correct next state, depending on the current state.

Special state machine used to parse HTML document. I was interesting how they parse HTML, because it’s pretty hard to parse such forgiving language. From this article it’s clear that state machine is used to parse the document. In article described:

The input to the tree construction stage is a sequence of tokens from the tokenization stage The first mode is the"initial mode". Receiving the html token will cause a move to the "before html" mode and a reprocessing of the token in that mode. This will cause a creation of the HTMLHtmlElement element and it will be appended to the root Document object. The state will be changed to "before head". We receive the "body" token. An HTMLHeadElement will be created implicitly although we don't have a "head" token and it will be added to the tree.

Browsers use stack to order tags correctly and fix a mix of opened/closed tags. Opened tags are added to the stack, and when they are closed and corrected, they are inserted into the DOM tree. As result DOM tree is always in correct state. Different browsers handle new DOM elements differently: either provide dirty bit or fire events. While parsing document DOM elements are processed and added to the Rendering tree, that becomes a source for painting phase.

When the parsing is finished browser will mark the document as interactive and start parsing scripts that are in "deferred" mode - those who should be executed after the document is parsed. The document state will be then set to "complete" and a "load" event will be fired.
The model of the web is synchronous. Authors expect scripts to be parsed and executed immediately when the parser reaches a <script> tag. The parsing of the document halts until the script was executed. If the script is external then the resource must be first fetched from the network - this is also done synchronously, the parsing halts until the resource is fetched. This was the model for many years and is also specified in HTML 4 and 5 specifications. Authors could mark the script as "defer" and thus it will not halt the document parsing and will execute after it is parsed. HTML5 adds an option to mark the script as asynchronous so it will be parsed and executed by a different thread.

Asynchronous scripts can be loaded faster maybe even before page is rendered, because browser does not need to wait to load it. Still there is a single rendering thread, so after script is loaded, any changes to the tree will be applied and processed by main thread.

While executing scripts, another thread parses the rest of the document and finds out what other resources need to be loaded from the network and loads them. This way resources can be loaded on parallel connections and the overall speed is better. Note - the speculative parser doesn't modify the DOM tree and leaves that to the main parser, it only parses references to external resources like external scripts, style sheets and images.

This is how browser can parse the document and load resources, like CSS, scripts and images.

The style contexts contain end values. The values are computed by applying all the matching rules in the correct order and performing manipulations that transform them from logical to concrete values. For example - if the logical value is percentage of the screen it will be calculated and transformed to absolute units. The rule tree idea is really clever. It enables sharing these values between nodes to avoid computing them again. This also saves space.

As it said in article, handling style correctly was made very clever. A tree structure and other optimizations help to handle correctly styles inheritance and large number or styles properties.

In the article also said:

All the matched rules are stored in a tree. The bottom nodes in a path have higher priority. The tree contains all the paths for rule matches that were found. Storing the rules is done lazily. The tree isn't calculated at the beginning for every node, but whenever a node style needs to be computed the computed paths are added to the tree.

As result, real values (like height, margin etc) for each style are not computed till they needed.

After parsing the style sheet, the rules are added one of several hash maps, according to the selector. There are maps by id, by class name, by tag name and a general map for anything that doesn't fit into those categories. If the selector is an id, the rule will be added to the id map, if it's a class it will be added to the class map etc. This manipulation makes it much easier to match rules. There is no need to look in every declaration - we can extract the relevant rules for an element from the maps. This optimization eliminates 95+% of the rules, so that they need not even be considered during the matching process.

Hash maps are used as indexes. This is another performance optimization.

Before repainting, webkit saves the old rectangle as a bitmap. It then paints only the delta between the new and old rectangles.

Well, another performance and usability optimization.

The browsers try to do the minimal possible actions in response to a change. So changes to an elements color will cause only repaint of the element. Changes to the element position will cause layout and repaint of the element, its children and possibly siblings. Adding a DOM node will cause layout and repaint of the node. Major changes, like increasing font size of the "html" element, will cause invalidation of caches, relyout and repaint of the entire tree.
Expected optimization - the larger change the more relayout and repaint is needed.

Get Total Commits Number in Git

In a few weeks there will be an anniversary as I've started working on current project. So I decided to find out the total number of made commits from the project begin. While it's obvious for Subversion, in Git you need something more that just look at the revision number.

First, I came out with a simple command that prints the total number of commits:

$ git log --pretty=format:'%h' | wc -l
prints out
2485

This is a bit primitive and that wouldn't be such a simple solution if need to find the total number of commits per author. I didn't want to believe there is not better way to get the total number of commits per author with standard Git tool.

After a few minutes of research I found there is a simple and powerful util git shortlog.

So next command

$ git shortlog -n -s
prints out
  2317  Ruslan Khmelyuk
   104  Author_1
    53  Author_2
     9  unknown
     2  Author_3

I've summed the commit numbers for each committer and found that the total commits number by git shortlog command is 2485 and that is equal to the result of git log | wc -l command. While git rev-list -all | wc -l command prints there are 2488 commits in the repository. Strange.

Short investigation helped me to understand that both commands working fine and show correct number. The difference between commands

$ git log --pretty=format:'%h' | wc -l
$ git shortlog -n -s
and
$ git rev-list --all | wc-l

is in the source they're using to calculate commits list.

git log and git shortlog use current branch commits log, while git rev-list uses all repository as source of commits list. I had 3 more commits in the other branch, so in my local repository I had in total 2488 commits.

Even more, If you need to calculate the total number of commits in the remote repository, you will need to use git rev-list command:

$ git fetch origin
$ git rev-list --remotes | wc -l

Thus I can say, that git rev-list command works best to find out the total number of made commits.

Lazy loading on demand in CMS

Context:
Application is responsible for generating pages from templates. User can edit such templates to show data from storage. Data shown on the page are defined by template, each template can show separate set of data (for instance, news, tasks, articles etc.)

Problem:
How to render a user defined template correctly, so any provided by user or application data can be included into template in runtime and have a good performance of rendering?

Solution:
Lets view the problem from MVC point:
View is a template defined by user, and can use any data entered by user (news, blog articles, tasks etc.)
Model is data that should be used by a template engine to render a View.
Controller is a place where user can prepare the Model, that will be sent to a template engine to generate specified View.

As we can’t predict what kind of data are used in any particular View (it could be news, blog articles etc.), the simplest implementation is to fetch all data from storage and push them as a Model to the template engine. In this case template engine will render a View correctly, as all required data are available from the Model. The bad side of this approach is a very slow performance as we need to fetch all data and put them as Model again and again for each request. The more data and different types of data we have to use to populate a Model, the more time this phase will take.

Another approach is to analyze template on save or before render, to fetch all types of data we may need to render it right. For example, before saving a template, we found that it contains a references to blog articles and latest news. Although, we may need to run such operation only once and populate the model only with specified type of data, there is a large number of case not covered by this method: using predefined macros, including other templates and access to data that will be need in runtime, for instance: a single article with specified id, a list of latest 5 articles or anything else that need to be fetched dynamically.

If we think more about the last case, where we need to fetch data dynamically, we will find that it's much easier and better to populate Model not with data, but with data access objects. Of course, in this case we will need to create these data access objects on each request, to pass a context to theme and use them while rendering, it’s still much better (and faster), then populate Model with all data.

When I say a data access object, I don’t mean this object should get the data directly from database or any other storage. But I mean that this object can store request context and can access a restricted set of data, e.g. the list of news posted by current user only or the list of tasks that belongs to current user.

OK, hope your engine template supports methods call and you can get the list of 5 latest news just as simple as saying ${news.latest(5)} or get current user information just with ${user.profile()}.

But here I would like to propose another way to access those less dynamical data. For instance, we need to show user profile information or list of active tasks. Of course, this can be done with appropriate DAO from Model, but there is another way.

This another way helps to save templates clean, - without calling all those DAO methods, but also without prefetching data on the Controller side. This way is based upon a famous Lazy Loading technique.

Although, this may look a bit harder to implement, it brings us a few valuable benefits. First of all, as I said before, code looks clean and easier for user to work with. It's much pleasant to write ${news.latest} than ${news.latest()} and ${user.profile.firstName} than ${user.profile().firstName}.

Second, and it's related to the lazy loading too, we load data once and only when it's needed. The emphasis in on the word 'once'. When we hit our object from the Model first time, it's loading data, which then are available since and to the end of rendering phase. So no matter, from what part of template you access it, it will use already loaded data without touching a storage again. When I say "any part of template", I mean it can be anything available in template engine: macro, function or included template. In this case, you don't need to use a variable or local cache to store loaded data, as you may need to do with data access objects.

Third, lazy loaded data can be complex structures. For instance, we load a current user profile. Profile object is a structure that contains user names, characteristics, address, contact information and a list of followings profiles. Here, address and contact information are embedded objects, which are stored in separate tables/collections and list of followings is a reference to another a few profile objects. If you need to output user full name, you just don't want to load address information, contact information and list of followings. That's where we can also use lazy loading to save user time and server powers. We load address information when need, we load the list of followings when need, we load contact information of some following only when need.


In my next article I’m going to show a simple implementation of lazy loading technique for FreeMarker template engine, which is my favourite one for many years.

Neverending Groovy: compareTo() simple as it can be with <=>

Groovy introduced a new operator to compare two values: <=>.

When you use this operator for 2 values a <=> b, Groovy translates this to code something like that a.compareTo(b). Well that's not the true, as it also can handle null values correctly. So you wouldn't get NPE, when you compare value with null or vice versa. For instance, next script:


a = null
b = 12

println "a.compareTo(b) = " + (a <=> b)
println "a.compareTo(a) = " + (a <=> a)
println "b.compareTo(b) = " + (b <=> b)

outputs

a.compareTo(b) = -1
a.compareTo(a) = 0
b.compareTo(b) = 0

where we can see, that null values are handled correctly, without any runtime exceptions.

With such operator we could write really nice and simple code to allow multiple comparisons for object values.

Lets we have a class Person with firstName, lastName and age data:


class Person {
String firstName
String lastName
int age
}


And we need to sort a list of Persons by age, than by name (first by last name, than by first name). Assume, this is standard sorting mode, so we may prefer to implement Comparable interface:



class Person implements Comparable {
String firstName
String lastName
int age

int compareTo(Person other) {
...
}
}


Method compareTo() isn't implemented yet. So lets try to implement in Java first. It would look something like this:


int compareTo(Person other) {
if (other == null) {
return 1;
}

int result = 0;
if (this.age > other.age) {
result = 1;
}
else if (this.age < other.age) {
result = -1;
}
if (result == 0) {
if (this.lastName != null && other.lastName != null) {
result = this.lastName.compareTo(other.lastName);
if (result == 0) {
if (this.firstName != null && other.firstName != null) {
result = this.firstName.compareTo(other.firstName);
}
else {
result = this.firstName != null ? 1 : -1;
}
}
}
else {
result = this.lastName != null ? 1 : -1;
}
}
return result;
}


Hm... Looks ugly! Too much if's, too much checks, too much 0, 1 and -1. Hard to see the logic behind the code ((.

And here how this can be written with Groovy's <=> operator:


int compareTo(Person other) {
age <=> other.age ?: lastName <=> other.lastName ?: firstName <=> other.firstName
}


And a simple test:


def john = new Person(firstName: 'John', lastName: 'Doe', age: 25)
def mark = new Person(firstName: 'Mark', lastName: 'White', age: 25)

print "john <=> mark = " + (john <=> mark)

prints

john <=> mark = -1


As for me, it looks wonderful.

"A Story of Caching"

I was playing with memcached a few days ago. Never used it before, so spent some time to read documentation and FAQ.

While reading FAQ, I found the link to the page with A Story of Caching, and decided to share link to this story here.

IMHO, this is the best non-technical technical documentation, that I've read recently. In this story, they use simple but widespread use cases of developing web application and using cache in this web application.

If you don't know what's for memcached, than read this story.
If you don't know what's the primary features of memcached, than read this story.
If you don't know if you need memcached, than read this story.

Assign <#nested> to variable in FreeMarker

FreeMarker is awesome template engine. I'm still wonder how much powerful it is and how flexible it can be when you need. Last week, when I was writing a simple macro to render tweets using user defined template, my though was that it may be good to define this template as macro nested content. For instance:

<@tweets 'rkhmelyuk', 10>
[picture] [username]: <br/>
[message]<br/>
<a href="[url]">[date]</a>
</@tweets>

Pretty simple, but the only way for me was to load tweets using Twitter API through JSONP protocol. So my macro was need to generate a javascript code, that should correctly define callback and fetch tweets. Callback function simply applies received JSON data to the user defined template.

And here is the problem. How to get macro's nested content into javascript variable? Well, who ever used FreeMarker knows that nested content can be rendered with command <#nested/>. This could be a solution, but isn't. Why? Because I need to do extra preparing of template to put it as a javascript string. I need to remove newlines and replace quotes. So the best decision is to put nested content as Freemarker variable and do preparation over it first and than insert into javascript. And here is the power of Freemarker. This was possible and easy to do with <#assign/> or <#local/> command:

<#local layout><#nested/></#local>
<#-- Now layout variable contains a template defined as nested -->
<script type="text/javascript">
function callback_${id}(data) {
var layout = "${layout?SomePreparationsGoHere}";
renderTweets(layout, data);
}
</script>

In this code, the nested content is assigned to variable layout. Then we generates a javascript code and insert layout variable as string, but first do some preparations.

I also like flexibility of Freemarker. I hope that could find a time and describe some other tips about it soon.

Facebook was failing to parse OG meta tags

Last week was having awful time fighting with strange work of Facebook. We had a page with OG meta tags and a Facebook Like button on the page. And Facebook was discarding to parse the page and extract data from OG meta tags. Using URL linter didn't help to find the problem but show the same information - page isn't parsed, OG meta tags are not used.

So I tried to check next things to find the issue:

  • whether property attribute in used <meta/> tags

  • whether correct fb:app_id meta tag value is set

  • whether nginx returns correct page when facebook requests the page

  • whether correct namespaces added to the <html/> tag

  • whether we are using UTF-8 encoding

  • and something other, don't remember all my tests

And was checking web page through URL Linter each time. With no success. Facebook discarded parsing the page and fetching page title with other data from meta tags.

So I reviewed response/request data through firebug to find suspicious things. I found one scanty detail: Content-Type header had value "text/html;charset=utf-8;charset=UTF-8".
I must say, this one web page is generated by application using user-defined template. Grails controller is responsible for handling request and renders output like:

render text: context.out, contentType: "text/html;charset=$AppConstant.RENDER_CHARSET"

where AppConstant.RENDER_CHARSET equals to "UTF-8". But the rendered page had Content-Type equal to "text/html;charset=utf-8;charset=UTF-8". And my thought was to fix header value to have something like "text/thml;charset=UTF-8" and hoped this may help. So I changed code to be like:

render text: context.out, contentType: "text/html"

Content-Type header value now is "text/html;charset=utf-8". So I tried URL Lint again. I wasn't expecting much from Facebook this time either, but miracle happened! Everything works fine now. I know, that is my fault, but, nevertheless, thank you Facebook for "nice" time spending :).

Linux tools to generate password

Few days ago was looking for simple tool to generate passwords in linux console.
As result of my searching are 3 useful tools. They are:

  • apg

  • makepasswd

  • pwgen


apg


Install: sudo apt-get install apg

Asks to enter random data, that should be used to generate new password.
Example:

Please enter some random data (only first 16 are significant)
(eg. your old password):>
Opt8Ovuf (Opt-EIGHT-Ov-uf)
Uc1Gryec (Uc-ONE-Gryec)
jadJoav5 (jad-Joav-FIVE)
IshtIvawam3 (Isht-Iv-aw-am-THREE)
lakVosAfUrg7 (lak-Vos-Af-Urg-SEVEN)
dyijDus8 (dyij-Dus-EIGHT)

You can pass random string as parameter and use many different options:

$ apg -n 10 -m 8 -x 12 somerandomdata

Will generate 10 passwords with min length 8 and max length 12.

makepasswd


Install: sudo apt-get install makepasswd

Generates password, by default one. User may need to use options to set length of password and count of passwords. For example:

$ makepasswd --count=10 --minchars=5 --maxchars=10

Results:

mzd4f9q
gUWamL
NYiUXrYvq3
6hWDXKA
gQpu20IJGD
BSAT5ASFX
37FcKyLPb
ma7pC66A
cFpWPBy
0oTNhT7


pwgen


Install: sudo apt-get install pwgen

My favorite tool to generate password. By default generates 160 different passwords each with length 8 symbol. Programs takes 2 parameters:
- 1st is the length of the password and
- 2nd is the count of passwords to generates. For example:

$ pwgen 12 24

Results:

oHo2iethieze cheiS6ohPeed Oozufiorohv9 eic3aethei4L hiRohYie6Ue4 aephoiDieb0y
hieTh8eizaeK aid1EeNgaiSe yoh6chohX9ha aiPhae7dieMe wedooD8nai7y aic7deeB9eS8
ohFor3Achied Thaequu4aiph zaeghiem7keT Shee5ooxaex0 wePh2eiNgien aicohroo2Go3
Aagh0gahcah0 Zie8eazaitah aoha9AeXi7Bo Oojoob0oosh6 Olahgh4aeji6 oobae9UZ2phi