memory: metrics graphs with gnuplot

So, at my spare time, I work a bit on my pet projects called memory. This is a simple virtual memory written on Java. User can allocate a large space of memory and then use it to allocate and free some blocks of it for personal use.

My main interest was to implement some memory allocation algorithms that could be working in concurrent environment with minimal of locks. I'm still on my way with it however, have some progress.

One of the problems with such projects is the way you can measure performance and quality. And what is more important, how you can assess you changes in these fields. Without a good set of performance tests, you can't know if your changes are for good. But having tests is not enough. You need to have metrics and gather them, and represent them in some charts for better visibility. Visualization matters. And then, when you have a way to look at those charts you can examine your's changes much much easier and better.

So that's what I did. First, wrote a simple metric gathering framework for internal use. Then added
metrics and used CodaHale Metrics to gather metrics data and save into CSV file.

 Simple Ruby script makes needed conversion, merging and somewhere aggregation to prepare data files that could be used for charts.

And Gnuplot is used to build charts for those data files. Although that was possible to build chart based on multiple data files, I decided to add extra logic to scripts to merge and aggregate some data. It's so convenient when you can view the raw data that was used to build a chart.

I've spent some time investigating how to use gnuplot to build somehow beautiful charts. After spending a few hours and reading a lot of resources over the internet, I've stopped on the next script:


#!/bin/sh

title=$1 # graph title
xlabel=$2
ylabel=$3
output=$4 # output image name
data=$5 # data file
cols=$6 # which columns to show in the graph

lines='' # the plot lines
index=0 # loop index
ls=1 # line style

# --- build the configuration of lines for plots
for col in $cols
do
if [ $index -ne 0 ]; then
lines=$lines", "
fi

lines=$lines"'$data' u 1:$col t '' w l ls 42 smooth bezier"
lines=$lines", '$data' u 1:$col w l ls $ls"
index=`expr $index + 1`
ls=`expr $ls + 1`
done
echo $lines

gnuplot << EOF output = "$output" xlabel = "$xlabel" ylabel = "$ylabel" title = "$title" # use TTF fonts on Mac OS X set fontpath "/Library/Fonts" # use pngcairo to get this beautiful antialias font and smooth chart set term pngcairo font "Courier,9" set term pngcairo size 500, 200 # image background is lightgray set term pngcairo background "#e0e0e0" set output output # chart background is white set object 1 rectangle from graph 0, graph 0 to graph 1, graph 1 behind fc rgbcolor '#ffffff' fs noborder set xlabel xlabel font "Arial Bold,11" textcolor rgb "#222222" norotate set ylabel ylabel font "Arial Bold,11" textcolor rgb "#222222" rotate parallel set autoscale xfix set title title font "Arial Bold,12" textcolor rgb "black" set key outside bottom horizontal left reverse set key font "Courier,10" textcolor rgb "#444444" set key autotitle columnheader set style line 81 lc rgb '#555555' lt 1 set border 3 back ls 81 set tics nomirror set style line 82 lc rgb '#aaaaaa' lt 0 lw 1 set grid back ls 82 set style line 1 lt rgb "#8b1a0e" lw 1.5 pt 1 ps 0.3 set style line 2 lt rgb "#5e9c36" lw 1.5 pt 2 ps 0.5 set style line 3 lt rgb "#0060ad" lw 1.5 pt 3 ps 0.6 set style line 4 lt rgb "#ff949E" lw 1.5 pt 4 ps 0.8 set style line 5 lt rgb "#674C87" lw 1.5 pt 5 ps 0.5 set style line 6 lt rgb "#777777" lw 1.5 pt 6 ps 0.8 set style line 7 lt rgb "#222222" lw 1.5 pt 7 ps 0.5 set style line 8 lt rgb "#FBDF61" lw 1.5 pt 8 ps 0.5 set style line 9 lt rgb "#F25900" lw 1.5 pt 12 ps 0.6 set style line 10 lt rgb "#014F4B" lw 1.5 pt 10 ps 0.5 # this is used for smooth bezier line set style line 42 lt rgb "#dddddd" lw 1.1 pt 10 ps 0.5 set xtics autofreq set xtics add 1 set datafile separator "," plot $lines EOF
So there are some examples of result charts:

Two different values are shown on this graph.

Barely seen lightgray line is very helpful in some moments, especially if graph is spiky.

I'm almost ready to push all related changes to the memory repository, but have some moments I want to polish, and some functionality to add.

Those graphs helped me to find some issues in the allocation algorithm and fix them. Actually, I use a single page with a set of different charts:

Clickable.

Not awesome. But pretty nice and, what is more important, it helps.