Parsing Archives - Versatile Programmer's Diary

To find the bandwidth usage of your Apache server, you can use many existing tools. Like vnstat, awstat. The most common thing about these tools is they need the utility installed. What if you dont have this installed and you want to calculate your bandwidth? This is can be easily done by parsing Apache access logs. This technique will only work if you are a web master and you have no other bandwidth eating service other than apache. Most web developers will fall in this category. So here is the technique to find apache bandwidth usage.

Note: You need ssh access to perform these actions. Also I assume you have not deleted your log files.

Determine the date range for which you want to find the bandwidth usage for. For example I want to deter mine bandwidth usage from Oct 1st 2012 to Oct 30 2012. Note you must have access log files for that range.

Now the big command. Assuming your apache log directory is /var/log/apache2

find /var/log/apache2 -type f \
    -name '*.gz' \
    -newermt "2012-10-01 00:00:00" \
    -not \
    -newermt "2012-10-30 23:59:59" \ 
    -exec zcat '{}' \; | 
egrep '"[^"]+" +200 [0-9]+' -o | 
awk '{sum+=$(NF)} END {print sum/1024/1024/1024 " GB"}'

This will print something like “34.345 GB“

Here is a snipped I wrote today to parse Query String by pure JavaScript. Some people uses different JavaScript libraries. But from my older days when I used to parse GET parameter by perl I had a habit of writing this algorithm. I just translated it to JavaScript.

Here goes the code.

[code language=”javascript”]
function getUrlParts(url){
// url contains your data.
var qs = url.indexOf("?");
if(qs==-1) return [];
var fr = url.indexOf("#");
var q="";
q = (fr==-1)? url.substr(qs+1) : url.substr(qs+1, fr-qs-1);
var parts=q.split("&");
var vars={};
for(var i=0;i<parts.length; i++){
var p = parts[i].split("=");
if(p[1]){
vars[decodeURIComponent(p[0])] = decodeURIComponent(p[1]);
}else{
vars[decodeURIComponent(p[0])] = "";
}
}
// vars contain all the variables in an array.
return vars;
}
[/code]

Tag: Parsing

Determine outgoing apache bandwidth usage with built in commands

Parse Query String by pure JavaScrirpt