Catching up!

A lot has changed since the last blog post (more than three years). I was happily running a successful business around Videocache till Google decided to push HTTPS really hard and enforced SSL even for video content. That rendered Videocache completely useless as YouTube video caching was the unique selling point. Though people are still using it for other websites (whatever supported and not HTTPS yet), I personally didn’t find it good enough for selling. To add to the trouble, Mozilla and friends announced that there will be free certs for everyone. Now, that took away whatever motivation was left to keep working on Videocache. I decided to open source Videocache and the source is now available on GitHub. If you have better ideas or you are looking forward to make things work by forging certs etc, fork it and give it a shot.

After all that mess with Videocache, I am left with a contract job which is not working out well. So, the learning hat is back and I am trying to catch up with the tech world. I didn’t really want to get into the whole JavaScript framework mess because there is no clear winner and there are too many of them but it looks rather unavoidable or Unflushable (if you remember Jeff from Coupling). I have been trying to make simple apps with Angularjs. There is little documentation and you have to dig really deep at times but still you can make things work if you persist. Once you spend some quality time with it, you may actually start liking Angularjs. So, I did give it some time and implemented angular fronted for Pixomatix-Api (another learning project I am working on) and I think I sort of like it now.

In 2015, if you are web developer, you must know how APIs work and you should be able to consume them. So, to learn to expose APIs and version them properly, I fired a small project Pixomatix. Being a Rails developer, you really get obsessed with it and try to implement everything using Rails. Even when you want an API with 2-3 endpoints, you tend to make the horrible mistake of doing it in Rails. This kept bugging me and a few weeks later, I decided to freshen up my Sinatra memories. But working with Sinatra is not all that easy especially if you are used to all the niceties of Rails. Dug up my attempt of implementing Videocache in Ruby, and extracted few tasks and configurations I had automated long time ago. Ended up working a lot more on it and packages into a template app with almost all the essential stuff. Though I need to document it a little more, the app has got everything needed to expose a versioned API via Sinatra.

On the other hand, I tried to use devise gem to authentication for Pixomatix. It was all good for integration with standard web apps and APIs but it sort of failed me when I tried to make the API versioned. Devise turned out to be black-hole when I tried to dig deeper to make things work. I tried a few other gems which supported token authentication but they were also no good for versioning. Generally, you may not need to version the authentication part of your API, but what if you do! Since, this was just a learning exercise, I was hell bent on implementing this. So, I just reinvented the wheel and coded basic authentication (including token authentication) for the API.

That’s it for this post. I am looking forward to post regularly on the new stuff I learn.


Getting Started With Git

I have been using Git for years now and I am still a novice user. Actually, if you are an individual developer or work in a really small team, you don’t really need to go beyond using the basic stuff like cloning a repo, committing code, branching and pushing back to the origin repo. And that’s exactly what happened with me. Working alone most of the times and working in my own repos. I never felt the need to explore advanced usage.

Last week, we didn’t have internet at home for like four days. So, I thought I’ll read a book. I downloaded ProGit : The Free Git Book and started reading. It’s an awesomely written book. I learned a lot of new things about Git. A few more concept about using branches efficiently. If you are a Git beginner or looking forward to getting started with Git and you don’t like reading too much, I’ll suggest that you read at least first three chapters thoroughly. That along with a free GitHub account should be enough to get you going if you are not going to be a heavy Git user soon.


News : Memepress (Yahoo! Meme) WordPress Plugin Version 0.3 is available

Memepress is a wordpress plugin which provides wordpress widgets for displaying public posts from Yahoo! Meme. Memepress is SEO ready and provides options to noindex/nofollow your Meme posts. New version fixes a few bugs and provides enhanced style control for the widget.

You can download memepress version 0.3 from wordpress plugin page.

For details on usage/installation instructions, please check plugin homepage.


Memepress Ready to be Translated (Supports Turkish now)

Thanks for an awesome response to the Memepress (Yahoo! Meme) WordPress plugin. A lot of people have blogged about it and contacted me via email for enhancements. New version also fixes the bug regarding image size in posts. I have also updated the plugin to support internationalization(i18n). Now you can translate about 20 strings in your local language and you’ll have Memepress in your own language.

Turkish Translation

Huge thanks to Yalçın Erdemir for translating the plugin to Turkish :)

How To Translate

If you are interested in translating Memepress plugin in your language, then download the memepress.pot (POT file) and open it in your favourite text editor. Now for every string against msgid should be translated to your language and populated in quotes against msgstr. See the example below.

Before translation

#: memepress.php:37
msgid "Title"
msgstr ""

After translation

#: memepress.php:37
msgid "Title"

Once you are done with translating all the strings, save the file and mail me at kulbirsaini25 [AT] .


How To: Install and Use Twython (Python Wrapper for Twitter API)

As promised in my previous post, here is a brief howto on getting started with twython. The main advantage of Twython over several other python (or any other language) wrappers for Twitter API is that it works even when you are behind your organizations proxy.

Download Twython

You can download latest version of twython from twython page on github. You can either clone using git (if you have git installed) or can click the download button.

Install Twython

Once you are done with extracting the downloaded tar file. Change directory to twython and run these command as root.

[root@fedora ~]$ git clone git://
[root@fedora ~]$ cd twython/dist
[root@fedora dist]$ tar -xvzf twython-0.8.tar.gz
[root@fedora dist]$ cd twython-0.8/
[root@fedora dist]$ python build
[root@fedora dist]$ python install

Use Twython from Python Interpreter

Below is a direct copy paste lines from my interpreter. See how things are working (learning by doing).

[saini@bordeaux ~]$ python
Python 2.6 (r26:66714, Mar 17 2009, 11:44:21) 
[GCC 4.4.0 20090313 (Red Hat 4.4.0-0.26)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> # first of all, import twython module
>>> import twython
>>> # Authenticate your twitter account with your twitter username
... # and password with twitter.setup method.
>>> client = twython.setup('Basic','myusername','mypassword')
>>> client.authenticated
>>> #Lets update our current status on twitter with some cool message.
>>> client.updateStatus('Testing #twython. The coolest #TwitterAPI :)')
>>> # Now go and check your current status on twitter. Surprised!!!
>>> # Get your or anyone's followers
>>> client.getFollowersIDs(screen_name='gofedora')
>>> # Output truncated.
>>> # Get help for any function.
>>> print client.createFriendship.__doc__
createFriendship(id = None, user_id = None, screen_name = None, follow = "false")
	Allows the authenticating users to follow the user specified in the ID parameter.
	Returns the befriended user in the requested format when successful. Returns a
	string describing the failure condition when unsuccessful. If you are already
	friends with the user an HTTP 403 will be returned.
		** Note: One of the following is required. (id, user_id, screen_name)
		id - Required. The ID or screen name of the user to befriend.
		user_id - Required. Specfies the ID of the user to befriend. Helpful for disambiguating when a valid user ID is also a valid screen name. 
		screen_name - Required. Specfies the screen name of the user to befriend. Helpful for disambiguating when a valid screen name is also a user ID. 
		follow - Optional. Enable notifications for the target user in addition to becoming friends. 

So now you are ready to do wonders with twython. Write your own code and blog/brag about it :)


I Love Twitter, I Love Python = I Love Twython

I was really late to join Twitter. As soon as I joined Twitter, I started exploring various aspects of Twitter and hence the Twitter API. While searching for a wrapper for Twitter API in python, I came across python-twitter which was good and tweepy which was ok and I finally found twython. Twython is awesome and more importantly upto date with Twitter API.

While I was using twython, I found one fact very irritating. It didn’t have docstrings. And I had to go to Twitter API Wiki for description of every single function :( I talked to Ryan (Twython developer, who works all day long at his day job with and develops twython when everybody sleeps 😛 ) about the same and offered to write docstrings for twython.

I spent a few hours and sent a patch with docstrings for all the functions. Now twython has complete documentation. Also Ryan is trying his best to implement wrappers for the newly introduced functions in Twitter API. So, currently Twython is THE best wrapper available in python for Twitter API.

PS : Next post will be “How to use twython”.


TIP: Simple Regular Expression for Email Validation

I was playing around with Ruby on Rails and needed an email field for some application. I thought of validating the email address before actually saving it. Googled for same and got tons of results, but none of them was perfect. Few were somewhat exhaustive but horribly difficult to understand and those which were simple were  not exhaustive enough for practical use. I thought I’ll just write the regex myself and came up with the following. I have tested it for few weird cases like an email hosted on (a subdomain deep down the hierarchy) etc.


I hope it’ll be of help. And don’t forget to pen down your suggestions via comments.


How To: Boot Fedora Faster

Note: These tricks apply to any Linux based OS. But I have tested them only on Fedora, so can’t say whether they’ll work on other Linux(s).

My current Fedora installation is now almost one and a half years old. Yes. I am still using Fedora 7 😀 I have Fedora 10 on my other machine. Coming to the agenda, my Fedora installation has grown beyond control and I have services from named, squid, drbl, privoxy, vsftpd, vbox*, smb and what not on a personal desktop. These services really force my system startup to slow down to more than two minutes. While shutting down, its very easy to just cut the power supply but while booting up I can’t help and it frustrates me. And what frustrates me further that I have 4GB DDR2 RAM and AMD64 X2 5600+ (2.8GHz x 2) and booting time is still more than two minutes.


  • Boot Fedora faster using whatever techniques possible.

Remove the services from normal order and delay their execution to a later stage. So, services like network, squid, privoxy, named, vsftpd, smb etc. doesn’t make sense unless I am not logged in and using them. Let us start them after we have login screen.

Turn off all the services by using the command

[root@bordeaux ~]# chkconfig service_name off

where service_name is the service you want to turn off.

Now create a file /etc/ Enter a line like this

[root@bordeaux ~]# service service_name start

for every service that you have turned off in the Step 1.1 and you want it to be running after your machine starts up. Now, your file should look like this

service network start &
service sshd start &
modprobe it87 &
modprobe k8temp &
/usr/bin/iptraf -s eth0 -B &
/usr/bin/iptraf -s lo -B &
service squid start &
service privoxy start &
service httpd start &
service mysqld start &
service named start &
service smb start &
service vboxdrv start &
service vboxnet start &
service vsftpd start &

Add the following line to /etc/rc.local file

/bin/bash /etc/ &

Done!!! Notice the &s in both files. They are for execution in background so that a process can block boot process. You’ll observe a drop of 10-20 seconds in system startup time.

Problem with Hack #1 : The execution is not really parallel. It executes like a process in the background. So we can’t get the real advantage of parallel execution.

Hack #2 solves this problem. Now we don’t put processes in background. We use daemon forking to fork a separate daemon process which will start all the services for us in parallel. Here we’ll get the real advantage and startup time will decrease further.

This step is totally similar to Step 1.1. So skipping it.

This step is also similar to Step 1.2. The /etc/ file should look like this.

service network start
service xinetd start
service crond start
service anacron start
service atd start
service sshd start
service rpcbind start
service rpcgssd start
service rpcimapd start
modprobe it87
modprobe k8temp
/usr/bin/iptraf -s eth0 -B
/usr/bin/iptraf -s lo -B
service nasd start
service squid start
service privoxy start
service httpd start
service iptables start
service lm_sensors start
service mysqld start
service named start
service nfs start
service nfslock start
service smb start
service vboxdrv start
service vboxnet start
service vsftpd start
service autofs start
service smartd start

Notice the absence of &s in the file.

Download the attached file attached at the end of this post or copy paste the following code to /etc/ file.

#!/usr/bin/env python
# (C) Copyright 2008 Kulbir Saini
# License : GPL
import os
import sys
def fork_daemon(f):
    """This function forks a daemon."""
    # Perform double fork
    r = ''
    if os.fork(): # Parent
        # Wait for the child so that it doesn't defunct
        # Return a function
        return  lambda *x, **kw: r
    # Otherwise, we are the child
    # Perform second fork
    if os.fork():
    def wrapper(*args, **kwargs):
        """Wrapper function to be returned from generator.
        Executes the function bound to the generator and then
        exits the process"""
        f(*args, **kwargs)
    return wrapper
def start_services(startup_file):
    command = '/bin/bash ' + startup_file + ' > /dev/null 2> /dev/null '
if __name__ == '__main__':
    forkd = fork_daemon(start_services)
    print 'Executing ', sys.argv[1], '[  OK  ]'

Add the following line to your /etc/rc.local file.

/usr/bin/python /etc/ /etc/

Thats it. Done!!! Now you’ll experience a boost of about 25-30 seconds of decrease in boot time.

Stats of my machine

With all services started in normal order : 2minutes.
With Hack #1 : 1minute 42 seconds.
With Hack #2 : 1minute.

Warning : These hacks may break your system and can make it unusable. Use at your own risk.


Hack: A Fast Network Scanning Program

I was searching for a simple tool which can do a port scanning in a huge network quickly without making me wait for ages. I first thought of using nmap, but it was a bit too complex and it takes a lot of time to discover the machines even after optimizing the parameters. After searching a lot, I wrote to one of my seniors, Sandeep Kumar, asking the details of his program which maintains a list of active FTP servers in the network. He replied with a reference to his own findings about the network scanning tools. He is using an enhanced version of a program originally written by Troy Robinson. I tried the program out of curiosity and found out that its damn fast as compared to nmap (no literal comparison) :) The program can be downloaded from here.

How to use

Compile the program using gcc as

[root@localhost ~]# gcc NetworkScanner.c [ENTER]

Now create a file IPRange.txt containing the IP address ranges for your network. The contents of the file may be

172.16.*.* Meaning all the IP address with first two parts as 172.16 and rest of the address will be generated by permutations.

172.16.1-16.* Meaning the first two parts are fixed. Third part will vary from 1 to 16. And the fourth part will be permuted from 0 to 255.

So an IPRange.txt may look like


Now run the program as

[root@localhost ~]# ./a.out port_to_be_scanned Parallel_attempts IP_list_file output.txt [ENTER]

Parallel_attempts is the number of processes that’ll be forked for scanning the network port. It is safe to have its value as 255. A very high value may hog the network or may even slow down your machine. So an example run would be

[root@localhost ~]# ./a.out 21 255 IPRange.txt Output.txt [ENTER]


I carried out a lot of test on my network using the following setup and parameters

Machine : AMD X2 5600+ (2.6GHz Dual Core), 4GB 800MHz DDR2 RAM, Gigabit Ethernet Card (on 100mbps network).

Port : 21 (FTP)

IPRange.txt : Total 16896 IP Addresses

Machines on wired (100mbps) network
Machines on wireless (54mbps) network

Network Scanner Benchmarks

Parallel Attempts

Scanning Time (seconds)

Upload Bandwidth (kbps)

255 180 13
512 90 25
1024 47 55
2048 25 100
4096 14 205
6144 11 307
8192 9 374

The interval between two scans was almost 30-40 seconds. I think parallelism beyond 8192 will crash my machine, so I didn’t try. You can try it at your own risk :) I hope this program help you scan your network.


How To: Write Custom Redirector or Rewritor Plugin For Squid in Python


To write a custom Python program which can act as a plugin for Squid to redirect a given URL to another URL. This is useful when already existing redirector plugins for Squid doesn’t suit your needs or you want everything of your own.

Use Cases

  1. When you want to redirect URLs using a database like mysql or postgresql.
  2. When you want to redirect based on mappings stored in simple text files.
  3. When you want to build a redirector which can learn by itself using AI techniques 😛

How to proceed

From Squid FAQ,

The redirector program must read URLs (one per line) on standard input, and write rewritten URLs or blank lines on standard output. Note that the redirector program can not use buffered I/O. Squid writes additional information after the URL which a redirector can use to make a decision.

The format of the line read from the standard input by the program is as follows.

URL ip-address/fqdn ident method
# for example - GET -

The implementation sounds very simple and it is indeed very simple to implement. The only thing that should be taken care of is the unbuffered I/O. You should immediately flush the output to standard output once decision is taken.

For this howto, we assume we have a method called ‘modify_url()‘ which returns either a blank line or a modified URL to which the client should be redirected.

#!/usr/bin/env python
import sys
def modify_url(line):
    list = line.split(' ')
    # first element of the list is the URL
    old_url = list[0]
    new_url = '\n'
    # take the decision and modify the url if needed
    # do remember that the new_url should contain a '\n' at the end.
    if old_url.endswith('.avi'):
        new_url = '' + new_url
    return new_url
while True:
    # the format of the line read from stdin is
    # URL ip-address/fqdn ident method
    # for example
    # - GET -
    line = sys.stdin.readline().strip()
    # new_url is a simple URL only
    # for example
    new_url = modify_url(line)

Save the above file somewhere. We save this example file in /etc/squid/ Now, we have the function for redirecting clients. We need to configure squid to use . Below is the squid configuration for telling squid to use the above program as redirector.

# Add these lines to /etc/squid/squid.conf file.
# /usr/bin/python should be replaced by the path to python executable if you installed it somewhere else.
redirect_program /usr/bin/python /etc/squid/
# Number of instances of the above program that should run concurrently.
# 5 is good enough but you should go for 10 at least. Anything below 5 would not work properly.
redirect_children 5

Now, start/reload/restart squid. That’s all we need to write and use a custom redirector plugin for squid.