IntelligentMirror Gets Even More Intelligent (1.0.1)

Warning : This version of IntelligentMirror is compatible with only squid-2.7 as of now. It is NOT compatible even with squid-3.0.

IntelligentMirror Version 1.0.1

I have been following squid development regularly (at least the part in which I am interested) and they have introduced a new directive in squid-2.7 known as StoreUrlRewrite (storeurl_rewrite_program). Using this directive you can instruct squid to cache url A (http://abc.com/foo/bar/version/crap.rpm) as url B (http://proxy.fedora.co.in/intelligentmirror/crap.rpm). In simple words you can direct squid to cache any url as any other url without any extra efforts.

So keeping the above directive in mind, I have worked out a different version of intelligentmirror especially for squid-2.7.

IntelligentMirror : Old method of operation

  1. IntelligentMirror gets a client request for a URL.
  2. Check: if URL is not in (RPM, metadata file)
    • Then its none of our business.
    • Let proxy handle it the normal way.
    • Done and exit.
  3. Check: if RPM/metadata is available in cache
    • Stream the RPM/metadata from cache.
    • Done and exit.
  4. Check: if RPM/metadata is not available in cache
    • Download in parallel for caching in some dir and stream.
    • Done and exit.

IntelligentMirror : New method of operation

  1. IntelligentMirror gets a client request for a URL.
  2. Check: if request for rpm
    1. Direct squid to cache the request as http://<same_host_all_the_time>/intelligentmirror/<rpmname>.rpm
  3. Check: if request for deb
    1. Direct squid to cache the request as http://<same_host_all_the_time>/intelligentmirror/<debname>.deb
  4. Done and exit.

So your squid will see every request for an rpm package as a request http://<same_host_all_the_time>/intelligentmirror/<rpmname>.rpm. So, if you happen to request the same rpm from a different mirror, it’ll still be served from cache :)

Improvements

  1. No need to check if the url supplied by squid is for rpm or not because storeurl_rewrite_program has an acl controller attached which will invoke intelligentmirror for urls ending in .rpm .
  2. No need to check if the url is already cached or not. No need to worry about the directory where you are going to store the packages. No human intervention is needed in maintaining the cache. Almighty squid is doing everything for us.
  3. No need to worry if the target package has changed because of the resigning or whatever because squid will do that for you.
  4. No need to actually download the package in parallel for caching because squid is already doing that.
  5. No need to worry about the hashing algorithms and storage optimizations for the cached content.

Availability

  1. RPM for Fedora/Red Hat
  2. Source RPM for Fedora/Red Hat
  3. Source Tarball

Install and Configure

The install and configure files should be enough to guide you through the installation if you choose the tar ball way. Otherwise you can always install from rpm from the above link.

Note1: You have to configure your squid to use intelligentmirror as a plugin even if you install via rpm. Check the configure file at the above link.

Note2: StoreUrlRewrite will probably be available in squid-3.1.

 

IntelligentMirror: RPM and DEB Caching Improved (0.4)

IntelligentMirror version 0.4 is available now. There have been significant improvements in intelligent mirror since last release.

Improvements

  1. Fixed defunct process problem. You will not see defunct python processes hanging around anymore. Previously every forked daemon used to got defucnt because parent never waited for the forked child to finish.
  2. IntelligentMirror now supports caching of Debian packages just like rpms. So now IntelligentMirror is best suited shared environments where people have different tastes.
  3. Intelligent Mirror now uses url_rewrite_program instead of redirect_program. This boosts the efficiency of IntelligentMirror by a significant factor as url_rewrite_program has an acl controller url_rewrite_access. And using url_rewrite_access only requests for rpm/deb packages will be passed to Intelligent Mirror. So, IM now need not process each and every incoming request. Also, it has redirector_bypass directive which will bypass IM in case all the instances of IM are busy serving requests. So, squid will not die with a fatal error in case of huge requests.
  4. Options to enable/disable caching for rpm and Debian packages have been added.
  5. Options to control the total size of caching directories and the size of individual package to be cached have also been introduced.
  6. Proxy authentication is also supported now just the way it is supported in yum.
  7. Packages are not checked for last-modified time anymore. Because in principle two rpms A and B can only have same name iff they have the same contents. So, the delay in response time in case of hits has reduced.

Availability

  1. RPMs for Fedora/Red Hat
  2. Source RPMs for Fedora/Red Hat
  3. Source Tar balls

Installation and configuration is easy and the INSTALL and README files should serve the purpose.

In case you have any suggestions or problems, leave a comment here or file a ticket on project page.

 

IntelligentMirror: Available for Testing

Note : A newer version of intelligentmirror is available now. Please check this.

Intelligent Mirror is basically a tool or squid plugin (redirector) to cache rpm packages so that the subsequent requests for the same package can be served from the local cache which will eventually save a lot of bandwidth and downloading time.

Who needs Intelligent Mirror?

  1. If you are on a shared network where a lot of people use linux distros with RPM as their package manager, then you need this. Universities should come under this category.
  2. If you have a set of systems having red hat derivatives and almost identical OS versions, you need this. LAN setups at home should come under this category.
  3. If you can’t afford to or don’t want to mirror entire fedora repo for local access due to bandwidth limitations, you need this.

What it does?

As described above, Intelligent Mirror, just caches rpms which are requested by the clients in a shared network. And subsequent requests for those rpms are served from the cache. For a detailed description, check the project page.

Why not use Squid in caching mode?

Squid caching is based on url hashing. Let me explain with an example how Intelligent Mirror is actually intelligent as compared to squid while caching rpms.

Let us say there is an rpm yum-3.2.0-1.fc7.i386.rpm . You executed “yum update yum“. And let us say the newer version of yum is yum-3.2.18-1.fc9.i386.rpm which was fetched from one of the fedora mirrors http://abc.com/ (say). Now someone on the same network launched “yum update yum” and he got the same rpm yum-3.2.18-1.fc9.i386.rpm. But this time rpm was fetched from another mirror http://xyz.com/ (say).

Case I : Squid caching

Squid will cache http://abc.com/linux/fc9/updates/i386/yum-3.2.18-1.fc9.i386.rpm . And when http://xyz.com/linux/fc9/updates/i386/yum-3.2.18-1.fc9.i386.rpm will be requested, it’ll result in a cache miss and squid will again download the same package and will cache this one as well. Now there are two problems

  1. Squid is not able to serve from the cache, though the package was the same.
  2. Additional storage space is being wasted in caching the same package. And this can really harm if unluckily a different mirror is picked in all the subsequent queries.

Case II : IntelligentMirror caching

Intelligent Mirror will cache the package yum-3.2.18-1.fc9.i386.rpm without bothering about its origin. And even if yum picks up a different mirror for the subsequent request, the package will be served from the cache and will not be fetched from upstream. So, the obvious advantage of saving the bandwidth and downloading time.

Download

Intelligent Mirror source tarball, rpm, source rpm are available for download from here.

Installing and Configuring Intelligent Mirror

Install Guide

Configuration Guide

Issues and Suggestions

If you see any issue or you have any suggestions for improving the functionality, either mail me at kulbirsaini25 AT GMAIL DoT COM or file a ticket on the project page.

 

How To: Configure Squid Proxy Server

Mission

To configure squid for simple proxying without caching anything.

Use Cases

  1. When you want to have control on what people browse on your lan.
  2. When number of machine is more than the number of IP addresses you can afford to buy.
  3. When you want to help this holy world in saving some IPV4 addresses 😛

Assumptions

  1. You have a machine connected directly to internet that you are going to use as a proxy server for other machines on your network.
  2. The machines on your network are using 192.168.0.0/16 as private address space. You can use anyone/multiple address spaces of the available but for this howto we assume 192.168.0.0/16 as the local network.
  3. The local IP address of the machine which will run squid proxy server is 192.168.36.204. You can have any IP, but for this howto we assume this.

How to proceed

First of all ensure that you have squid installed. After installing squid, you need to set access control in squid configuration file which resides in /etc/squid by default. Open /etc/squid/squid.conf and add/edit following lines according to your preferences. Few lines already exist in the configuration file, you can add the rest.

# The port on which squid will listen for requests
http_port 8080
# If 'cgi-bin' or '?' is in query, squid should not check with neighbours'/parents' cache
# and should go to target web-server.
hierarchy_stoplist cgi-bin ?
# If url contains 'cgi-bin' or '?', then it must not be cached
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
# Absolute path to squid access log.
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320
# Access control list to control every IP address
acl all src 0.0.0.0/0.0.0.0
# Access control list for source machine in LAN
acl lan_src src 192.168.0.0/16
# Access control list for destination machine in LAN
acl lan_dst dst 192.168.0.0/16
# Access control list to manage squid cache
acl manager proto cache_object
# Access control list to define IP address allowed for source localhost
acl localhost src 127.0.0.1/255.255.255.255
# Access control list to define IP addresses allowed for localhost as destination
acl to_localhost dst 127.0.0.0/8
# Access control list to define Safe ports that should be allowed by default
acl SSL_ports port 443 563 1863 5190 5222 5050 6667
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
# Allow cache management only from localhost
http_access allow manager localhost
# Deny cache management from remote hosts
http_access deny manager
# Deny http access via all the ports which are not listed as safe
http_access deny !Safe_ports
# Deny all connections via all ports which are not listed as safe
http_access deny CONNECT !SSL_ports
# Allow http access from localhost
http_access allow localhost
# Allow http access from machines on LAN
http_access allow lan_src
http_access deny all
http_reply_access allow all
icp_access allow all
# Deny caching for everyone so that there is not caching at all
cache deny all
coredump_dir /var/spool/squid
# Never allow direct connection to machines on the internet
prefer_direct off
never_direct allow all
# Allow direct connetion if the destination machine is on LAN
always_direct allow lan_dst
# Delete this line if you don't have /etc/hosts file
hosts_file /etc/hosts
# Allow AIM connections
# Delete the following 9 lines if you don't want people to connect to AIM
acl AIM_ports port 5190 9898 6667
acl AIM_domains dstdomain .oscar.aol.com .blue.aol.com .freenode.net
acl AIM_domains dstdomain .messaging.aol.com .aim.com
acl AIM_hosts dstdomain login.oscar.aol.com login.glogin.messaging.aol.com toc.oscar.aol.com irc.freenode.net
acl AIM_nets dst 64.12.0.0/255.255.0.0
acl AIM_methods method CONNECT
http_access allow AIM_methods AIM_ports AIM_nets
http_access allow AIM_methods AIM_ports AIM_hosts
http_access allow AIM_methods AIM_ports AIM_domains
# Allow connections to Yahoo Messenger
# Delete the following 6 lines if you don't want people to connect to Yahoo Messenger
acl YIM_ports port 5050
acl YIM_domains dstdomain .yahoo.com .yahoo.co.jp
acl YIM_hosts dstdomain scs.msg.yahoo.com cs.yahoo.co.jp
acl YIM_methods method CONNECT
http_access allow YIM_methods YIM_ports YIM_hosts
http_access allow YIM_methods YIM_ports YIM_domains
# Allow connections to Google Talk
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl GTALK_ports port 5222 5050
acl GTALK_domains dstdomain .google.com
acl GTALK_hosts dstdomain talk.google.com
acl GTALK_methods method CONNECT
http_access allow GTALK_methods GTALK_ports GTALK_hosts
http_access allow GTALK_methods GTALK_ports GTALK_domains
# Allow connections to MSN
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl MSN_ports port 1863 443 1503
acl MSN_domains dstdomain .microsoft.com .hotmail.com .live.com .msft.net .msn.com .passport.com
acl MSN_hosts dstdomain messenger.hotmail.com
acl MSN_nets dst 207.46.111.0/255.255.255.0
acl MSN_methods method CONNECT
http_access allow MSN_methods MSN_ports MSN_hosts

Now, start the squid proxy server as

service squid start

Also, if you want squid to be started every time you boot the machine, execute the following command

chkconfig --level 345 squid on

You have a squid proxy server running now. You can ask clients to configure there browsers to use 192.168.36.204 as a proxy server with 8080 as proxy port. Command line utilities like elinks, lynx, yum, wget etc. can be asked to use proxy by exporting http_proxy variable as below. Users can also add these lines to ~/.bashrc file to avoid exporting every-time.

export http_proxy='http://192.168.36.204:8080'
export ftp_proxy='http://192.168.36.204:8080'

I highly recommend the book “Squid Proxy Server 3.1: Beginner’s Guide (Paperback)” for further reading.

 

How To: Configure Hierarchicy of Proxy Servers (Squid)

Yesterday I came across this idea of caching all the data that I browse on my hard disk so that the average load time of a website decreases. Actually the idea is I’ll cache all the static data that I browse like images, static html pages, CSS files and similar things which does not change frequently and can be served from the cache. But while setting up the proxy server on my machine, I faced the problem that my machine which is going to act as a proxy server is behind my institute’s proxy. So, a simple caching proxy server can’t serve my needs and I have to really figure out how to setup a hierarchical proxy server. Below we’ll see how to setup a hierarchical proxy server.

Approach

When I thought of setting up a caching proxy server, squid immediately struck my mind. Actually I don’t know about any other proxy servers. I never setup proxy server before this ( I tried a lot of time, but in vain). So, I started googling about squid setup. There were a lot of tutorials, but either they were too small to get things going or they were too verbose that I couldn’t manage to read them. So, I directly jump into squid configuration file squid.conf . And with references from here and there, I managed to setup the proxy server successfully.

Note: The configurations below worked on Fedora 7 with squid 2.6STABLE16. The same configurations may work with other squid versions and on other operating systems as well, but try them at your own risk.

Part 1 : Setting up simple proxy server with squid

Setting up a very simple and usable proxy server is really easy. You need to add/edit only 2-3 lines /etc/squid/squid.conf to get started.

Add your ip to the access list.

1
2
3
acl myip src 172.17.8.175 #<your_ip_which_will_use_the_proxy_server> (e.g. )
http_access allow myip
http_port 8080 #<http_proxy_port> (this is 3128 by default. you can set it to anything you like. e.g. 8080)

Save the squid.conf file. Then issue these commands.

1
2
[root@localhost squid]# squid -z [Enter] (as root) (This needs to be executed only once.)
[root@localhost squid]# service squid start [Enter] (as root)

If you want to start the squid server on boot, issue this command.

[root@localhost squid]# chkconfig --level 345 squid on [Enter] (as root)

Now, your machine is a proxy server. You can setup your browser to use the machine as a proxy server.

Conditions

The proxy server will work only if your machine has a public IP and is directly connected to internet.

Part 2: Setting up a hierarchical caching proxy server with squid

The above setup works fine if a machine is directly connected to internet. But my machine itself is behind a proxy, so setting up a proxy on my machine is of no use unless the proxy on my machine uses the institute proxy for connecting to internet. So, here we jump into squid.conf again and this time we have to really do some brain storming. If you are a newbie to Linux and don’t know how to make a system work when nothing seems to help, you will probably be better off by using institute’s proxy.

Here is the scenario.

1
2
3
4
5
6
7
8
9
10
11
12
13
1. Your browser sends a content request to proxy on your machine.
2. Check: if a cache HIT from institute proxy cache (HIT means content was found in cache)
	2a. Check: if content is older than the original upstream content
		2aa. Fetch content from upstream and serve the client
	2b. else
		2ba. Serve the content from the cache
3. Check: if cache HIT from proxy on your machine
	3a. Check: if content is older than the original upstream content
		3aa. Fetch content from upstream and serve the client
	3b. else
		3ba. Serve the content from the cache
4. Cache MISS from both the proxies
	4a. Fetch the content from upstream and serve the client

The above method of operation is very basic and is my understanding of squid. It may not be the exact squid behavior.

Now, lets see the configurations needed for setting up the hierarchical caching proxy server with squid.

Assumptions

I assume that we already have squid setup at institute’s proxy whether in caching mode or not. The best way to add/edit the following lines in your squid.conf is to search for particular parameter and then edit the value to set as given.

I also assume that you have simple proxy server setup on your machine and now we want to make it act as child proxy of the institute’s proxy.

Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
# Your local machine will act as a sibling proxy
cache_peer 172.17.8.175 sibling 3128 3130 no-query weight=10
# The institute's proxy server will act as a parent proxy
# 'default' mean the last-resort
cache_peer 192.168.36.204 parent 8080 3130 no-query proxy-only no-digest default
# allow accessing peer cache for access list 'myip'
cache_peer_access 172.17.8.175 allow myip
# Don't cache dynamic content
hierarchy_stoplist cgi-bin ?
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
# Size of main memory to be used for caching
cache_mem 200 MB
# max size of content to be stored in main memory
maximum_object_size_in_memory 7000 KB
# policy for cache replacement if memory is full
cache_replacement_policy heap LFUDA
# the directory to be used for storing cache on your hdd
cache_dir aufs /var/spool/squid 200 16 256
# max file descriptor open at a time .. 0(unlimited)
max_open_disk_fds 0
# min object size to cache on hdd
minimum_object_size 0 KB
# max object size to cache on hdd
maximum_object_size 16384 KB
# access log
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320
store_avg_object_size 20 KB
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
refresh_stale_hit 5 seconds
acl SSL_ports port 443 563 1863 5190 5222 5050 6667
# Allow AIM protocols
acl AIM_ports port 5190 9898 6667
acl AIM_domains dstdomain .oscar.aol.com .blue.aol.com .freenode.net
acl AIM_domains dstdomain .messaging.aol.com .aim.com
acl AIM_hosts dstdomain login.oscar.aol.com login.glogin.messaging.aol.com toc.oscar.aol.com irc.freenode.net
acl AIM_nets dst 64.12.0.0/255.255.0.0
acl AIM_methods method CONNECT
http_access allow AIM_methods AIM_ports AIM_nets
http_access allow AIM_methods AIM_ports AIM_hosts
http_access allow AIM_methods AIM_ports AIM_domains
# Allow Yahoo Messenger
acl YIM_ports port 5050
acl YIM_domains dstdomain .yahoo.com .yahoo.co.jp
acl YIM_hosts dstdomain scs.msg.yahoo.com cs.yahoo.co.jp
acl YIM_methods method CONNECT
http_access allow YIM_methods YIM_ports YIM_hosts
http_access allow YIM_methods YIM_ports YIM_domains
# Allow GTalk
acl GTALK_ports port 5222 5050
acl GTALK_domains dstdomain .google.com
acl GTALK_hosts dstdomain talk.google.com
acl GTALK_methods method CONNECT
http_access allow GTALK_methods GTALK_ports GTALK_hosts
http_access allow GTALK_methods GTALK_ports GTALK_domains
# Allow MSN
acl MSN_ports port 1863 443 1503
acl MSN_domains dstdomain .microsoft.com .hotmail.com .live.com .msft.net .msn.com .passport.com
acl MSN_hosts dstdomain messenger.hotmail.com
acl MSN_nets dst 207.46.111.0/255.255.255.0
acl MSN_methods method CONNECT
http_access allow MSN_methods MSN_ports MSN_hosts
# Turn this off if hierarchical behavior is needed
nonhierarchical_direct off
never_direct deny myip
hosts_file /etc/hosts
coredump_dir /var/spool/squid

That’s the minimal configuration you need for running squid in hierarchical way. Save the squid.conf file and start/restart/reload the squid service. Setup your browser to use your machine as proxy and while using it’ll cache all the static content. You should experience some reduction in average page load time.

Advantages

I am currently using squid in above configuration. And its turning out to be nice for me. I am browsing websites faster and saving a chunk of bandwidth for my institute.

Disadvantages

Introduction of another proxy server increases the latency for dynamic content.

Notice

The above configurations and views are a result of my understanding of squid. If you feel this may break your system or it may have adverse effects, don’t use them. At least don’t use these on a production system.

 

How To: Install FFMPEG and FFMPEG-PHP

I was randomly browsing the internet and reading about making a website look better and I encountered ffmpeg-php. ffmpeg is a very powerful tool to record, convert and stream audio and video. Its a very rich tool almost supporting every format out there in the world. It can convert any format to any other format provided the codec. ffmpeg-php is an extension for PHP that provides a rich library to access info about audio and video files. The good thing about ffmpeg-php is that it can retrieve all info about any audio/video file subjected to the condition that the particular audio/video format is supported by your ffmpeg installation. So, now you have a clear idea that you can do wonders with audio/videos while showing them on your site :)

I tried some of the functionalities and they worked out of the box. Here’s is complete how to on installing ffmpeg and ffmpeg-php.

FFMPEG:

I tried installing ffmpeg from rpms provided by several Fedora repositories but after installation ffmpeg doesn’t seem to work. After several tries, I installed ffmpeg from source rpms and it worked. Below, I will describe how to install ffmpeg from source rpm.

Step 1:

Make sure that you have ‘rpmbuild’ installed by issuing

[root@bordeaux saini]# rpm -q rpmbuild [Enter]

command. If the above says that rpmbuild is not installed, then install it using yum as given below

[root@bordeaux saini]# yum install rpmbuild [Enter] (do as root)

Step 2:

Download the latest src rpm of ffmpeg from rpmfind.net. Issue the command given below

[root@bordeaux saini]# rpm -hiv ffmpeg-x.x.x.xx-xxx.src.rpm [Enter] (do as root)

Step 3:

Go to ‘/usr/src/redhat/SPECS/’ directory and issue the command given below

1
2
[root@bordeaux saini]# cd /usr/src/redhat/SPECS/ [Enter]
[root@bordeaux SPECS]# rpmbuild -ba ffmpeg.spec [Enter] (do as root)

If it gives an error like package ‘xyz’ is need by ffmpeg. Then install the package ‘xyz’ using yum as

[root@bordeaux SPECS]# yum install xyz [Enter] (do as root)

After installing the dependencies, issue the rpmbuild command ‘rpmbuild -ba ffmpeg.spec’. Now ffmpeg rpms will be build and they will be stored in ‘/usr/src/redhat/RPMS/i386/’.

Step 4:

Go the ‘/usr/src/redhat/RPMS/i386/’ (x86_64 instead of i386 if your OS is 64 bit). Install all the rpms that were built by rpmbuild.

[root@bordeaux saini]# rpm -hiv *.rpm [Enter] (do as root)

Thats it. ffmpeg is now successfully installed on your computer. Half the job is done. Now lets proceed with ffmpeg-php installation.

FFMPEG-PHP:

We will install ffmpeg-php from source bundle.

Step 1:

Make sure that ‘php-devel’ installed on your machine by issuing

[root@bordeaux saini]# rpm -q php-devel [Enter]

command. If the above command says the ‘php-devel’ is not installed, then install it using the following command.

[root@bordeaux saini]# yum install php-devel [Enter] (do as root)

Step 2:

Download the latest version of ffmpeg-php from here. Unpack the file you have downloaded.

1
2
[root@bordeaux saini]# bunzip2 -d ffmpeg-php-0.5.1.tbz2 [Enter]
[root@bordeaux saini]# tar -xvf ffmpeg-php-0.5.1.tar [Enter]

Step 3:

Issue the following command in sequence if everything goes fine.

1
2
3
4
5
[root@bordeaux saini]# cd ffmpeg-php-0.5.1 [Enter]
[root@bordeaux ffmpeg-php-0.5.1]# phpize [Enter]
[root@bordeaux ffmpeg-php-0.5.1]# ./configure [Enter]
[root@bordeaux ffmpeg-php-0.5.1]# make [Enter]
[root@bordeaux ffmpeg-php-0.5.1]# make install [Enter] (do as root)

Step 4:

Open ‘/etc/php.ini’ and add a line ‘extension=ffmpeg.so’ in the category ‘Dynamic Extensions’. For help see the image below.
FFMPEG PHP PHPini Module

Step 5:

Restart apache web server aka ‘httpd’ service by issuing the command.

[root@bordeaux saini]# service httpd restart [Enter] (do as root)

Step 6:

Write a test php file and test your ffmpeg-php installation.

phpinfo();

Save the above code in ‘info.php’ and save the file in ‘/var/www/html/’ and browse http://localhost/info.php . If you see something like this.
FFMPEG PHP Linux
Then the ffmpeg-php is successfully installed on your machine. Now you can jump into the world of video manipulation via your website.

 

How To: Configure VNC Server

Vncserver is just another application available in almost all the available Linux based distros. Configuring vncserver is very easy. But the default desktop view in vncviewer is gray scale desktop with very pathetic GUI. To view normal Gnome or KDE desktop in vncviewer, assigning a custom port for usage, user configuration and to adjust the resolution of the vncviewer window, some files need to be configured properly. Here are the required configuration in any version of Fedora/Fedora Core. But with minor modifications these can be applied to other distros like Ubuntu, SuSE, Gentoo etc.

Step 1: Installing Vncviewer, Vncserver

As root do this

[root@bordeaux saini]# yum install vnc [Enter]

Step 2: Configuring resolution, port & user

The default location of server configuration file for vncserver is ‘/etc/sysconfig/’. To configure the resolution, user and port open ‘/etc/sysconfig/vncservers’ in you favorite editor and add two lines per user configuration shown ..

VNCSERVERS=":"
VNCSERVERARGS[]="-geometry x x"

Example :

VNCSERVERS="3:saini"
VNCSERVERARGS[3]="-geometry 1000x700"

You can choose any display port, but it should not be in use by another X server. Window height and width can be anything (not in fraction of course). But keep in mind that the system on which you are going to view the desktop using vncviewer should have greater resolution than what you specify here, otherwise scrollbars will appear.

Step 3: Configuring Desktop Environment

The user specific configuration files of vncviewer resides in ‘.vnc’ directory in user’s home directory. (e.g. ‘/home/saini/.vnc/’). Open ‘.vnc/xstartup’ in your favorite editor and edit as below

  • For Gnome

The ‘xstartup’ file shout look like this

#!/bin/sh
 
# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc
 
[ -x /etc/vnc/xstartup ] &amp;&amp; exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] &amp;&amp; xrdb $HOME/.Xresources
#xsetroot -solid grey
#vncconfig -iconic &amp;
#xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &amp;
#twm &amp;
startx &amp;
  • For KDE

The ‘xstartup’ file should look like this

#!/bin/sh
 
# Uncomment the following two lines for normal desktop:
#unset SESSION_MANAGER
#exec /etc/X11/xinit/xinitrc
 
[ -x /etc/vnc/xstartup ] &amp;&amp; exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] &amp;&amp; xrdb $HOME/.Xresources
#xsetroot -solid grey
#vncconfig -iconic &amp;
#xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &amp;
#twm &amp;
startkde &amp;

Step 4: Configuring password

For setting up vncviewer password for user ‘xyz’, login as user ‘xyz’ and issue ‘vncpasswd’ command on a terminal/konsole (or whatever). Enter password twice and you are done with password setting.

Step 5: Starting Vncserver

To start vncserver, login as root and issue ‘service vncserver start’ command. If service started successfully, you are ready to use vncviewer on a remote/local machine.

Step 6: Accessing through Vncviewer

  • From Linux based machines
[saini@bordeaux saini]# vncviewer IP_Address:displayPort [Enter] #(IP Address is for the machine where you set up vncserver)
  • From Windows

On windows there is a software called RealVnc. Install it and enter <IPAddress>:<displayPort> in the dialog box.
[I explained how to connect via windows because it may help someone get a bit of relief.]

Screenshots:

  • Vncviewer with Gnome as Desktop environment

VNC Server VNC Viewer Screenshot GNOME

  • Vncviewer with KDE as Desktop environment

VNC Server VNC Viewer Screenshot KDE

Vnc is nice tool if you want to avoid using windows. I use it all the time. While I am forced to work on windows system, I install vncviewer and use it in full screen mode 😛 It also helps when you want to run some gui based application and monitor is remotely. Because if you close vncviewer window and use vncviewer again, you will be given the desktop session where you left it (all windows open and applications running).

 

How To: Install PHP-Ming

Yesterday, I came across a library called ming for php which can do wonders while converting videos to .swf (flash), images to .swf and much more. The best thing is that ming provides wrappers in C/C++/PHP/Python/Perl/Tcl etc. So, can be used within any one of these languages. But, the damn thing drove me crazy while installing. It was very tough experience installing it as the available howtos didn’t help much. But finally I somehow managed with the installation and it worked :) So, I thought of writing this how to, so that anyone having the same problem can find it helpful.

Here is a complete howto on installing php-ming in Fedora. (I installed it on Fedora 7 and with php5, but howto is supposed to work with other versions as well.) Though there are rpms available for earlier versions of ming, but I wanted the latest, so installed from source.

Resources:
libming : http://www.libming.net/
PHP : http://php.net/

Step 1

First of all make sure that you have php-devel package, otherwise you will not be able to install ming at all. Do the following as root.

[root@bordeaux root]# rpm -q php-devel [Enter]

If the above command says php-devel is not installed, then fetch php-devel for your php version from here and install as given below, else proceed to step 2.

[root@bordeaux root]# rpm -hiv php-devel-5.2.2-3.i386.rpm [Enter]

Step 2

Download latest version of ming from here.

Step 3

Untar the downloaded package.

[saini@bordeaux saini]# tar -xvzf ming-0.4.0.beta5.tar.gz [Enter]

Step 4

Change directory to ‘./ming-0.4.0.beta5′ and issue ./configure and make as given below

1
2
3
4
[saini@bordeaux saini]# cd ming-0.4.0.beta5 [Enter]
[saini@bordeaux ming-0.4.0.beta5]# ./configure --enable-cXX --enable-php [Enter]
[saini@bordeaux ming-0.4.0.beta5]# make [Enter]
[root@bordeaux ming-0.4.0.beta5]# make install [Enter] (as root)

Step 5

Go to ‘./php_ext’ directory and issue ‘make’ and ‘make install’.

1
2
3
[saini@bordeaux ming-0.4.0.beta5]# cd php_ext [Enter]
[saini@bordeaux php_ext]# make [Enter]
[root@bordeaux php_ext]# make install [Enter] (as root)

Step 6

Open make_modules.sh in your favorite editor and uncomment the line ‘make install’ and run make_modules.sh.

[root@bordeaux php_ext]# bash make_modules.sh [Enter] (as root)

Step 7

Go to ‘./tmp/modules/’ directory and copy ming.so to ‘/usr/lib/php/modules/’

1
2
[root@bordeaux php_ext]# cd tmp/modules/ [Enter]
[root@bordeaux modules]# cp ming.so /usr/lib/php/modules/ [Enter] (as root)

If ming.so is not found in ‘php_ext/tmp/modules/, then update your db using ‘updatedb’ and locate ming.so and if found on locate, then copy it to ‘/usr/lib/php/modules/’. If not found at all, try recompiling or issue some random commands related to make.

Step 8

Go to ‘/usr/local/lib/’ and check if libming.so, libming.so.0 etc. are present there. If yes proceed else I don’t know what to do :(

Step 9(a)

Go to ‘/usr/lib/php/modules/’ and issue these commands.

1
2
3
4
[root@bordeaux modules]# ln -s /usr/local/lib/libming.so libming.so (as root)
[root@bordeaux modules]# ln -s /usr/local/lib/libming.so.0 libming.so.0 (as root)
[root@bordeaux modules]# ln -s /usr/local/lib/libming.so.0.3.0 libming.so.0.3.0 (as root)
[root@bordeaux modules]# ln -s /usr/local/lib/libming.so.0.4.0 libming.so.0.4.0 (as root)

Step 9(b)

Go to ‘/usr/lib/’ and issue these commands.

1
2
3
4
[root@bordeaux lib]# ln -s /usr/local/lib/libming.so libming.so (as root)
[root@bordeaux lib]# ln -s /usr/local/lib/libming.so.0 libming.so.0 (as root)
[root@bordeaux lib]# ln -s /usr/local/lib/libming.so.0.3.0 libming.so.0.3.0 (as root)
[root@bordeaux lib]# ln -s /usr/local/lib/libming.so.0.4.0 libming.so.0.4.0 (as root)

Step 10

Restart the httpd service by issuing the command below

[root@bordeaux root]# service httpd restart [Enter] (as root)

Step 11

Write a file ‘/var/www/html/info.php’ with following contents

phpinfo();

and point your web browser to http://localhost/info.php . Now search for ‘ming’ and check whether its enabled or not.

PHP Ming in Fedora

If enabled, yes you have successfully installed php-ming :) Hope this helps.

 

How To: Install and Configure Shoutcast Radio

Shoutcast is a mp3 broadcasting/streaming media server software provided by NullSoft. One can setup a server on any system, GNU/Linux / Windows/ MacOS and can stream mp3 over the network, internet/intranet. I setup shoutcast on my system a long back and found it very useful. Here is a step by step how to on how you can setup shoutcast on a GNU/Linux system.

Shoutcast can be installed even if you don’t have root privileges. But in that case you can’t use port less than 1024 for broadcasting. Below, I’ll explain how to install it for a non-root user.

Shoutcast server depends on a tool shoutcast DNAS for audio input in Linux. So, here we go

Step 1

Download the latest version of shoutcast from here. Download the one for Linux (glibc).

Step 2

Let us assume we want to install shoutcast in a directory named ‘shoutcast’ in user’s home directory and we want to broadcast punjabi songs.

1
2
3
[saini@bordeaux shoutcast]# tar -xvzf sc_serv_1.9.8_Linux.tar.gz [Enter]
[saini@bordeaux shoutcast]# mkdir punjabi [Enter]
[saini@bordeaux shoutcast]# mv sc_serv.conf punjabi/sc_serv_punjabi.conf [Enter]

Step 3

Open sc_serv_punjabi.conf in your favorite editor and modify certain parameters as per you requirements. The essentials are below.

1
2
3
4
5
6
7
8
9
10
11
12
13
MaxUser=20
Password=yourPassword
PortBase=8300 #(Confirm that this port and the port PortBase+1 is not being used)
LogFile=none
RealTime=0
ScreenLog=0
ShowLastSongs=10
SrcIP=ANY
DestIP=ANY
Yport=80
NameLookups=0
AdminPassword=yourAdminPassword
TitleFormat=%s [IIIT Radio]

etc. My sc_serv.conf can be accessed here. That was all for installing the shoutcast server.

Now, the installation of DNAS tool is still pending. Here is a step by step procedure to install DNAS tool.

Step 1

Download the latest version of DNAS tool from here.

Step 2

1
2
3
4
[saini@bordeaux shoutcast]# tar -xvzf sc_trans_posix_040.tgz [Enter]
[saini@bordeaux shoutcast]# cd sc_trans_040/ [Enter]
[saini@bordeaux sc_trans_040]# mv sc_trans_linux ../ [Enter]
[saini@bordeaux sc_trans_040]# mv sc_trans.conf ../punjabi/sc_trans_punjabi.conf [Enter]

Step 3

Go to punjabi directory and open sc_trans_punjabi.conf in your favorite editor and make changes according to your needs. Here are some

1
2
3
4
5
6
7
8
9
PlaylistFile=/exactPathTo/punjabi.lst
ServerIP=
ServerPort=
 # 8300 in this case
Password=
 # yourPassword in this case
StreamTitle= %s
StreamURL=
Shuffle=1 # (1 for random songs)

etc. My sc_trans.conf can be accessed here.

Step 4

Generate a list of all the songs (mp3) you have and put it in punjabi.lst in punjabi directory.

[saini@bordeaux punjabi]# find /pathToPunjabiDir/ -type f -name "*.mp3" > punjabi.lst [Enter]

My dummy punjabi.lst can be accessed here.

The configuration part of shoutcast server with audio input is complete. Now we have to run the server so that we can listen to music.

Go to the shoutcast directory and run the sc_serv first and then run the sc_trans_linux. Here is way to do that.

1
2
[saini@bordeaux shoutcast]# ./sc_serv punjabi/sc_serv_punjabi.conf > /dev/null 2> /dev/null &
[saini@bordeaux shoutcast]# ./sc_trans_linux punjabi/sc_trans_punjabi.conf > /dev/null 2> /dev/null &

Now your system is a shoutcast server. Any client can use mplayer, vlc, amarok or any other multimedia player that support streaming media to listen to the music being played on your server.

1
2
[saini@bordeaux saini]# mplayer http://yourIp:port [Enter]
[saini@bordeaux saini]# mplayer http://localhost:8300 [Enter] # (in the above case).

If you want shoutcast to start every time your system boots. Put these lines in /etc/rc.local

1
2
/home/saini/shoutcast/sc_serv /home/saini/shoutcast/punjabi/sc_serv_punjabi.conf > /dev/null 2> /dev/null &
/home/saini/shoutcast/sc_trans_linux /home/saini/shoutcast/punjabi/sc_trans_punjabi.conf > /dev/null 2> /dev/null &

Shoutcast is fun and its more fun when everyone listens to what you are listening to :)

 

How To: Install WordPress on LAMP

Recently, I bought this domain name and shifted all my blogs here on saini.co.in . As I chose wordpress as the software to blog. So, just installed it and started using it right away. Installing wordpress is one of the most easiest things that you’ll ever do. Below is a step by step howto on installing wordpress and importing your current wordpress.com blog to your domain.

Step 1:

Download latest version of wordpress from here.

Step 2:

Go to the directory where you want to install it. Here I am taking gofedora.com as reference website where we want to install the wordpress and ~/htdocs/ as the web directory for gofedora.com . Modify above details in accordance with your needs.

If you downloaded .zip format then do this

[saini@bordeaux htdocs]# unzip wordpress-X.X.X.zip [Enter]

If you download .tar.gz format then do this

[saini@bordeaux htdocs]# tar -xvzf wordpress-X.X.X.zip [Enter]

Step 3:

[saini@bordeaux htdocs]# mv wordpress/* . [Enter]

If you visit http://gofedora.com now, you’ll get an error as shown in the image below …

Wordpress Error Without wp-config

Step 4:

Open the file wp-config-sample.php in your favorite text editor and edit the things shown in the image below.

Sample wp-config

Enter your database name, database username, database password and the table prefix for the wordpress database. e.g. one can use linux_ as table prefix. If you have the database on the same system, then there is no need to change the database host otherwise change it accordingly.

Step 5:

[saini@bordeaux htdocs]# mv wp-config-sample.php wp-config.php [Enter]

Step 6:

Go to http://gofedora.com/ and you’ll see a welcome page if you gave correct setting in wp-config.php . The page is like in the image below.

Wordpress Blog Config

Fill the settings properly. Anyway you can change them afterwards. Fill the blog name and email id and click Install WordPress.

Step 7:

You’ll see a web-page like in the image below, if everything goes fine. Copy the password on the page and paste at some temporary place and click login.

Wordpress Copy Password

Step 8:

You’ll be presented with a login page. Enter ‘admin’ as username and the password you copied from the previous page.

Wordpress Admin Login

Step 9:

Congratulations ! You have successfully installed wordpress and created the admin login. Go to Users -> Authors & Users and click Edit in the row with admin as author. Change the password on the next page as below ..

Wordpress Change Password

This is all you have to do to install wordpress. Below is how you import wordpress.com blog to your domain.

Step 1:

Login as admin on you wordpress.com blog and go to Manage -> Export and download the exported blog to your system as show in the image below.

Wordpress Export Blog

Step 2:

Create directory named ‘uploads’ in ~/htdocs/wp-content/ and change permissions to 777 so that wordpress can upload your blog there.

1
2
3
[saini@bordeaux htdocs]# cd wp-content [Enter]
[saini@bordeaux wp-content]# mkdir uploads [Enter]
[saini@bordeaux wp-content]# chmod 777 uploads [Enter]

Step 3:

Go to Manager -> Import section and click on ‘wordpress’ . Upload the .xml file that you download from your wordpress.com blog in step 1 on the page as shown in the image below ..

Wordpress Import Blog

Step 4:

If uploading goes successful, then all your posts will be imported from your wordpress.com blog and will be stored in the database as gofedora.com . Now it will ask you for the author of the new blog posts at gofedora.com as show below …

Choose New Author

Step 5:

Change the permission of uploads back to normal as show below

[saini@bordeaux wp-content]# chmod 744 uploads [Enter]

Everything is done. Now you blog has been shifted to your domain and you can go on with the new blog. But keep in the mind the the images you used in wordpress.com blog posts are not stored in your local database, they are still linked back to your wordpress.com blog images. So, don’t just go and delete your wordpress.com blog. That would be fatal.

The steps for installing wordpress on a windows system are the same, but you have to use some notepads and whatever to edit files :)

Enjoy blogging at your domain. Hope the above howto helps.