How To: Install ATI Catalyst (fglrx) Drivers

Note: This How To is valid up to Catalyst Driver version 10.1.
Note: With minor changes this How To may work for other Linux distributions like Mandriva, Suse, Ubuntu, Red Hat, Cent OS etc.

Following the suggestions in comments on my last post about ATI Catalyst (fglrx), I tried to reinstall ATI drivers version 9.8 and finally it worked. I thought of noting down the the process in few simple steps which anyone can follow and get the drivers working on Fedora 11 with latest kernel versions.

Step 1 : Update Kernel, Install kernel-devel

Huge thanks to Richard Lloyd and mxyzptlk2063 for pointing this out.
This step is not necessary. You may like to stick to the kernel version you already have. In case you want the latest kernel, just use yum to get the latest kernel.

Note: Do not forget to update the kernel-devel package as well as its needed to compile the fglrx kernel module.

DONT UPDATE THE KERNEL. Instead stick to whatever kernel version you have. If you have 2.6.30.*, you are unlucky :( Get a kernel version 2.6.29.* and install the kernel-devel for the same.

With Catalyst Driver version 9.10 you can update kernel to latest version before installing the Drivers.

[root@fedora ~]$ yum update kernel
[root@fedora ~]$ yum install kernel-devel

Step 2 : Download Drivers

Download the ATI Catalyst™ 9.8 Proprietary Linux x86 Display Driver from ATI/AMD website.

Step 3 : Install Drivers

Install the drivers that you have downloaded using the following command.

[root@fedora ~]$ bash ./ati-driver-installer-9-8-x86.x86_64.run

Step 4 : Check Installation Result

Check the /usr/share/ati/fglrx-install.log file for result of installation. If there are no errors, proceed to the next step.

Note: You may see lines like this at the end of file

You must change your working directory to /lib/modules/fglrx
and then call ./make_install.sh in order to install the built module.
– recreating module dependency list
– trying a sample load of the kernel modules
done.

You can ignore these lines happily if they don’t contain any error message. You don’t really need to execute the command mentioned in those lines.

Step 5 : Blacklist radeon and radeonhd kernel modules

Add these lines to /etc/modprobe.d/blacklist.conf.

blacklist radeon
blacklist radeonhd

Step 6 : Generate initial Xorg configuration file

Note: This step is optional with Catalyst Driver version 9.10, but it wont harm even if you execute this.

You have to generate the initial xorg.conf file which will use fglrx as display device.

[root@fedora ~]$ aticonfig --initial

Step 7 : Modify xorg.conf

Open your /etc/X11/xorg.conf file and add the following line to “Device” section (the one with fglrx as driver)

Option "SWCursor" "true"

Step 8 : Reboot

Reboot your machine. And you’ll have the latest ATI Catalyst drivers working on your Fedora 11 :)

Step 9 : Testing

You can test your newly installed drivers and get the performance benchmarks for your graphics card using the commands below.

[saini@fedora ~]$ glxgears
[saini@fedora ~]$ fgl_glxgears

My graphics card is ATI Radeon HD 3200 (256MB, Onboard) and I get 1500FPS with glxgears and 300FPS with fgl_glxgears.

In case you messup things somewhere, you can uninstall the fglrx drivers using the following command

[root@fedora ~]$ /usr/share/ati/fglrx-uninstall.sh
 

How To: Wireless LAN with Broadcom BCM4312 in Fedora 11

Fedora 11 does have support for Broadcom wireless drivers, but it didn’t really work out on a friend’s laptop. Finally we got it working and I thought I’ll just note the steps down. Below are the three easy steps you need to take to make it work properly.

Step 1

Install needed packages

[root@fedora ~]$ yum install broadcom-wl  wl-kmod

Step 2

Once the packages are installed successfully, reboot your laptop.

Step 3

Use the following command

[root@fedora ~]$ system-config-network

And add a new wireless device wlan0 or whatever you want by filling the required fields properly. If you want the device to be managed by NetworkManager, you can do so while editing the device you just added.

Activate the device. And you are on wifi :)

 

How To: Install/Configure GNUMP3d – Streaming Audio Server

Mission

GNUMP3d is the GNU Streaming MP3/Media Server written in perl. Our mission is to setup GNUMP3d and stream audio over LAN or over internet. Below are the essential steps to install and configure GNUMP3d.

Download

Download latest version of GNUMP3d from GNUMP3d Website.

Extract

[kulbirsaini@fedora ~]$ tar -xjf gnump3d-x.x.tar.bz2

Install

[root@fedora ~]$ cd gnump3d-3.0
[root@fedora ~]$ make install (as root)

Now gnump3d is installed on your system. Now you need to configure it according to your taste.

Configure

The configuration file is located at /etc/gnump3d/gnump3d.conf. For casual use, you just need to configure port, binding_host and root.

# Port to which gnump3d will be accessible via web interface or via a media player like xmms or winamp.
port = 1111
# The IP Address where gnump3d will bind itself.
binding_host = 172.17.8.64
# If you want the stream to be accessible via a fully qualified domain name, set hostname variable.
# You don't need to set this in most cases e.g. while setting up gnump3d on LAN.
hostname = gofedora.com
# The directory where are your music files resides.
root = /stuff/Music/

Though you can skip rest of the configuration, you may try to explore other options. My gnump3d.conf file can be download from here.

Thats all you need to do to configure gnump3d.

Indexing

Now you need to index all you music collection (the audio files in gnump3d root). Run the following command to index

[root@fedora ~]$ gnump3d-index --verbose

Run gnump3d

Once the indexing is done, you are all set to run gnump3d. By default gnump3d tries to index all files whenever you start it, to avoid this we need to use –fast option.

[root@fedora ~]$ gnump3d --fast

By default gnump3d runs in foreground. If you want it to go in background and run quietly, run it as follows.

[root@fedora ~]$ gnump3d --fast --background

Accessing Media Server

To access your gnump3d streaming media server, please visit url http://ip_address:port/ .

Run at startup

If you want gnump3d to start when your computer starts add the following line to /etc/rc.local file.

gnump3d --fast --background

Feel free to comment in case you have a problem.

 

How To: Install and Configure GitWeb

UPDATE : I recommend using GitList instead of GitWeb. GitList is much easier to setup and has a better web interface. Continue reading this post if you looking for GitWeb setup instructions specifically.

Goal

Setting up gitweb (web interface for SCM software git) for your project’s git repository for public access and developer commits via ssh.

Assumptions

  1. You already have your project’s git repository.
  2. You have hosting space somewhere to host gitweb.
  3. You have root access.
  4. You are using Apache as webserver.

Example for this howto

Project : VideoCache
Domain for gitweb : git.cachevideos.com
URL for git access for videocache : http://git.cachevideos.com/videocache.git
Actual path on server : /home/saini/domains/cachevideos.com/git
Git repository : /home/saini/projects/videocache/

Installation

Installation is pretty easy. Just one single command would do everything.

[root@localhost ~]# yum install gitweb (do as root)

This will create a directory /var/www/git which is default for gitweb.

Copy the directory /var/www/git/ to /home/saini/domains/cachevideos.com/git

[root@localhost ~]# cp -r /var/www/git /home/saini/domains/cachevideos.com/git

Configuration

1. GitWeb

Open the file /etc/gitweb.conf (it may or may not be there) and add the following lines to it.

# Change This
$projectroot = '/home/saini/domains/cachevideos.com/git';
# Change This
$site_name = "Kulbir Saini's git trees.";
# Don't Change the variables below
$my_uri = "/";
$home_link = '/';
@stylesheets = ("/gitweb.css");
$favicon = "/git-favicon.png";
$logo = "/git-logo.png";

2. Apache

Open the file /etc/httpd/conf.d/git.conf and clear all the lines that are already there and add the following lines to it

  DocumentRoot /home/saini/domains/cachevideos.com/git
  ServerName git.cachevideos.com
  ErrorLog "/home/saini/domains/cachevideos.com/logs/error_log"
  CustomLog "/home/saini/domains/cachevideos.com/logs/access_log" combined
  SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
  DirectoryIndex gitweb.cgi
 
    Allow from all
    AllowOverride all
    Order allow,deny
    Options +ExecCGI
    AddHandler cgi-script .cgi
 
      SetHandler cgi-script
 
    RewriteEngine on
    RewriteRule ^[a-zA-Z0-9_\-]+\.git/?(\?.*)?$ /gitweb.cgi%{REQUEST_URI} [L,PT]

3. Git repository configuration

Go to your git repository (/home/saini/projects/videocache/) and make the following changes.

(a). Open file .git/description and add a short nice description for your project.

videocache is a squid url rewriter plugin written in Python to facilitate youtube, metacafe, dailymotion, google, vimeo, msn soapbox, tvuol.uol.com.br, blip.tv, break.com videos and wrzuta.pl audio caching.

(b). Open file .git/config and append the following lines

[gitweb]
  owner = "Kulbir Saini"

Copy project’s git repository for gitweb

Copy the /home/saini/projects/videocache/.git directory to /home/saini/domains/cachevideos.com/git/videocache.git

[root@localhost ~]# cp -r /home/saini/projects/videocache/.git /home/saini/domains/cachevideos.com/git/videocache.git

Finishing Step

Restart Apache webserver.

[root@localhost ~]# service httpd restart

Now you can browser a list of your projects’ git repositories at http://git.cachevideos.com/ .

Adding another project repository

Just copy the project repository’s .git directory to /home/saini/domains/cachevideos.com/git/prjoect_name.git. And it’ll be shown on the list.

Committing (pushing) to the repository

For committing to the repository via ssh use the following command.

# Pushing everything (Please see the username)
[root@localhost videocache]# git push --all ssh://saini@git.cachevideos.com/~saini/domains/cachevideos.com/git/videocache.git

To update tags on the remote repository use this command.

# Pushing all tags
[root@localhost videocache]# git push --tags ssh://saini@git.cachevideos.com/~saini/domains/cachevideos.com/git/videocache.git

Well, if you consider just the web interface and committing part for your project, thats all. But things can be fine tuned further. Below are few hacks!

1. Enabling nice urls.

By default the urls for browsing repository via git web are pretty crappy and difficult to remember. The RewriteRule and RewriteEngine lines in your Apache configuration file (/etc/httpd/conf.d/git.conf) takes care of that and produce nice and clean urls.

So you can browser the repository via http://git.cachevideos.com/videocache.git instead of http://git.cachevideos.com/?p=videocache.git;a=summary.

2. Enabling remote ls (git-ls-remote or git ls-remote)

This is the most trickiest part. If you try the command below, it won’t produce any output

[root@localhost ~]# git-ls-remote http://git.cachevideos.com/videocache.git

You need to go to project’s repository in gitweb and then run the following command to update the server info for git.

[root@localhost ~]# cd /home/saini/domains/cachevideos.com/git/videocache.git/
[root@localhost ~]# git-update-server-info

Try the ls-remote command now and it should succeed by producing all the branches and tags in the remote repository.

But there is a problem, you have to run the above command after every commit to the remote repository. To solve this issue, you can enable post-update hook for the project’s repository in gitweb. Use the following command to enable it.

[root@localhost ~]# cd /home/saini/domains/cachevideos.com/git/videocache.git/
[root@localhost ~]# chmod +x post-update

The above command will update the server info automatically every time you commit.

Thats all you need to do for setting up gitweb. I hope this will be helpful.

 

How To: Boot Fedora Faster

Note: These tricks apply to any Linux based OS. But I have tested them only on Fedora, so can’t say whether they’ll work on other Linux(s).

My current Fedora installation is now almost one and a half years old. Yes. I am still using Fedora 7 😀 I have Fedora 10 on my other machine. Coming to the agenda, my Fedora installation has grown beyond control and I have services from named, squid, drbl, privoxy, vsftpd, vbox*, smb and what not on a personal desktop. These services really force my system startup to slow down to more than two minutes. While shutting down, its very easy to just cut the power supply but while booting up I can’t help and it frustrates me. And what frustrates me further that I have 4GB DDR2 RAM and AMD64 X2 5600+ (2.8GHz x 2) and booting time is still more than two minutes.

Agenda

  • Boot Fedora faster using whatever techniques possible.

Remove the services from normal order and delay their execution to a later stage. So, services like network, squid, privoxy, named, vsftpd, smb etc. doesn’t make sense unless I am not logged in and using them. Let us start them after we have login screen.

Turn off all the services by using the command

[root@bordeaux ~]# chkconfig service_name off

where service_name is the service you want to turn off.

Now create a file /etc/startup.sh. Enter a line like this

[root@bordeaux ~]# service service_name start

for every service that you have turned off in the Step 1.1 and you want it to be running after your machine starts up. Now, your startup.sh file should look like this

service network start &
service sshd start &
modprobe it87 &
modprobe k8temp &
/usr/bin/iptraf -s eth0 -B &
/usr/bin/iptraf -s lo -B &
service squid start &
service privoxy start &
service httpd start &
service mysqld start &
service named start &
service smb start &
service vboxdrv start &
service vboxnet start &
service vsftpd start &

Add the following line to /etc/rc.local file

/bin/bash /etc/startup.sh &

Done!!! Notice the &s in both files. They are for execution in background so that a process can block boot process. You’ll observe a drop of 10-20 seconds in system startup time.

Problem with Hack #1 : The execution is not really parallel. It executes like a process in the background. So we can’t get the real advantage of parallel execution.

Hack #2 solves this problem. Now we don’t put processes in background. We use daemon forking to fork a separate daemon process which will start all the services for us in parallel. Here we’ll get the real advantage and startup time will decrease further.

This step is totally similar to Step 1.1. So skipping it.

This step is also similar to Step 1.2. The /etc/startup.sh file should look like this.

service network start
service xinetd start
service crond start
service anacron start
service atd start
service sshd start
service rpcbind start
service rpcgssd start
service rpcimapd start
modprobe it87
modprobe k8temp
/usr/bin/iptraf -s eth0 -B
/usr/bin/iptraf -s lo -B
service nasd start
service squid start
service privoxy start
service httpd start
service iptables start
service lm_sensors start
service mysqld start
service named start
service nfs start
service nfslock start
service smb start
service vboxdrv start
service vboxnet start
service vsftpd start
service autofs start
service smartd start

Notice the absence of &s in the file.

Download the attached startup.py file attached at the end of this post or copy paste the following code to /etc/startup.py file.

#!/usr/bin/env python
# (C) Copyright 2008 Kulbir Saini
# License : GPL
import os
import sys
def fork_daemon(f):
    """This function forks a daemon."""
    # Perform double fork
    r = ''
    if os.fork(): # Parent
        # Wait for the child so that it doesn't defunct
        os.wait()
        # Return a function
        return  lambda *x, **kw: r
    # Otherwise, we are the child
    # Perform second fork
    os.setsid()
    os.umask(077)
    os.chdir('/')
    if os.fork():
        os._exit(0)
    def wrapper(*args, **kwargs):
        """Wrapper function to be returned from generator.
        Executes the function bound to the generator and then
        exits the process"""
        f(*args, **kwargs)
        os._exit(0)
    return wrapper
 
def start_services(startup_file):
    command = '/bin/bash ' + startup_file + ' > /dev/null 2> /dev/null '
    os.system(command)
    return
 
if __name__ == '__main__':
    forkd = fork_daemon(start_services)
    forkd(sys.argv[1])
    print 'Executing ', sys.argv[1], '[  OK  ]'

Add the following line to your /etc/rc.local file.

/usr/bin/python /etc/startup.py /etc/startup.sh

Thats it. Done!!! Now you’ll experience a boost of about 25-30 seconds of decrease in boot time.

Stats of my machine

With all services started in normal order : 2minutes.
With Hack #1 : 1minute 42 seconds.
With Hack #2 : 1minute.

Warning : These hacks may break your system and can make it unusable. Use at your own risk.

 

How To: Configure Caching Nameserver (named)

Mission

To configure a caching nameserver on a local machine which will cascade to another previously configured and functional nameserver (may or may not be caching. It’ll generally be your ISP nameserver or the one provided by your organization).

Advantage

  • Reduces the delay in domain name resolution drastically as the requests for frequently accessed websites are served from cache.

Working

  • named gets a request for domain resolution.
  • It checks whether the request can be satisfied from cache. If the answer is in cache and not stale, the request is satisfied from cache itself saving a lot of time :)
  • If request can’t be satisfied from cache, named queries the first parent. If it replies with the answer, then named will cache the response and subsequent requests for the same domain name will be satisfied from the cache.
  • In case first parent fails to reply, named will query the second parent and so on.

(The working is my understanding of caching-nameserver using wireshark as traffic analysis tool and caching-nameserver may not behave exactly as explained above.)

How to install

named is by default on most of the systems by the package name ‘caching-nameserver‘. If its not present on your system, install using

[root@localhost ~]# yum install caching-nameserver [ENTER]
# If that doesn't work try this
[root@localhost ~]# yum install bind [ENTER]

How to configure

The main configuration file for named resides in /var/named/chroot/etc/named.caching-nameserver.conf which is also soft linked from /etc/named.caching-nameserver.conf . named configuration file supports C/C++ style comments.

For a caching nameserver which will cascade to another nameserver, there is nothing much to be configured. You need to configure “options” block. Below is a configuration file for a machine with IP address 172.17.8.64 cascading to two nameserver 192.168.36.204 and 192.168.36.210. The comments inline explain what each option does.

options {
  // Set the port to 53 which is standard port for DNS.
  // Add the IP address on which named will listen separated by semi-colons.
  // It'll be your own IP address.
  listen-on port 53 {127.0.0.1; 172.17.8.64;};
  // These are default. Leave them as it is.
  directory   "/var/named";
  dump-file   "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  // The machines which are allowed to query this nameserver.
  // Normally you'll allow only your machine. But you can allow other machines also.
  // The address should be separated by semi-colons. To allow a network 172.16.31.0/24,
  // the line would be
  // allow-query {localhost; 172.16.31.0/24; };
  // Don't forget the semi-colons.
  allow-query     { localhost; 172.17.8.64; };
  recursion yes;
  // The parent nameservers. List all the nameserver which you can query.
  forwarders { 192.168.36.204; 192.168.36.210; };
  forward first;
};
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
zone "." IN {
  type hint;
  file "named.ca";
};
include "/etc/named.rfc1912.zones";

Start caching-nameserver

Now start the caching-nameserver using the following command

[root@localhost ~]# server named start [ENTER]

OR

[root@localhost ~]# /etc/init.d/named start [ENTER]

To make named start every time your reboot your machine use following command

[root@localhost ~]# chkconfig named on [ENTER]

Using caching-nameserver

To use your caching-nameserver, open /etc/resolv.conf file and add the following line

nameserver 127.0.0.1

Comment all other lines in the file, so that finally the file looks like

; generated by /sbin/dhclient-script
#search wlan.iiit.ac.in
#nameserver 192.168.36.204
#nameserver 192.168.36.210
nameserver 127.0.0.1

Now your system will use your own nameserver (in caching mode) for resolving all domain names. To test if your nameserver use the following command

[root@localhost ~]# dig fedora.co.in [ENTER]

Now if you use that command for the second time, the resolution time will be around 2-3 milli seconds while first time it would be around 400-700 milli seconds.

Example

Below is two subsequent runs of dig for fedora.co.in . Notice the Query time.

[root@bordeaux SPECS]# dig fedora.co.in
; <<>> DiG 9.4.2rc1 <<>> fedora.co.in
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7839
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;fedora.co.in.                  IN      A
;; ANSWER SECTION:
fedora.co.in.           83629   IN      A       72.249.126.241
;; AUTHORITY SECTION:
fedora.co.in.           79709   IN      NS      ns.fedora.co.in.
;; ADDITIONAL SECTION:
ns.fedora.co.in.        79709   IN      A       72.249.126.241
;; Query time: 531 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 19 18:04:47 2008
;; MSG SIZE  rcvd: 79
[root@bordeaux SPECS]# dig fedora.co.in
; <<>> DiG 9.4.2rc1 <<>> fedora.co.in
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64233
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;fedora.co.in.                  IN      A
;; ANSWER SECTION:
fedora.co.in.           83625   IN      A       72.249.126.241
;; AUTHORITY SECTION:
fedora.co.in.           79705   IN      NS      ns.fedora.co.in.
;; ADDITIONAL SECTION:
ns.fedora.co.in.        79705   IN      A       72.249.126.241
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 19 18:04:51 2008
;; MSG SIZE  rcvd: 79
[root@bordeaux SPECS]#
 

How To: Install Fedora without CD or DVD

Note: If you are new to Fedora/Linux, I highly recommend the book “Fedora Linux Toolbox

[amazon-product alink=”0000FF” bordercolor=”000000″ height=”240″]0470082917[/amazon-product]

Use Case

  1. When you don’t have CD / DVD drive on your system.
  2. You have Fedora DVD but your system has only a CD Drive.
  3. You don’t want to waste time and resources in burning iso on optical media.

Pre-requisites

  1. You have a Fedora DVD iso or rescue cd iso.
  2. You have a Linux installation on your system.
  3. You have a partition (FAT32, ext2, ext3) which you will not format while installing the new OS.

How to proceed

Let us assume you want to install Fedora 9 on your system and you have a Linux distro already installed on your system. You have downloaded the Fedora DVD iso (Fedora-9-DVD-i686.iso). And you have a FAT32/ext2/ext3 partition /stuff/ which you will not format during installation.

Step 1 : Move the Fedora DVD iso to /stuff/ directory.

[root@saini saini]# mv Fedora-9-DVD-i686.iso /stuff/ [Enter]

Step 2 : Mount Fedora DVD iso on /mnt/

[root@saini saini]# mount /stuff/Fedora-9-DVD-i686.iso /mnt/ -ro loop [Enter] (do as root)

Step 3 : Copy the initrd.img and vmlinuz to /boot/ partition

[root@saini saini]# cd /mnt/isolinux/ [Enter]
[root@saini isolinux]# cp initrd.img vmlinuz /boot/ [Enter] (do as root)

Step 4 : Create grub entry for booting into Fedora 9

Add these lines at the end of your /boot/grub/grub.conf file.

title Fedora 9 (New installation)
    kernel /vmlinuz
    initrd /initrd.img

Step 5 : Note the device having Fedora DVD iso

[root@saini saini]# df -h [Enter]
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              15G  9.5G  4.1G  70% /
/dev/sda8             135G  116G   13G  91% /stuff
/dev/sda5             4.8G  1.2G  3.4G  26% /home
/dev/sda1              99M   12M   82M  13% /boot

In this case /dev/sda8 contains Fedora DVD iso. Note this down as you need it later.

Step 6 : Reboot

Reboot your system and boot into the Fedora 9 (New installation) grub entry.

Step 7 : Install from hard disk

While in installation wizard, select “Hard drive” as installation method and choose /dev/sda8 as it contains the Fedora DVD iso. And rest is damn easy.

 

How To: Test Fedora Pre-release

If you want your Fedora to be best ever, just grab a copy of Fedora DVD or live iso from here and test the features you like. Checkout the Fedora 10 feature list and test as many as possible.

If you don’t have access to optical media (cd/dvd rom) and want to try the live iso, VirtualBox is your friend. Grab the latest version of VirtualBox for your distro and platfrom from Sun download center and install using rpm or yum. Using VirtualBox is quite easy. Here is a video tutorial for noobs on how to test pre-release live ISOs using VirualBox. Have fun :)

 

How To: Configure Squid Proxy Server

Mission

To configure squid for simple proxying without caching anything.

Use Cases

  1. When you want to have control on what people browse on your lan.
  2. When number of machine is more than the number of IP addresses you can afford to buy.
  3. When you want to help this holy world in saving some IPV4 addresses 😛

Assumptions

  1. You have a machine connected directly to internet that you are going to use as a proxy server for other machines on your network.
  2. The machines on your network are using 192.168.0.0/16 as private address space. You can use anyone/multiple address spaces of the available but for this howto we assume 192.168.0.0/16 as the local network.
  3. The local IP address of the machine which will run squid proxy server is 192.168.36.204. You can have any IP, but for this howto we assume this.

How to proceed

First of all ensure that you have squid installed. After installing squid, you need to set access control in squid configuration file which resides in /etc/squid by default. Open /etc/squid/squid.conf and add/edit following lines according to your preferences. Few lines already exist in the configuration file, you can add the rest.

# The port on which squid will listen for requests
http_port 8080
# If 'cgi-bin' or '?' is in query, squid should not check with neighbours'/parents' cache
# and should go to target web-server.
hierarchy_stoplist cgi-bin ?
# If url contains 'cgi-bin' or '?', then it must not be cached
acl QUERY urlpath_regex cgi-bin \?
cache deny QUERY
acl apache rep_header Server ^Apache
broken_vary_encoding allow apache
# Absolute path to squid access log.
access_log /var/log/squid/access.log squid
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern .               0       20%     4320
# Access control list to control every IP address
acl all src 0.0.0.0/0.0.0.0
# Access control list for source machine in LAN
acl lan_src src 192.168.0.0/16
# Access control list for destination machine in LAN
acl lan_dst dst 192.168.0.0/16
# Access control list to manage squid cache
acl manager proto cache_object
# Access control list to define IP address allowed for source localhost
acl localhost src 127.0.0.1/255.255.255.255
# Access control list to define IP addresses allowed for localhost as destination
acl to_localhost dst 127.0.0.0/8
# Access control list to define Safe ports that should be allowed by default
acl SSL_ports port 443 563 1863 5190 5222 5050 6667
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
# Allow cache management only from localhost
http_access allow manager localhost
# Deny cache management from remote hosts
http_access deny manager
# Deny http access via all the ports which are not listed as safe
http_access deny !Safe_ports
# Deny all connections via all ports which are not listed as safe
http_access deny CONNECT !SSL_ports
# Allow http access from localhost
http_access allow localhost
# Allow http access from machines on LAN
http_access allow lan_src
http_access deny all
http_reply_access allow all
icp_access allow all
# Deny caching for everyone so that there is not caching at all
cache deny all
coredump_dir /var/spool/squid
# Never allow direct connection to machines on the internet
prefer_direct off
never_direct allow all
# Allow direct connetion if the destination machine is on LAN
always_direct allow lan_dst
# Delete this line if you don't have /etc/hosts file
hosts_file /etc/hosts
# Allow AIM connections
# Delete the following 9 lines if you don't want people to connect to AIM
acl AIM_ports port 5190 9898 6667
acl AIM_domains dstdomain .oscar.aol.com .blue.aol.com .freenode.net
acl AIM_domains dstdomain .messaging.aol.com .aim.com
acl AIM_hosts dstdomain login.oscar.aol.com login.glogin.messaging.aol.com toc.oscar.aol.com irc.freenode.net
acl AIM_nets dst 64.12.0.0/255.255.0.0
acl AIM_methods method CONNECT
http_access allow AIM_methods AIM_ports AIM_nets
http_access allow AIM_methods AIM_ports AIM_hosts
http_access allow AIM_methods AIM_ports AIM_domains
# Allow connections to Yahoo Messenger
# Delete the following 6 lines if you don't want people to connect to Yahoo Messenger
acl YIM_ports port 5050
acl YIM_domains dstdomain .yahoo.com .yahoo.co.jp
acl YIM_hosts dstdomain scs.msg.yahoo.com cs.yahoo.co.jp
acl YIM_methods method CONNECT
http_access allow YIM_methods YIM_ports YIM_hosts
http_access allow YIM_methods YIM_ports YIM_domains
# Allow connections to Google Talk
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl GTALK_ports port 5222 5050
acl GTALK_domains dstdomain .google.com
acl GTALK_hosts dstdomain talk.google.com
acl GTALK_methods method CONNECT
http_access allow GTALK_methods GTALK_ports GTALK_hosts
http_access allow GTALK_methods GTALK_ports GTALK_domains
# Allow connections to MSN
# Delete the following 6 lines if you don't want people to connect to Google Talk
acl MSN_ports port 1863 443 1503
acl MSN_domains dstdomain .microsoft.com .hotmail.com .live.com .msft.net .msn.com .passport.com
acl MSN_hosts dstdomain messenger.hotmail.com
acl MSN_nets dst 207.46.111.0/255.255.255.0
acl MSN_methods method CONNECT
http_access allow MSN_methods MSN_ports MSN_hosts

Now, start the squid proxy server as

service squid start

Also, if you want squid to be started every time you boot the machine, execute the following command

chkconfig --level 345 squid on

You have a squid proxy server running now. You can ask clients to configure there browsers to use 192.168.36.204 as a proxy server with 8080 as proxy port. Command line utilities like elinks, lynx, yum, wget etc. can be asked to use proxy by exporting http_proxy variable as below. Users can also add these lines to ~/.bashrc file to avoid exporting every-time.

export http_proxy='http://192.168.36.204:8080'
export ftp_proxy='http://192.168.36.204:8080'

I highly recommend the book “Squid Proxy Server 3.1: Beginner’s Guide (Paperback)” for further reading.

 

How To: Write Custom Redirector or Rewritor Plugin For Squid in Python

Mission

To write a custom Python program which can act as a plugin for Squid to redirect a given URL to another URL. This is useful when already existing redirector plugins for Squid doesn’t suit your needs or you want everything of your own.

Use Cases

  1. When you want to redirect URLs using a database like mysql or postgresql.
  2. When you want to redirect based on mappings stored in simple text files.
  3. When you want to build a redirector which can learn by itself using AI techniques 😛

How to proceed

From Squid FAQ,

The redirector program must read URLs (one per line) on standard input, and write rewritten URLs or blank lines on standard output. Note that the redirector program can not use buffered I/O. Squid writes additional information after the URL which a redirector can use to make a decision.

The format of the line read from the standard input by the program is as follows.

1
2
3
URL ip-address/fqdn ident method
# for example
http://saini.co.in 172.17.8.175/saini.co.in - GET -

The implementation sounds very simple and it is indeed very simple to implement. The only thing that should be taken care of is the unbuffered I/O. You should immediately flush the output to standard output once decision is taken.

For this howto, we assume we have a method called ‘modify_url()‘ which returns either a blank line or a modified URL to which the client should be redirected.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/usr/bin/env python
 
import sys
def modify_url(line):
    list = line.split(' ')
    # first element of the list is the URL
    old_url = list[0]
    new_url = '\n'
    # take the decision and modify the url if needed
    # do remember that the new_url should contain a '\n' at the end.
    if old_url.endswith('.avi'):
        new_url = 'http://fedora.co.in/errors/accessDenied.html' + new_url
    return new_url
 
while True:
    # the format of the line read from stdin is
    # URL ip-address/fqdn ident method
    # for example
    # http://saini.co.in 172.17.8.175/saini.co.in - GET -
    line = sys.stdin.readline().strip()
    # new_url is a simple URL only
    # for example
    # http://fedora.co.in
    new_url = modify_url(line)
    sys.stdout.write(new_url)
    sys.stdout.flush()

Save the above file somewhere. We save this example file in /etc/squid/custom_redirect.py. Now, we have the function for redirecting clients. We need to configure squid to use custom_redirect.py . Below is the squid configuration for telling squid to use the above program as redirector.

1
2
3
4
5
6
# Add these lines to /etc/squid/squid.conf file.
# /usr/bin/python should be replaced by the path to python executable if you installed it somewhere else.
redirect_program /usr/bin/python /etc/squid/custom_redirect.py
# Number of instances of the above program that should run concurrently.
# 5 is good enough but you should go for 10 at least. Anything below 5 would not work properly.
redirect_children 5

Now, start/reload/restart squid. That’s all we need to write and use a custom redirector plugin for squid.