How To: Install/Configure GNUMP3d – Streaming Audio Server

Mission

GNUMP3d is the GNU Streaming MP3/Media Server written in perl. Our mission is to setup GNUMP3d and stream audio over LAN or over internet. Below are the essential steps to install and configure GNUMP3d.

Download

Download latest version of GNUMP3d from GNUMP3d Website.

Extract

[kulbirsaini@fedora ~]$ tar -xjf gnump3d-x.x.tar.bz2

Install

[root@fedora ~]$ cd gnump3d-3.0
[root@fedora ~]$ make install (as root)

Now gnump3d is installed on your system. Now you need to configure it according to your taste.

Configure

The configuration file is located at /etc/gnump3d/gnump3d.conf. For casual use, you just need to configure port, binding_host and root.

# Port to which gnump3d will be accessible via web interface or via a media player like xmms or winamp.
port = 1111
# The IP Address where gnump3d will bind itself.
binding_host = 172.17.8.64
# If you want the stream to be accessible via a fully qualified domain name, set hostname variable.
# You don't need to set this in most cases e.g. while setting up gnump3d on LAN.
hostname = gofedora.com
# The directory where are your music files resides.
root = /stuff/Music/

Though you can skip rest of the configuration, you may try to explore other options. My gnump3d.conf file can be download from here.

Thats all you need to do to configure gnump3d.

Indexing

Now you need to index all you music collection (the audio files in gnump3d root). Run the following command to index

[root@fedora ~]$ gnump3d-index --verbose

Run gnump3d

Once the indexing is done, you are all set to run gnump3d. By default gnump3d tries to index all files whenever you start it, to avoid this we need to use –fast option.

[root@fedora ~]$ gnump3d --fast

By default gnump3d runs in foreground. If you want it to go in background and run quietly, run it as follows.

[root@fedora ~]$ gnump3d --fast --background

Accessing Media Server

To access your gnump3d streaming media server, please visit url http://ip_address:port/ .

Run at startup

If you want gnump3d to start when your computer starts add the following line to /etc/rc.local file.

gnump3d --fast --background

Feel free to comment in case you have a problem.

 

How To: Install and Configure GitWeb

UPDATE : I recommend using GitList instead of GitWeb. GitList is much easier to setup and has a better web interface. Continue reading this post if you looking for GitWeb setup instructions specifically.

Goal

Setting up gitweb (web interface for SCM software git) for your project’s git repository for public access and developer commits via ssh.

Assumptions

  1. You already have your project’s git repository.
  2. You have hosting space somewhere to host gitweb.
  3. You have root access.
  4. You are using Apache as webserver.

Example for this howto

Project : VideoCache
Domain for gitweb : git.cachevideos.com
URL for git access for videocache : http://git.cachevideos.com/videocache.git
Actual path on server : /home/saini/domains/cachevideos.com/git
Git repository : /home/saini/projects/videocache/

Installation

Installation is pretty easy. Just one single command would do everything.

[root@localhost ~]# yum install gitweb (do as root)

This will create a directory /var/www/git which is default for gitweb.

Copy the directory /var/www/git/ to /home/saini/domains/cachevideos.com/git

[root@localhost ~]# cp -r /var/www/git /home/saini/domains/cachevideos.com/git

Configuration

1. GitWeb

Open the file /etc/gitweb.conf (it may or may not be there) and add the following lines to it.

# Change This
$projectroot = '/home/saini/domains/cachevideos.com/git';
# Change This
$site_name = "Kulbir Saini's git trees.";
# Don't Change the variables below
$my_uri = "/";
$home_link = '/';
@stylesheets = ("/gitweb.css");
$favicon = "/git-favicon.png";
$logo = "/git-logo.png";

2. Apache

Open the file /etc/httpd/conf.d/git.conf and clear all the lines that are already there and add the following lines to it

  DocumentRoot /home/saini/domains/cachevideos.com/git
  ServerName git.cachevideos.com
  ErrorLog "/home/saini/domains/cachevideos.com/logs/error_log"
  CustomLog "/home/saini/domains/cachevideos.com/logs/access_log" combined
  SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
  DirectoryIndex gitweb.cgi
 
    Allow from all
    AllowOverride all
    Order allow,deny
    Options +ExecCGI
    AddHandler cgi-script .cgi
 
      SetHandler cgi-script
 
    RewriteEngine on
    RewriteRule ^[a-zA-Z0-9_\-]+\.git/?(\?.*)?$ /gitweb.cgi%{REQUEST_URI} [L,PT]

3. Git repository configuration

Go to your git repository (/home/saini/projects/videocache/) and make the following changes.

(a). Open file .git/description and add a short nice description for your project.

videocache is a squid url rewriter plugin written in Python to facilitate youtube, metacafe, dailymotion, google, vimeo, msn soapbox, tvuol.uol.com.br, blip.tv, break.com videos and wrzuta.pl audio caching.

(b). Open file .git/config and append the following lines

[gitweb]
  owner = "Kulbir Saini"

Copy project’s git repository for gitweb

Copy the /home/saini/projects/videocache/.git directory to /home/saini/domains/cachevideos.com/git/videocache.git

[root@localhost ~]# cp -r /home/saini/projects/videocache/.git /home/saini/domains/cachevideos.com/git/videocache.git

Finishing Step

Restart Apache webserver.

[root@localhost ~]# service httpd restart

Now you can browser a list of your projects’ git repositories at http://git.cachevideos.com/ .

Adding another project repository

Just copy the project repository’s .git directory to /home/saini/domains/cachevideos.com/git/prjoect_name.git. And it’ll be shown on the list.

Committing (pushing) to the repository

For committing to the repository via ssh use the following command.

# Pushing everything (Please see the username)
[root@localhost videocache]# git push --all ssh://saini@git.cachevideos.com/~saini/domains/cachevideos.com/git/videocache.git

To update tags on the remote repository use this command.

# Pushing all tags
[root@localhost videocache]# git push --tags ssh://saini@git.cachevideos.com/~saini/domains/cachevideos.com/git/videocache.git

Well, if you consider just the web interface and committing part for your project, thats all. But things can be fine tuned further. Below are few hacks!

1. Enabling nice urls.

By default the urls for browsing repository via git web are pretty crappy and difficult to remember. The RewriteRule and RewriteEngine lines in your Apache configuration file (/etc/httpd/conf.d/git.conf) takes care of that and produce nice and clean urls.

So you can browser the repository via http://git.cachevideos.com/videocache.git instead of http://git.cachevideos.com/?p=videocache.git;a=summary.

2. Enabling remote ls (git-ls-remote or git ls-remote)

This is the most trickiest part. If you try the command below, it won’t produce any output

[root@localhost ~]# git-ls-remote http://git.cachevideos.com/videocache.git

You need to go to project’s repository in gitweb and then run the following command to update the server info for git.

[root@localhost ~]# cd /home/saini/domains/cachevideos.com/git/videocache.git/
[root@localhost ~]# git-update-server-info

Try the ls-remote command now and it should succeed by producing all the branches and tags in the remote repository.

But there is a problem, you have to run the above command after every commit to the remote repository. To solve this issue, you can enable post-update hook for the project’s repository in gitweb. Use the following command to enable it.

[root@localhost ~]# cd /home/saini/domains/cachevideos.com/git/videocache.git/
[root@localhost ~]# chmod +x post-update

The above command will update the server info automatically every time you commit.

Thats all you need to do for setting up gitweb. I hope this will be helpful.

 

IntelligentMirror: RPM and DEB Caching Improved (0.5)

After spending a lot of time with youtube cache, now I am trying to devote some time to update intelligentmirror with required features and enhancements that youtube cache already enjoys. In the same direction here is version 0.5 of intelligentmirror.

Improvements

  • Added max_parallel_downloads options to controll the maximum threading fetching from upstream to cache the packages.
  • Fine grained control on logging via max_logfile_size and max_logfile_backups option.
  • Added setup script to help you install intelligentmirror. No need to execute commands one by one for installation. Just run
 [root@localhost]# python setup.py install [ENTER]
  • Added update script (update-im). So in case you decide to change the locations for caching rpm/deb packages, just run
 [root@localhost]# update-im [ENTER]

OR

 [root@localhost]# /usr/sbin/update-im [ENTER]
  • Download scheduler similar to youtube cache is added to facilitate the download queing in case of large number of requests.
  • More informative logging.
  • cache.log is not flooding anymore with XMLRPC logs and python tracebacks.
  • Added extensive exception handling thoughout the program.

Availability

  1. RPMs for Fedora/Red Hat/Cent OS
  2. Source RPMs for Fedora/Red Hat/Cent OS
  3. Source Tar balls

Installation and Configuration

INSTALL and README files should help you throughout the installation and configuration process.

In case you have questions, ask them here in comments. Suggestions for improvement are welcome 🙂

 

How To: Configure Caching Nameserver (named)

Mission

To configure a caching nameserver on a local machine which will cascade to another previously configured and functional nameserver (may or may not be caching. It’ll generally be your ISP nameserver or the one provided by your organization).

Advantage

  • Reduces the delay in domain name resolution drastically as the requests for frequently accessed websites are served from cache.

Working

  • named gets a request for domain resolution.
  • It checks whether the request can be satisfied from cache. If the answer is in cache and not stale, the request is satisfied from cache itself saving a lot of time 🙂
  • If request can’t be satisfied from cache, named queries the first parent. If it replies with the answer, then named will cache the response and subsequent requests for the same domain name will be satisfied from the cache.
  • In case first parent fails to reply, named will query the second parent and so on.

(The working is my understanding of caching-nameserver using wireshark as traffic analysis tool and caching-nameserver may not behave exactly as explained above.)

How to install

named is by default on most of the systems by the package name ‘caching-nameserver‘. If its not present on your system, install using

[root@localhost ~]# yum install caching-nameserver [ENTER]
# If that doesn't work try this
[root@localhost ~]# yum install bind [ENTER]

How to configure

The main configuration file for named resides in /var/named/chroot/etc/named.caching-nameserver.conf which is also soft linked from /etc/named.caching-nameserver.conf . named configuration file supports C/C++ style comments.

For a caching nameserver which will cascade to another nameserver, there is nothing much to be configured. You need to configure “options” block. Below is a configuration file for a machine with IP address 172.17.8.64 cascading to two nameserver 192.168.36.204 and 192.168.36.210. The comments inline explain what each option does.

options {
  // Set the port to 53 which is standard port for DNS.
  // Add the IP address on which named will listen separated by semi-colons.
  // It'll be your own IP address.
  listen-on port 53 {127.0.0.1; 172.17.8.64;};
  // These are default. Leave them as it is.
  directory   "/var/named";
  dump-file   "/var/named/data/cache_dump.db";
  statistics-file "/var/named/data/named_stats.txt";
  memstatistics-file "/var/named/data/named_mem_stats.txt";
  // The machines which are allowed to query this nameserver.
  // Normally you'll allow only your machine. But you can allow other machines also.
  // The address should be separated by semi-colons. To allow a network 172.16.31.0/24,
  // the line would be
  // allow-query {localhost; 172.16.31.0/24; };
  // Don't forget the semi-colons.
  allow-query     { localhost; 172.17.8.64; };
  recursion yes;
  // The parent nameservers. List all the nameserver which you can query.
  forwarders { 192.168.36.204; 192.168.36.210; };
  forward first;
};
logging {
        channel default_debug {
                file "data/named.run";
                severity dynamic;
        };
};
zone "." IN {
  type hint;
  file "named.ca";
};
include "/etc/named.rfc1912.zones";

Start caching-nameserver

Now start the caching-nameserver using the following command

[root@localhost ~]# server named start [ENTER]

OR

[root@localhost ~]# /etc/init.d/named start [ENTER]

To make named start every time your reboot your machine use following command

[root@localhost ~]# chkconfig named on [ENTER]

Using caching-nameserver

To use your caching-nameserver, open /etc/resolv.conf file and add the following line

nameserver 127.0.0.1

Comment all other lines in the file, so that finally the file looks like

; generated by /sbin/dhclient-script
#search wlan.iiit.ac.in
#nameserver 192.168.36.204
#nameserver 192.168.36.210
nameserver 127.0.0.1

Now your system will use your own nameserver (in caching mode) for resolving all domain names. To test if your nameserver use the following command

[root@localhost ~]# dig fedora.co.in [ENTER]

Now if you use that command for the second time, the resolution time will be around 2-3 milli seconds while first time it would be around 400-700 milli seconds.

Example

Below is two subsequent runs of dig for fedora.co.in . Notice the Query time.

[root@bordeaux SPECS]# dig fedora.co.in
; <<>> DiG 9.4.2rc1 <<>> fedora.co.in
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7839
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;fedora.co.in.                  IN      A
;; ANSWER SECTION:
fedora.co.in.           83629   IN      A       72.249.126.241
;; AUTHORITY SECTION:
fedora.co.in.           79709   IN      NS      ns.fedora.co.in.
;; ADDITIONAL SECTION:
ns.fedora.co.in.        79709   IN      A       72.249.126.241
;; Query time: 531 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 19 18:04:47 2008
;; MSG SIZE  rcvd: 79
[root@bordeaux SPECS]# dig fedora.co.in
; <<>> DiG 9.4.2rc1 <<>> fedora.co.in
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64233
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1
;; QUESTION SECTION:
;fedora.co.in.                  IN      A
;; ANSWER SECTION:
fedora.co.in.           83625   IN      A       72.249.126.241
;; AUTHORITY SECTION:
fedora.co.in.           79705   IN      NS      ns.fedora.co.in.
;; ADDITIONAL SECTION:
ns.fedora.co.in.        79705   IN      A       72.249.126.241
;; Query time: 1 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Nov 19 18:04:51 2008
;; MSG SIZE  rcvd: 79
[root@bordeaux SPECS]#
 

IntelligentMirror Gets Even More Intelligent (1.0.1)

Warning : This version of IntelligentMirror is compatible with only squid-2.7 as of now. It is NOT compatible even with squid-3.0.

IntelligentMirror Version 1.0.1

I have been following squid development regularly (at least the part in which I am interested) and they have introduced a new directive in squid-2.7 known as StoreUrlRewrite (storeurl_rewrite_program). Using this directive you can instruct squid to cache url A (http://abc.com/foo/bar/version/crap.rpm) as url B (http://proxy.fedora.co.in/intelligentmirror/crap.rpm). In simple words you can direct squid to cache any url as any other url without any extra efforts.

So keeping the above directive in mind, I have worked out a different version of intelligentmirror especially for squid-2.7.

IntelligentMirror : Old method of operation

  1. IntelligentMirror gets a client request for a URL.
  2. Check: if URL is not in (RPM, metadata file)
    • Then its none of our business.
    • Let proxy handle it the normal way.
    • Done and exit.
  3. Check: if RPM/metadata is available in cache
    • Stream the RPM/metadata from cache.
    • Done and exit.
  4. Check: if RPM/metadata is not available in cache
    • Download in parallel for caching in some dir and stream.
    • Done and exit.

IntelligentMirror : New method of operation

  1. IntelligentMirror gets a client request for a URL.
  2. Check: if request for rpm
    1. Direct squid to cache the request as http://<same_host_all_the_time>/intelligentmirror/<rpmname>.rpm
  3. Check: if request for deb
    1. Direct squid to cache the request as http://<same_host_all_the_time>/intelligentmirror/<debname>.deb
  4. Done and exit.

So your squid will see every request for an rpm package as a request http://<same_host_all_the_time>/intelligentmirror/<rpmname>.rpm. So, if you happen to request the same rpm from a different mirror, it’ll still be served from cache 🙂

Improvements

  1. No need to check if the url supplied by squid is for rpm or not because storeurl_rewrite_program has an acl controller attached which will invoke intelligentmirror for urls ending in .rpm .
  2. No need to check if the url is already cached or not. No need to worry about the directory where you are going to store the packages. No human intervention is needed in maintaining the cache. Almighty squid is doing everything for us.
  3. No need to worry if the target package has changed because of the resigning or whatever because squid will do that for you.
  4. No need to actually download the package in parallel for caching because squid is already doing that.
  5. No need to worry about the hashing algorithms and storage optimizations for the cached content.

Availability

  1. RPM for Fedora/Red Hat
  2. Source RPM for Fedora/Red Hat
  3. Source Tarball

Install and Configure

The install and configure files should be enough to guide you through the installation if you choose the tar ball way. Otherwise you can always install from rpm from the above link.

Note1: You have to configure your squid to use intelligentmirror as a plugin even if you install via rpm. Check the configure file at the above link.

Note2: StoreUrlRewrite will probably be available in squid-3.1.