Useful Web Applications to run yourself

There are several useful web applications out there. A lot of free sites like blogger, tumblr, etc. allow you to do blogs or other websites for free. But you don't always have the control you'd like. And you can't always make them private. If you're going to have a server up, might as well make it a web server. There are tons of walkthroughs on how to setup a LAMP server on various distributions out there. (LAMP stands for Linux, Apache, MySQL, PHP/Perl/Python. Which basically just means a web server using Apache that has server-side programing capabilities and a database.)

Once you have a LAMP server setup and you're looking to either do database work or install web applications that utilize databases, you may find it nice to have phpMyAdmin. phpMyAdmin is a web gui for managing your mysql databases. as it's name implies it runs on php and it utilizes mysql also for it's self.

You may also find Webmin a useful tool. Many distros have Webmin in the repos so you can easily install it. Webmin provides a gui/web interface to administer many aspects of the server from mysql/postresql database management plugins, to managing users and package updates. There's a lot to it. Just remember to make sure that you firewall off that port from anything but the local network. You don't your router forwarding to that port, major security issue.

Webalizer is an old tool that's been around for a long time. Webalizer is a set of scripts that takes your Apache log files and scans through them every night (via a cron job you setup) and then makes a database of those data points and provides some graphs around usage. It's locally hosted and can be a nice tool to have. It's not going to be a robust as Google Analytics though (but then you don't have to setup Google Analytics code on every page of your sites for tracking either).

For publishing there's a bunch of options, WordPress is probably the biggest blogging tool out there. It's same purpose behind blogger. The difference is you can host WordPress yourself. And it's way more customizable. If you want more of a Content Management System, go with Joomla. It's a fork of the old Mambo CMS and very powerful. But with that power comes a lot of configuration. If you want to go a bit simpler with a CMS try Drupal. If you like a wiki type environment, I like MediaWiki.

Sometimes you may want to go more of an educational route. Moodle is probably the best bet here. It's easy to build and maintain. Sakai is another open source LMS tool though it will require going to Tomcat and will me more difficult to setup. Also, due to the major issues over the past year or two with the way Oracle has handled Java, you could find yourself, like many java-based software companies have, scrambling to fix things when a new version of Java breaks it. (I'd personally stay away from Java for any tools right now because of this; I see Java being dumped for other platforms due to the problematic changes Oracle has introduced recently.)

Well that's a few to get you started. All of those projects are well supported and you can find plenty of documentation on the web for installing/customizing them.

Backing up a Wiki (MediaWiki)

So a friend is working with a wiki and was asking about backing up and moving if necessary. So I thought I'd put out what I've done with mine. copy the script and name it backupwiki.sh or something.

Note that I got the bulk of this from here: http://www.mediawiki.org/wiki/User:Flominator/Backup_MW

#!/bin/bash
FNAME=`date +%Y-%m-%d`
mysqldump -u username --password=pwd --add-drop-table -B my_wiki > ${FNAME}.sql
zip -r /media/external1/bkup/wikibackup/${FNAME}.zip images/ ${FNAME}.sql LocalSettings.php extensions/
rm ${FNAME}.sql

cd /media/external1/bkup/wikibackup
 
#Count files in directory (hidden files (filename starts with a dot) are ignored) 
file_count=`ls | wc -l` 
 
#Do until there are more than or equal 6 files present
while [ $file_count -ge 6 ]
do
        #you can save deleted filenames in variable (e.g. for deleting files also in backup directory)
        #not recommended for filenames with space(s)
        del_files="${del_files} `ls | head -n 1`"
        #Delete alphabetically oldest file (ls sort by name is default)
        rm `ls | head -n 1` 
        #Count files again
        file_count=`ls | wc -l`
done

So what this does is dump the mysql database and copy the relevant files/directories you'll need.

Lets look at it in depth:

#!/bin/bash
FNAME=`date +%Y-%m-%d`
mysqldump -u username --password=pwd --add-drop-table -B my_wiki > ${FNAME}.sql

So here we are creating a variable FNAME with todays date in the format yyyy-mm-dd and then using mysqldump to dump the database to a sql file. Replace username and pwd with your corresponding mysql username/password. Also replace my_wiki with the name of your wiki database name.

Note that you want to make sure this file whateveryoucallit.sh is not readable by anyone but root; root also needs execute ability. Otherwise everyone can see your username/password for mysql. Not a smart thing.

zip -r /media/external1/bkup/wikibackup/${FNAME}.zip images/ ${FNAME}.sql LocalSettings.php extensions/
rm ${FNAME}.sql

Next we will create a zip file with FNAME variable so we know what date the backup is and will include in that the sql file and other relevant files and folders. You should create a directory to hold these backups and use the path to that in place of where I have /media/external1/bkup/wikibackup. (yes I use an external drive for my backups)

cd /media/external1/bkup/wikibackup
 
#Count files in directory (hidden files (filename starts with a dot) are ignored) 
file_count=`ls | wc -l` 
 
#Do until there are more than or equal 6 files present
while [ $file_count -ge 6 ]
do
        #you can save deleted filenames in variable (e.g. for deleting files also in backup directory)
        #not recommended for filenames with space(s)
        del_files="${del_files} `ls | head -n 1`"
        #Delete alphabetically oldest file (ls sort by name is default)
        rm `ls | head -n 1` 
        #Count files again
        file_count=`ls | wc -l`
done

This last part of the script needs to have you replace /media/external1/bkup/wikibackup with your path as well. It will make sure that we only keep 5 days of backups.

And last you want to edit root's crontab (I run it as root since I set the file to only be readable/executable by root.) and add this script. If you set it up to run every day, you'll have the last 5 days for backups. If you set it to once a week, you'll have the last 5 weeks backups. Your choice. This is a simple script and you can also use this script one time if you need to backup to move your wiki to another host.

Hope this helps...

OpenVPN on Ubuntu Server 11.10

To install OpenVPN on Ubuntu is pretty simple. Configuring it correctly is where most people's problems lie. There are two ways to setup the vpn: 

1. Bridged (VPN clients get their IP from the LAN's DHCP server. VPN Server acts as a bridge to the LAN.)
2. Routed (VPN server creates a separate network and acts as the DHCP server and router handing out IPs to VPN clients. VPN clients then access the LAN via the VPN Server acting as the gateway.)

Routed is a little easier to setup and will work for most scenarios unless you need non-IP or broadcast traffic. So that's what I'll walk through here.

First, lets install the packages:

# sudo apt-get install openvpn

That was easy. Next we need to start the configuration and create some keys for encryption of the VPN tunnel.

Let me start of saying I've done this on Ubuntu 10 and 11 and CentOS 4 and 5. This walkthrough was done based on what I did to setup on Ubuntu 11 but you can adapt for various OS'.

After installation you should have a /etc/openvpn directory to work in and we have some examples in /usr/share/doc/openvpn/examples. We're going to start by copying some example files to /etc/openvpn to work with (no use recreating the wheel).

# cp -R /usr/share/doc/openvpn/examples/easy-rsa/ /etc/openvpn

Now you'll have an easy-rsa directory inside your /etc/openvpn and it'll have 2 sub-directories, we're only interested in the 2.0 directory. (so feel free to delete the 1.0 directory. not also that you may want to restrict rights to this directory after we're done so that only root can read/edit it.)

You may want to read the /etc/openvpn/easy-rsa/2.0/vars file and edit anything in there you want (key size for example). Once you're satisfied, we need to run the clean-all script followed by the build scripts:

# cd /etc/openvpn/easy-rsa/2.0

# source ./vars

# ./clean-all

# ./build-ca

# ./build-key-server server

Now you have the key for your server. Next we need to create client keys. Feel free to replace "client" in the following commands with descriptive client names that you prefer if you're setting up more than 1 client. If this is just for you, you can leave it as client.

# ./build-key client

# ./build-dh

Next we want to copy the keys that were generated for the server from the keys directory up to /etc/openvpn. Copy the following files to /etc/openvp/:

• ca.crt
• ca.key
• dh1024.pem
• server.crt
• server.key

Now we need a basic configuration file for the server that we can start with and edit to our needs. Luckily this is in the examples area. Go to /usr/share/doc/openvpn/examples/sample-config=files/ and unzip server.conf.gz if necessary. We then want to copy that server.conf file to /etc/openvpn. Next copy the client.conf file out of that directory into another directory like your home directory to work on later.

Next we want to edit some of the config file:

First, if you have multiple NICs, then you may want to set the local a.b.c.d to the IP address of the NIC you want to dedicate to openvpn traffic. (should go without saying that your NIC needs a static IP address for this to work) This is optional. I do recommend if you are using multiple NICs and they are not bonded that you dedicate VPN traffic to one NIC.

Next we will run openvpn on the standard port 1194 but you can change this if you like. Just make sure nothing else is running on that port on any servers on your network. (last thing you want to do is pick something like port 80 and then have your firewall forwarding 80 to your webserver on a different box; then your VPN will never work)

Leave UDP set for protocol unless you need to run it on TCP for some reason.

The next section is where you determine how your going to setup VPN bridged or routed. If doing bridged, you must select dev tap here and server-bridge setting further down. (there's also other things that you need to do to get bridged working that we aren't covering here) Since we are doing routed, we'll select dev tun here and server setting further down. You will see:

server 10.8.0.0 255.255.255.0

This is ok to leave as is unless you use 10.8.0.0 for your LAN or another VPN network. If so then change this. 

Next we need to configure the server to push the route to the LAN to the VPN client. We set that via (replace 192.168.1.0 with your LAN subnet):

push "route 192.168.1.0 255.255.255.0"

That will expose the LAN to the VPN clients and allow them to communicate to the network. However, to allow traffic to truly flow, we have to tell the network how to route return traffic to the VPN clients. This is done by configuring a static route on your gateway/router to direct all traffic to the 10.8.0.0 (or whatever subnet you chose above) to go to the static IP of the VPN server's NIC (the address you may have used above in the a.b.c.d section). Also, you need to enable IP forwarding which can be done by entering the following at a shell after you've finished editing the config:

# sudo echo 1 > /proc/sys/net/ipv4/ip_forward

The last things I'll mention in the server config is that if you want vpn clients to see each other, you will want to uncomment the line:

client-to-client

And I would suggest uncommenting the following so that the server doesn't run as root. Make sure that "nobody" and "nogroup" exists on your server. If not, you'll need to create them or use a comparable user.

user nobody

group nogroup

Now save the file and setup the additional routing and IP forwarding mentioned above if you haven't yet. One additional thing you'll need to do is to setup port forwarding on your router to allow port 1194 to go to your VPN server unless you have your VPN server in a DMZ.

Next you'll want to edit the copy of the client.conf file you put in your home directory earlier to match up with the server.conf file you just edited. Then copy that client.conf and the following files from /etc/openvpn/easy-rsa/2.0/keys directory to your client machine

• client.key
• client.crt
• ca.crt

Now start the openvpn service on the server:

# /etc/init.d/openvpn start

If there are no errors, that's a good start. Next try to connect from the client machine and if it works, test that you have access to the LAN. Note you can test connection to the openvpn server from inside the network if you use it's IP for the server IP in the client config file. Then you can change that to the public IP or FQDN.

(a sub note here for windows users, you will need to rename client.conf to client.ovpn on windows, but it's still a text file and you can edit on windows with wordpad)

* Once you have it up and running, I recommend reviewing the following information on ways to make it more secure as necessary:

http://openvpn.net/index.php/open-source/documentation/howto.html#security

Static IP's in linux

If you're like me, you like centralized control of your configurations in Linux.  I'm not a big fan of letting gui tools manage things like my network settings and stuff. Especially on a server. So a friend of mine recently asked how to set a static IP for a box. So here's how I do it in the core config files for linux networking.

(Note that if you want to try static DHCP, you can do that with Netgear SOHO routers. I'm not sure about other brands (Belkin, Linksys, Dlink,etc. static DHCP is where the client still asks the DHCP server/router for an address but the DHCP server/router is configured to always give that client MAC address the same IP rather than whatever is first available out of the pool. It's a simple way to manage static IP's and lets your DHCP server/router still handle controling the other information including DNS servers. If you change DNS servers, for example, then you only need to change the info in the DHCP server/router.)

First you need to understand that if you are using DHCP to get info and change to static, you'll need more than just an IP configured. You'll also need other info that the DHCP server gives out when you lease an IP address: netmask, gateway, and DNS servers. I'm only going to discuss IP4 here since most SOHO applications will still be using that for a while. I use Ubuntu/Debian so some of the locations here may be specific to that distribution however, there are similar files in RedHat/CentOS that are configured the same.

The files I want to modify are located in /etc and all need to be modified as root. I recommend doing this from terminal since if you restart the box and it doesn't come up, you'll

First lets setup our IP address, gateway, and netmask.

Use vi, nano, emacs or whatever to edit the file /etc/networks/interfaces. You should at least have an eth0 interface. So lets set that up.

auto eth0
iface eth0 inet static
   address  10.0.0.5
   netmask  255.255.255.0
   gateway  10.0.0.1

So address will be whatever you want to give your box. IMPORTANT: Do not give it an address that is in your DCHP pool. This will cause big problems. Netmask is typically that unless you're doing something unique. Gateway is typically set to your router.

One additional thing we need to setup is the DNS info since we won't be getting it handed to us from the DHCP server. Look at your server and find out what the DNS server IPs should be. This could be a local DNS server if you have one on your network. It could be your internet provider's DNS servers. Or you can use openDNS servers. I prefer openDNS over internet providers. I actually run my own DNS caching server locally but it forwards out to openDNS servers. There are two basic IPs for openDNS server pools: 208.67.220.220 and 208.67.222.222. So I'll use those here to configure in my example.

We now want to edit /etc/resolv.conf to say the following:

nameserver 208.67.220.220
nameserver 208.67.222.222

It's important to at least list two nameservers in case one isn't reachable. You can use any number of servers and it will try them in the order listed. Therefore, you may combine your internet providers DNS servers with the open DNS servers and list them all if you like. If you run a local DNS list it first, then list an external DNS. While the local DNS should always respond and should forward out to another DNS on the internet for lookups it doesn't know, it's safe to have that backup one listed on your server too. That way if local DNS is down, your server still has a backup to ask.

With that, a restart of the network service or rebooting, and you should be back up with your static IP. I recommend rebooting the box if you can and testing. Never hurts to make sure that it'll work if it does end up rebooting. You don't want to have to troubleshoot that when you're in a crunch trying to bring a box back up after a power failure or something.

One additional note, if you have multiple nics, you may try bonding them. This works pretty easily with the current linux kernel, there are a bunch of resources on how to do this and I may right this up in the future for what I did. You can bond them in a round robbin way that will provide redundancy and some network boost. I have not been successful at bonding wireless nics. (whether bonding wireless to wired or wireless to wireless) I've read that due to the way WPA/WPA2 encryption works that it's likely impossible. If you decided to try to bond wireless, be prepared for a long path with many hours of trial and error and lots of internet research. But if you succeed, please post a tutorial somewhere for those of us who weren't successful.

Want to do SMART analysis on a USB hard drive?

If you've ever run into a problem with an external hard drive, you've probably tried to run a variety of checks against it and likely decided at some point that plugging it in internally would be easier to run some checks. SMART helps monitor drives and allows you to run some basic tests against them (when not mounted).  So you can run SMART commands over USB! So before you go grab that screwdriver, give this site a read and try to run some checks against the drive over USB.

SMARTmontools over USB

A great resource for *nix for all levels

Thought I'd post this as it is a great resource I've run across many times.
nixCraft

Raid and LVM ... more info

Here's some links for more information on Raid and LVM. Personally, I have no problems running Raid without LVM, but I'd never run LVM without Raid. Make sure you fully understand both before you start using them. I am not using LVM on my rebuild, just the 4x40 Raid10 and 2x80 Raid1 setups.(Note: Raid0 and LVM are similar except that Raid0 has no snapshots or redundancy; I would never use Raid0 to be honest. Raid10 is basically Raid0 + Raid1; much better option because it provides the redundancy.)

Besides redundancy, Raid can possibly increase performance of read access to the disks. Just make sure your disks for your raid array are on different channels.

Raid is not a substitute for backups! It only protects against hard drive failure; not against filesystem corruption. LVM has the ability to do snapshots at filesystem level but still that's not a replacement for a good backup system. Make sure you have a good backup system in place. A good disaster recovery system includes at minimum Raid, snapshots, and offline backup strategies.

Here are some links to some more information on LVM and RAID:

• Linux Software Raid with mdadm HowTo
• A beginners guide to LVM
• Wikipedia: Raid
• Wikipedia: LVM (Linux)