TIP: Run it with the switch –dryrun to show you what it would have deleted.

Here is the code:

To get started – simply use the safari browser on your iphone and point to the address:

http://fanwidgets.com/iphone

You can select your team from the team dropdown. If you like the schedule, you should hit the “+” at the bottom of the safari browser, and tap “Add to home screen”. A nice little icon of your favorite team will allow quick-access to the schedule. The schedules are updated automatically as the games are played.

Here’s a screenshot for the Louisville 2010 schedule:

I hope you enjoy!!!

In an earlier posting – I showed you how to automatically create snapshots of your volumes. This can come in handy. It can also mean you’ll end up with dozens of snapshots for volumes that no-longer exist. The stored snapshots are taking up space and costing you money, and it can really add up if you aren’t good about cleaning that sort of thing up. IF you automate snapshot creation – you will over time end up with snapshots that are of no value for you.

KEEP IN MIND that you may purposely have snapshots for volumes that no longer exist. You may (for example) have snapshots that were backups of older volumes that you wish to keep around. EC2 allows you to label/describe your snapshots so you can differentiate those.

So I’ve designed a quick shell script that will show you all of the snapshots you have that do not match up with a current volume. Make sure you have your environment setup and the latest version of the EC2 Tools API installed and in the PATH. You can place this command as a cron entry, or simply run it from the command line. Here is the command:

for s in $(comm -13 <(ec2-describe-volumes |cut -f2 |sort -u) <(ec2-describe-snapshots |cut -f3| sort -u)); do ec2-describe-snapshots |grep $s; done;

Just like a few of my earlier scripts - this one-line code loop uses the AWS command line tools. I'll explain what is happening.

The bulk of the activity happens in this chunk of code:

comm -13 <(ec2-describe-volumes |cut -f2 |sort -u) <(ec2-describe-snapshots |cut -f3| sort -u)

The comm command will compare the output from the first list (the list of volumes) to the second list (the list of volumes that have snapshots).

The first list is the unique and sorted list of volume identifiers.
The second list is the unique list of volume identifiers for all of the snapshots.

The comm parameters -13 tell comm to suppress lines unique to stream 1 and lines that are common to both streams. So what we end up with is a list of snapshotted-volumes that aren't included in the list of volumes. Pretty clever, huh?

Next we iterate through this list to run the ec2-describe-snapshots command - grepping the output for each volume. WARNING - this can be a time-consuming operation - especially if you have hundreds of volumes!!

Now the output from this command only shows you the list of snapshots. Here is an example:

SNAPSHOT snap-xxxxxxxx vol-xxxxxxxx completed 2010-05-24T20:49:17+0000 100% XXXXXXXXXXXX 15 Lengthy Snapshot Description provided by you.
SNAPSHOT snap-xxxxxxxx vol-xxxxxxxx completed 2010-08-02T02:34:02+0000 100% XXXXXXXXXXXX 15 KEEP THIS snapshot forever
SNAPSHOT snap-xxxxxxxx vol-xxxxxxxx completed 2010-05-24T20:49:17+0000 100% XXXXXXXXXXXX 15 Automated backup of a volume we don't care about anymore
SNAPSHOT snap-xxxxxxxx vol-xxxxxxxx completed 2010-05-24T20:49:17+0000 100% XXXXXXXXXXXX 15 Perhaps another backup we don't care about


Now that you have this list, you can use the ec2-delete-snapshot to individually delete the un-needed snapshots.

Don't you feel better for keeping your EC2 world a little cleaner.

See more information here

for a in $(ec2-describe-volumes |grep -Ev "ATTACHMENT" |cut -f2);do ec2-create-snapshot $a; done;

All this is doing is grabbing the volume ID’s from the ec2-describe-volumes output – removing the extraneous information, and executing the ec2-create-snapshot command on each volume. The good thing about this is that it doesn’t have to be modified as you add new volumes. Please see my earlier posting on snapshot cleanup to have a full solution from creation to cleanup.

Quick and easy.

So the problem in a nutshell is I have 10 volumes, each of which is cron’ed to be snapshotted at various times of the day (depends on the specific volume as to how often it is backed up). With 10 volumes, my S3 storage costs can get out of hand quite quickly. So I needed to develop a set of scripts that would scan my snapshots – and remove the oldest ones – so I’m not paying for that storage. It is important to keep a couple of snapshots for each volume (at the very least) – and in some cases, I’d like to keep several snapshots. For example, one of my volumes is responsible for storing the main database for the CMS. This is backed up once every two hours. For that specific volume, I’d like to always have my choice of the last 10 snapshots to restore. If the database all of the sudden becomes corrupt, it may be necessary to restore earlier backups to see where and when the corruption started. Other volumes may only require the last 1 or 2 snapshots. So this script needed to be flexible – in that I could specify how many backups I’d like to keep for each volume.

PHP as an OS scripting language

My SHELL scripting experience is pretty limited – especially when performing advanced functions. So instead of using BASH, I chose to use PHP to develop this script. I’ve had decent success automating functionality with PHP over the years. It’s not nearly as clean as PERL or some OS shell scripting, but it does work, and it serves 2 other purposes: 1. consistency across the entire application, and 2. it allows me to leverage other components/libraries written for other parts of the application.

A few things to know before you try to get this working:

1. This script assumes you have the Amazon EC2 API installed, and the appropriate environment variables properly set. See Amazon’s documentation to get through that step.
2. This script is setup to run as a cron job – so you must have it set as eXecutable.
3. I make no gurantees this will work for you – but it works for perfectly for me.

Here’s the code:

        #!/usr/bin/php
        # Set the line above to the location of your PHP command. - don't forget your start PHP tag

        # How many snapshots to keep by volume
        $keep_qty = 2;

        # Customize how many to keep by volume.  You must specify the volume ID and the # to keep.  If you exclude a volume - it uses the default $keep_qty.
        $keep_by_volume = array("vol-identifier-1" => 2, "vol-identifier-2" => 10, "vol-identifier-3" => 6, "vol-identifier-4" => 10);

        # This script has a depdency on the EC2 tools and environment variables
        # $out stores the result of the command-line script "ec2-describe-snapshots"
        $out = array();
        exec('ec2-describe-snapshots', $out);

        /*
        The command line output looks like this:
        SNAPSHOT	snap-0Xa4X6Xe	vol-ident-1	completed	2009-10-29T05:00:34+0000	100%
        SNAPSHOT	snap-7X3XdXX8	vol-ident-1	completed	2009-10-30T05:00:34+0000	100%
        SNAPSHOT	snap-X6aX2XX5	vol-ident-2	completed	2009-10-29T05:00:21+0000	100%
        SNAPSHOT	snap-XeXXcX76	vol-ident-2	completed	2009-10-29T10:00:08+0000	100%
        SNAPSHOT	snap-X8XdXfX1	vol-ident-2	completed	2009-10-29T16:00:18+0000	100%
       */

        # Store each snapshot in it's own array element
        $snaps = array();
        foreach ($out as $snap) {
                # Notice the output above is separated by tabs.
                $snaps[] = split("\t",$snap);
        }

        # convert to a unix timestamp
        $inx = 0;   # counter
        foreach ($snaps as $s) {
                $snaps[$inx][4] = strtotime($s[4]);
                $inx++;
        }

        # You can't really sort a PHP array on an element within the array without doing some tricks
        # So here, we're going to turn the array inside out so we can sort on the volume
        # and the timestamp
        foreach ($snaps as $key => $row) {
                $column1[$key] = $row[0];
                $column2[$key] = $row[1];
                $column3[$key] = $row[2];
                $column4[$key] = $row[3];
                $column5[$key] = $row[4];
                $column6[$key] = $row[5];
        }

        # sort it
        array_multisort($column3, SORT_ASC, $column5, SORT_DESC, $snaps);

        # Now store a consolidated array of each volume with it's snapshots
        # This will look like
        $all_snaps = array();
        foreach ($snaps as $s) {
                if (empty($all_snaps[$s[2]])) {
                        $all_snaps[$s[2]] = $s[1];
                } else {
                        $all_snaps[$s[2]] .= "," . $s[1];
                }
        }

        /*
         # At this point, we should have an array that contains looks like this
         $all_snaps['volume-id-1'] = 'snapshot-id-1,snapshot-id-2'
         $all_snaps['volume-id-2'] = 'snapshot-id-1,snapshot-id-2'
        */

        # Since these are sorted from newest to oldest, we can go through these rows
        # and delete all of the entries past the $keep_qty count
        foreach ($all_snaps as $volume => $vol_snaps) {
                $snap_arr = split(",",$vol_snaps);

                $count = 1;
                # is this volume in the special keep_by_volume array we declared up top?
                if (array_key_exists($volume, $keep_by_volume)) {
                        $keep = $keep_by_volume[$volume];
                } else {
                        $keep = $keep_qty;
                }

                # Show how many snapshots we're keeping for this particular volume
                print "Volume $volume; keeping $keep\n";

                # Iterate through snapshots for this volume
                foreach ($snap_arr as $s) {
                        if ($count <= $keep) {
                                print "Keeping: $volume/ $s \n";
                        } else {
                                # Delete the snapshot, and print the output from the ec2-delete-snapshot command
                                print "Deleting: $volume/ $s\n";
                                $out = array();
                                $cmd = "ec2-delete-snapshot $s";
                                exec($cmd, $out);
                                print "Output from command $cmd: \n";
                                foreach ($out as $o) {
                                        print $o . "\n";
                                }
                        }
                        $count++;
                }
        }

       # Close your PHP here

So there you have it. I'd love to see someone post a BASH version of a script that would do this exact same thing.

I hope you found this useful.

UPDATE: I was made aware of a solution that does what I've built here. Oren Solomianik offered up his solution as an alternative. See http://code.google.com/p/ec2-delete-old-snapshots/. It looks pretty good! The only real difference is his script works with how many days old the snapshot is - where mine just guarantees that a certain number of backups (configurable by volume ID) are always available.

Also, Oren's tool uses the Amazon EC2 PHP Library - which I purposely chose not to use. It's not that I have anything against it - I was simply trying to avoid extra requirements.

Another Update: dohpaz42 informs me that the split command is being deprecated in PHP 5.3 and should'nt be depended on. You should use explode instead. Thanks Ken!!

Perhaps the most exciting thing is integration of the lists experience into their API. As a developer, I can see where this will allow a more realistic way of integrating relevant Twitter’ers into content site. I can see where domain experts will build lists of experts in their field who, if they have something meaningful to say, could provide for some very useful and relevant integration into content sites. Like Robert Scoble’s list for Most Influential in Tech.

The missing piece of the puzzle
I wonder if Twitter has plans to give Twitterers the ability to categorize their tweets to allow developers to filter out the nonsense. If a domain expert is tweeting that he’s standing in line at the bank (or something comparably lame) – we as developers would rather skip the noise and get to the more relevant tweets. The noise is possibly OK for some applications (like celebrity stalking or fan-sites) – but for someone looking for meaningful micro-postings from experts in a particular field – it’s going to be increasingly more important to be able to differentiate the junk from the substance. I’m not sure exactly how they would do this, but if the stream of consciousness is ever to be considered relevant – I personally think this has to be an important part of their plans.

One possibility exists today – without getting Twitter to make any changes. It would be to have the Twitter’er use something comparable to hash-tags. For example, if a domain expert wants to tweet that their kid has a runny nose, then maybe they tag-it to be ignored by using an asterisk or an underscore. Like:

“_ Little Billy has a snotty nose today”

If there is an accepted standard for differentiating the noise from the normally brilliant bursts of genius – we developers could easily ignore these where it’s applicable to do so.

It’s important

Obviously, the reason Twitter has become so important is the fact that so many folks are engaged very actively in it. Many very influential folks use it as a very big component of their day-to-day communications about the world they live and work. I’m a big believer in capturing the important nuggets that come out of this. I think at the heart of it will be very useful and engaging content. As a developer, I want to bring that layer out and present it to users!

As a Twitterer myself, I like being able to use this platform to communicate about my world. But if anyone were to ever consider me someone they’d like to listen to about Technology – I want to make it easier for them to do so.

Anyways – that’s enough rambling for now. Let me know what your thoughts are.

In the past, I put together scripts that had hard-coded commands to tar/ftp the applications and data to a remote server. As I added new applications – or as applications would mature (and spread into other areas of the server), I would simply modify the original scripts to include the new stuff. So, of course (because I’m lazy sometimes), I had on several occasions inadvertently neglected to include something in my backup scripts. Over time, my backup scripts became ugly, unmanageable, and over-complicated.

So I came up with a new strategy. Why not develop a set of scripts that:

a. will back up the server nightly directly to remote storage
b. do not have to be constantly modified as new applications or application components are added.

Those seem like reasonable goals.

First, a little background; I have a dedicated Linux server running CentOS (although this technique should be Linux version agnostic). I host roughly 20 different domains on this server (not all of them are mine). For the stuff I develop, these LAMP applications typically consist of:

1. an htdocs directory (where the web-application is served from).
2. one or more MySQL databases (for application data and framework specific databases and tables)
3. associated scripts (for cron jobs, external activities, etc)

So to do this, you have to possibly reorganize your application. Just be careful when you are following these instructions not to break your application, and make sure you think of config files, crontabs, etc.

Let’s get started:

1. Re-organize your applications, and use consistent naming conventions: Choose a path on your server, and store your application folder and scripts folders there. I use /usr/local. My application folders always follow the pattern htdocs.domain where domain is obviously the name of the domain (i.e. techkismet.com). So for this website, the full-path/name of the application is /usr/local/htdocs.techkismet. Remember, if you move your application – you will need to make changes to your apache configuration files to point to your new htdocs directory.

I do the same thing with my application scripts. Using the same example before, my scripts folder would be /usr/local/scripts.techkismet

2. Create a home for your backup scripts and associated files: I used /usr/local/backups. . You should also create a sub-folder within that folder called “temp”. This is where your script will temporarily put all of the files that are being backed-up.

3. Build your backup shell script

The main backup script is a BASH shell-script that identifies and backs-up each of your applications and databases. This file is called /usr/local/backups/backup.sh. Here is my version as a starting point:

#!/bin/sh

# First remove the contents of the temp directory - the files in there are from yesterday.
rm -rf /usr/local/backups/temp/*

# auto-dump all mysql databases
for a in $(echo "show databases" | mysql -u user --password=password | egrep -v '^Database|^information_schema|^test'); do mysqldump -u user --pasword=password $a > /usr/local/backups/temp/$a.sql; done;

# Auto backup all htdocs.* and scripts.* folders in /usr/local (these are all of the application and scripts directories);
cd /usr/local
for a in $(ls -d htdocs.*); do tar -czf /usr/local/backups/temp/$a-backup.tgz $a/* >/dev/null 2>&1; done;
for a in $(ls -d scripts.*); do tar -czf /usr/local/backups/temp/$a-backup.tgz $a/* >/dev/null 2>&1; done;

# Additional directories/files to handle - put any other files you wish to back up here.
tar -czf /usr/local/backups/temp/other-files.tgz /path/to/local/files/* /path/to/another/file >/dev/null 2>&1

# Backup this script and others in this directory
tar -czf /usr/local/backups/temp/backupscripts.tgz /usr/local/backups/*.sh  /usr/local/backups/s3-bash/* >/dev/null 2>&1

Let’s take a closer look at what we’re doing here:

MySql database backup:

for a in $(echo "show databases" | mysql -u user --password=password | egrep -v '^Database|^information_schema|^test'); do mysqldump -u user --pasword=password $a > /usr/local/backups/temp/$a.sql;

This nifty command will iterate through your mysql databases, and do a mysqldump of these databases into a file of the same-name (with a .sql extension) in the temp folder. You should obviously change the user and password to a mysql user that has the appropriate rights from localhost.

The “egrep -v” command will exclude from the list the items identified in the string. The first item in the list (“Database”) is required. This text is included in the output of “show databases” and it’s not a real database obviously. You can include whichever database you prefer not to backup in this list. I’m excluding the databases “information_schema” and “test” from the backups.

So this multi-piped command produces a list of databases as strings, and each of these strings will be passed to the mysqldump command. The corresponding backup files will be created and stored in the temp folder.

Application and Scripts Folder backup:

The next few commands will go through a comparable process for the htdocs and scripts folders. The commands:

cd /usr/local
for a in $(ls -d htdocs.*); do tar -czf /usr/local/backups/temp/$a-backup.tgz $a/* >/dev/null 2>&1; done;
for a in $(ls -d scripts.*); do tar -czf /usr/local/backups/temp/$a-backup.tgz $a/* >/dev/null 2>&1; done;

These are a little less complicated, but they achieve the same objective. The ls -d command lists all directories that match the pattern htdocs.* (and scripts.*). The results of the “ls” operation are streamed one at a time to the tar command. When this is finished, all of these directories will have been “tar’d” (using a filename that matches the directory) and stored in the temp directory.

Other Items to backup

There will always be additional files (not included in the strucutres identified above) that you will want to backup. Apache configuration files, php.ini, digital certificates, server documentation/notes, other application configuration files, etc. You can back these up either in groups, or individually using something like the following command:

# Additional directories/files to handle
tar -czf /usr/local/backups/temp/other-files.tgz /path/to/local/files/* /path/to/another/file >/dev/null 2>&1

Add any other files as you see fit.

And, for good measure, you may wish to backup the “backup” directory and scripts.

# Backup this script and others in this directory
tar -czf /usr/local/backups/temp/backupscripts.tgz /usr/local/backups/*.sh  /usr/local/backups/s3-bash/* >/dev/null 2>&1

4. CRON it
You should cron this backup script to run at an appropriate time. I chose 4:00 AM, as that is when my domain activity is at it’s lowest.

Next step: Sending the files to Amazon’s S3
So we’ve only really done 1/2 the work so far, right? You need to send these locally backed up files to a remote storage area. This is where Amazon’s S3 comes in to play. You probably noticed that I’ve decoupled the backup script above from the transmission script (that we’re about to talk about). This was my goal. If I need to re-transmit for any reason, I can do that without regenerating all of the files.

I recently started using S3, and I’m very cooled out by it. It’s a web-services accessible data-storage service, and it can be used to publicly or privately store anything you want. You can find out more by visiting http://aws.amazon.com/s3. Pricing is reasonable. They charge for bandwidth and storage. My typical bill runs around $2.40 a month – and my typical nightly backup is right around 600 MB.

As I mentioned earlier, I send full-backups each night. There are more-advanced tools that allow you to “sync” between local data-stores and S3. This should cut down on your bandwidth usage – and could create a substantial savings. I may make a future posting about this technique.

So, on to the details. Once I signed up for AWS S3, I was able to retrieve my access identifier and my “secret” access key. For my scripts, I used a utility called S3-Bash by Raphael James Cohn. This set of BASH scripts allow you to access your S3 buckets from the Linux command line (get, put, and delete).

So do this:

1. Sign up for AWS S3 (if you haven’t already). Get your access identifier and secret keys. Copy and paste them someplace where you can have them readily available.
2. Download the S3-Bash code.
3. Create a folder in your backups folder called s3-bash, and place the S3-bash scripts in there.
4. Create a text file for each of your access key and secret key. I used the file names “key.txt” and “secret_40.txt” respectively. These files will be used with the s3-bash commands to associate the transmission with your AWS account. Please NOTE: Make sure to follow the instructions here if you are using an editor such as “vim”, it will store a line-feed automatically at the end of the line. The files need to be 20 and 40 bytes respectively and the commands won’t work if the files sizes are not exact. Also, please note that your server time should be accurate for s3-bash commands to work. You can read about this requirement on the AWS knowledge-base.

5. You will need to create your destination “bucket” (or folder) on S3. There are command-line scripts that can do this, but I found it easiest to download and install a browser plugin for FireFox called s3Fox. You can download/install it at https://addons.mozilla.org/en-US/firefox/addon/3247. If you don’t use FireFox – you might want to look into the other options on the AWS knowledge base.

Otherwise, once you’ve installed the plugin (and set it up to access your account) you should create the bucket to store your backup-files. Mine is called “daily-backup”

6. Create a script called /usr/local/backups/daily2s3.sh. Here is what mine looks like including comments:

# This script is executed nightly after backup.sh which produces the files.
# The key files below are required for access to S3
# The key files have to be 20 (for key.txt)  and 40 bytes (for secret_40.txt) .  Use dd to remove the lf if the file was created using VI
# for example
#
# dd if=<21bytefile.txt> of=<20bytefile> bs=1 count=20

cd /usr/local/backups/temp
for a in $(ls); do /usr/local/backups/s3-bash/s3-put -k `cat /usr/local/backups/s3-bash/key.txt` -s /usr/local/backups/s3-bash/secret_40.txt -T $a -c application/octet-stream /daily-backup/$a; done;

As you can see, this script iterates through the files created in the temp folder – and “put”s them in the “daily-backup” bucket on S3. The “cat” commands executed inline incorporate the keys when needed, and keep the keys private (in case someone happens to do a “ps ax” when the command is running. You should cron this script as well. Make sure you schedule it after the backup script – and give your backup script plenty of time to finish. In my case, I scheduled these two scripts 30 minutes apart.

That’s it

I hope you find this useful. Please feel free to leave me comments/feedback!

So I had this great idea that I should make a WordPress Widget that does the same thing. I’m releasing it to the world today! You can see a live example on the bottom left of this page. For more information, I’ve setup this special page where you can get additional information and download it for yourself!

I’d love to get your feedback and suggestions for the next version!