1. Lets me use my standard unix toolset.
2. Gives me a command line interface to the host machine WITHOUT requiring me to do something ridiculous like ssh’ing to localhost or firing up an x11 server…
3. Realizes that sometimes the display is wider than 80 characters…
4. Provides customizable color codes (#006 on black really stinks).
5. Doesn’t set TERM=something-nobody-supports.

One wouldn’t think this was too much to ask. But none of the major open source projects of which I am aware provide this. I even tried my hand at writing one myself but got distracted before it was any good.

So, for years, I have used Cygwin xterms and rxvt as a mildly tolerable alternative to, well, nothing.

Today, a coworker and I discovered a 3-year-old blog post promoting Console, a GPL licensed CMD.exe replacement that matches all of my base criteria plus my big dream feature of tabs. TABS!

Console2, Where have you been all my life?!

The project is ancient – but I was using linux desktops for work back in its early days so that probably accounts for my missing it back then.

In the grand tradition of old Sourceforge projects, there is no installer. You just decompress it somewhere and run the exe directly.

When I launched it the first time, I was unsurprised by the 80×25 courier 10 cmd.exe shell it launched by default. I opened the settings menu and was very very pleased with what I found on the first screen. A few minutes later, I had it pointing at my cygwin install:

And a few minutes later:

I am a very happy camper now. It is apparently still an actively developed project – the last SF commit was two days ago.

The only feature requests I can think of is dynamic tab renaming to follow my host/path as I ssh around places, and profiles that recognize when I am connected to a given host and therefore tint the background to warn that I am on a production server, etc…

To be fair, this is a pretty minor gripe as the application actually has similar functionality already – you can configure individual tab templates, and can set ssh as your shell:

And if you select the little dropdown next to the new tab button you can choose to launch the different kind of tab:

So… I’m content. This is the coolest toy I’ve discovered in a very long time.

The recipe:

1. Compatible CPU with VT-x/AMD-V enabled in the BIOS
2. Innotek/Oracle/Sun VirtualBox (a current version) with hardware virtualization enabled
3. Profit!

The one downside to this? 64-bit VM’s running on 32-bit host OS can’t see multiple cpu’s. Boo. Hoo. I’ll just run more VM’s!

I think this is as simple as I can possibly make it:

A sample series of executions might look like this:

This doesn't have any failsafes, doesn't compress anything, depends on an external call to 'ls', and it actually deletes old files in stead of overwriting them... but it is the shortest, simplest method I've come up with to get the job done.

If I feel like making this a full-fledged series, I might actually post a more thorough implementation later

This simple script scans all normal files in a log directory, and if they are older than a certain cutoff, moves them into a holding directory for old logs. Future passes will check files in the old directory for another age setting and will delete them. That's all there is to it.

Configure your cutoffs, directories of interest, and optionally plug in a better logging mechanism and you're set. (Oh, and change the #! if necessary, of course).

The FILES variable should be populated with a whitespace separated list of files, directories, and block devices to track.

The DB_ABCD variables should be populated with appropriate credentials to talk to a mysql server.

The actual script looks something like this:
[bash]
#!/bin/bash

FILES='/var/lib/mysql/ibdata1 /var/lib/mysql/db/table.ibd /dev/sda1 /var/log/mysql'

LOCAL=`hostname -s`
DB_HOST='aaa'
DB_USER='bbb'
DB_PASS='ccc'

function insert {
FILE=$1
SIZE=$2
QUERY="replace into metrics.storage_usage values( now(), '$LOCAL', '$FILE', $SIZE )"
mysql --host=${DB_HOST} --user=${DB_USER} --password="${DB_PASS}" -e "${QUERY}"
}

for FILE in $FILES
do
if [ -d $FILE ]; then
BASE=$FILE
SIZE=`du -ks $FILE/ | awk '{print $1}'`
elif [ -b $FILE ]; then
TMP=`df -k -P $FILE | tail -n1 | awk '{print $3 " " $6}'`
SIZE=`echo $TMP | awk '{print $1}'`
BASE=`echo $TMP | awk '{print $2}'`
else
BASE=`basename $FILE`
SIZE=`du -k $FILE | awk '{print $1}'`
fi

echo "$BASE = $SIZE"
insert $BASE $SIZE
done
[/bash]

I am putting my data into a table called "storage_usage" in a database called "metrics":

Obviously, this could be tweaked in any different number of ways, based on your needs. One tweak you might want to consider if you're running it in a daily cron is to remove the echo so you don't get an email report of every run. Also, if you might want to record more than one snapshot per file per host per day - in the which case you probably need to change the type of the timestamp column to a datetime. Or there might be cases where you want to change the replace to an insert or... whatever

Last week, I needed to set up new replication of an 80gb database. This should have been routine by now, but when I attempted to prepare the backup this time, it whined and complained and failed. I was kind of frazzled by the time I gave up on the issue and declared it a fluke of one sort or another.

Last night, I tried again from Sunday's full backup, and it happened again:

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=75546624, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 75546624
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818  2:54:34  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
090818  2:54:34  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//tmp/#sql6e1e_8cce1_0.ibd
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

I gave up after poking a few things.

This morning's fresh look turned up this bug report.

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo mkdir tmp
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
090818 12:27:41  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=84983808, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 84983808
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818 12:27:41  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 86 1298333184 (6 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1303576064 (13 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1308818944 (20 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1314061824 (27 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1319304704 (34 %)
090818 12:27:44  InnoDB: Starting an apply batch of log records to the database...
... snip ...

That's right. There's a bug in innodb restoration that interprets location of /tmp (configurable in my.cnf) to be relative in stead of absolute.

So, if you have problems while trying to restore from an xtrabackup/ibbackup snapshot (or if you're trying to recover innodb after a crash), just creating the offending tmp directory appears to work.

It works under the assumption that your server daemon has a unique name and only ever runs a single instance - this also means that the binary and the init.d script cannot share a name - otherwise strange things happen

Actual invocation logic may need to be updated on a per-service basis and chkconfig style headers would have to be added manually, but it works well for what it is.

[bash]
#!/bin/bash

DIR='' # path to the daemon executable
CMD='' # name of the command itself
ARG='' # optional. any arguments to pass when starting
NAM='' # descriptive name of the daemon so it shows up pretty

function get_ps {
ps --no-header -C${CMD}
}

function do_start {
echo -n "Starting ${NAM}... "
cd ${DIR}
nohup ./${CMD} ${ARG} &
SUCC=`get_ps | wc -l`
if [ "1" == "$SUCC" ]; then
echo "[SUCCESS]"
else
echo "[FAILURE]"
fi
}

function do_stop {
echo -n "Stopping ${NAM}... "
PID=`get_ps | awk '{print $1}'`
kill $PID
SUCC=`get_ps | wc -l`
if [ "0" == "$SUCC" ]; then
echo "[SUCCESS]"
else
echo "[FAILURE]"
fi
}

case "${1:-''}" in
'start')
do_start
;;
'stop')
do_stop
;;
'restart')
do_stop
do_start
;;
*)
#echo "Usage: $SELF start|stop|restart|reload|force-reload|status"
echo "Usage: $SELF start|stop|restart"
exit 1
;;
esac
[/bash]

I've been doing a lot of backup scripting recently. Yes, there are tons of commercial apps out there, but none of them that I've looked into are a perfect match for all of our needs. I'll eventually settle on one and it will probably replace 80% of my scripts, but plenty will remain.

One problem that I've encountered while doing the whole backup juggling bit is the ferocious rate of change in the nature of the data we're archiving. Code I'd written a year ago was obsoleted 6 months ago by code that was obsoleted 3 months ago by code that I replaced a few weeks back that is being replaced by the code I'm writing right now.

Another one of the problems is that the sheer quantity of data involved is growing in a very uncontrolled way. Early last May (the oldest archive I have easy access to), a full archive of the entire system was barely 3gb in size. Today, it is closer to 60gb. ~20x growth over the last 9 months.

It's been fun, if somewhat frustrating, dealing with all of the growth.

On January 20th, I needed to perform a long-deprecated sort of snapshot. The code that generated this sort of file no longer worked because so many things had changed. I wound up digging out old scripts from SVN and updating them to run against the new environment.

Because of the amount of data involved, this took a very long time. It didn't help that the scripts consumed an unfair amount of system resources - I couldn't run them with any meaningful priority during the day without crippling everyone else.

Lots of low priority io later, I finally had a 54gb tar file... In one of the three places I needed it.

The first transfer was simple, the hosts are on the same gigabit switch as each other. Unfortunately, scping that much data between two hosts at that kind of speed has negative effects on the systems involved. I had to throttle the transfer way down to before it could run without visibly impacting performance.

The second transfer... wasn't so easy. I needed to move the file to my office without negatively impacting everyone's ability to work - and I couldn't wait for the transfer to run at low enough speeds not to cripple the T1.

We have a backup 6mbit DSL link that I only use for emergencies and for testing. Even at a full 6mbit, the transfer would have taken more than 36 hours. Compressing the file took a while but brought the file size down to a much more manageable 24gb (~11 hours over the DSL).

The only remaining gotcha was that DSL link can't actually SSH through the firewall into the colo So... I started the transfer over https last night and went home.

This morning, it was finally time to decompress the monstrosity locally, but I noticed a hiccup in dsl traffic overnight and figured I'd run a check on things first - just to make sure that http resume had worked correctly.

This was not good. I had a 24gb file that was somehow corrupted... somewhere.

Since re-downloading the whole thing would cost me another whole day... I had to find out a way to repair the file in a reasonable amount of time. Some research and suggestion gathering later, it was confirmed that rsync would probably handle the task.

Assuming that I wouldn't be using an unfair amount of bandwidth for this, I switched back to the T1 link so I could tunnel through SSH again.

(remember, this is unix, no output implies success)

So, yeah. Rsync, I love it when you work.

It took some time and generated a lot of disk activity when the process started, but it worked almost painlessly and only transferred the data I needed - thus leaving the shared network resource free for everyone else

I never really cared about networking much beyond that needed to make sure clients on a lan can talk to their dns server... but we've been growing enough here at work that the needs quickly outpaced my prior skillset. And since I was the closest thing we had to a network admin, I got signed up for classes

It's been fun and profoundly enlightening. I didn't expect to have my way of thinking so radically altered, but I'm hardly complaining.

I'll be starting up the second class in a week or two. This'll include such topics as VLANs, IPv6, and fancy routing protocols. I'm stoked.

It's funny. I never finished college (though I took classes for roughly 10 years), so this is actually the first certificate of education I've received since highschool.

The comment was made at work that I'd dinged as a sysadmin. I haven't. I'm just cherry picking my next few levels in netadmin

What I want:

But of course, nothing like this exists.

Thankfully, svn info's output IS easy enough to parse. You just have to do it your self.

Will give you the revision of your current checkout without the network hit of a call to svn log.

To get the current version of the repo itself (hits the network), add "-r HEAD" to the svn info call:

Of course, svn info also supports outputting info as xml, so you could use that to parse things in a more advanced environment but one where you're still not using the svn api bindings.