howto » Untitled

console2

For -years- I have hoped and searched and wished and moaned for lack of a halfway decent terminal emulator on Windows. Specifically, one that:

Lets me use my standard unix toolset.
Gives me a command line interface to the host machine WITHOUT requiring me to do something ridiculous like ssh’ing to localhost or firing up an x11 server…
Realizes that sometimes the display is wider than 80 characters…
Provides customizable color codes (#006 on black really stinks).
Doesn’t set TERM=something-nobody-supports.

One wouldn’t think this was too much to ask. But none of the major open source projects of which I am aware provide this. I even tried my hand at writing one myself but got distracted before it was any good.

So, for years, I have used Cygwin xterms and rxvt as a mildly tolerable alternative to, well, nothing.

Today, a coworker and I discovered a 3-year-old blog post promoting Console, a GPL licensed CMD.exe replacement that matches all of my base criteria plus my big dream feature of tabs. TABS!

Console2, Where have you been all my life?!

The project is ancient – but I was using linux desktops for work back in its early days so that probably accounts for my missing it back then.

In the grand tradition of old Sourceforge projects, there is no installer. You just decompress it somewhere and run the exe directly.

When I launched it the first time, I was unsurprised by the 80×25 courier 10 cmd.exe shell it launched by default. I opened the settings menu and was very very pleased with what I found on the first screen. A few minutes later, I had it pointing at my cygwin install:

And a few minutes later:

Read the rest of this entry »

virtualbox rules

Nothing much to say here, but with absolutely minimal pain and suffering, I have 64-bit linux virtual machines running on top of my 32-bit windows XP install. This pleases me.

The recipe:

Compatible CPU with VT-x/AMD-V enabled in the BIOS
Innotek/Oracle/Sun VirtualBox (a current version) with hardware virtualization enabled
Profit!

The one downside to this? 64-bit VM’s running on 32-bit host OS can’t see multiple cpu’s. Boo. Hoo. I’ll just run more VM’s!

lazy php profiler

Caveman profiling with a side of "where were you at 9pm on the night in question?" As always, season to taste.

<?
$_profile_log = "/tmp/php-profile.log";

function _profile() {
static $fh;
if( !isset($fh) ) {
global $_profile_log;
if( !file_exists($_profile_log) ) {
@touch( $_profile_log );
@chmod( $_profile_log, 0664 );
}
$fh = @fopen( $_profile_log, "a" );
}
if( !$fh )
return false;

$stack = debug_backtrace();
if( $stack[1] )
$base = $stack[1];
else
$base = $stack[0];
$buf = $base['file'].":".$base['line'].", ";
if( $base['class'] )
$buf .= $base['class'].$base['type'];
$buf .= $base['function'];

$buf = sprintf("[%s] %s\n",date("H:i:s"),$buf);
return @fwrite( $fh, $buf );
}
?>

Read the rest of this entry »

xtrabackup innodb_file_per_table gotcha

Several months ago, I'd switched to using Percona's xtrabackup & innobackupex for all of my mysql backup needs. I had successfully used these backups to restore and replicate databases across several systems. It is good stuff.

Last week, I needed to set up new replication of an 80gb database. This should have been routine by now, but when I attempted to prepare the backup this time, it whined and complained and failed. I was kind of frazzled by the time I gave up on the issue and declared it a fluke of one sort or another.

Last night, I tried again from Sunday's full backup, and it happened again:

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=75546624, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 75546624
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818  2:54:34  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
090818  2:54:34  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//tmp/#sql6e1e_8cce1_0.ibd
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

I gave up after poking a few things.

This morning's fresh look turned up this bug report.

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo mkdir tmp
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
090818 12:27:41  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=84983808, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 84983808
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818 12:27:41  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 86 1298333184 (6 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1303576064 (13 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1308818944 (20 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1314061824 (27 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1319304704 (34 %)
090818 12:27:44  InnoDB: Starting an apply batch of log records to the database...
... snip ...

That's right. There's a bug in innodb restoration that interprets location of /tmp (configurable in my.cnf) to be relative in stead of absolute.

So, if you have problems while trying to restore from an xtrabackup/ibbackup snapshot (or if you're trying to recover innodb after a crash), just creating the offending tmp directory appears to work.

init.d template

This is a rudimentary template that I've been using for very quick and dirty /etc/init.d scripts recently.

It works under the assumption that your server daemon has a unique name and only ever runs a single instance - this also means that the binary and the init.d script cannot share a name - otherwise strange things happen

Actual invocation logic may need to be updated on a per-service basis and chkconfig style headers would have to be added manually, but it works well for what it is.

[bash]
#!/bin/bash

DIR='' # path to the daemon executable
CMD='' # name of the command itself
ARG='' # optional. any arguments to pass when starting
NAM='' # descriptive name of the daemon so it shows up pretty

function get_ps {
ps --no-header -C${CMD}
}

function do_start {
echo -n "Starting ${NAM}... "
cd ${DIR}
nohup ./${CMD} ${ARG} &
SUCC=`get_ps | wc -l`
if [ "1" == "$SUCC" ]; then
echo "[SUCCESS]"
else
echo "[FAILURE]"
fi
}

function do_stop {
echo -n "Stopping ${NAM}... "
PID=`get_ps | awk '{print $1}'`
kill $PID
SUCC=`get_ps | wc -l`
if [ "0" == "$SUCC" ]; then
echo "[SUCCESS]"
else
echo "[FAILURE]"
fi
}

case "${1:-''}" in
'start')
do_start
;;
'stop')
do_stop
;;
'restart')
do_stop
do_start
;;
*)
#echo "Usage: $SELF start|stop|restart|reload|force-reload|status"
echo "Usage: $SELF start|stop|restart"
exit 1
;;
esac
[/bash]

gearman 0.3 php extension api

No real preamble to be made here. Gearman is a distributed job queuing system by the fine folks who brought us memcached. It is nicer than anything else I've looked at. I am attempting to switch one of my projects over to it (replacing a crufty curl + unix sockets + memcached monstrosity that attempted to do the same job).

The documentation is lacking, but if the discussion group is any indication, real docs are a high priority for the project team. Today, I visited the IRC channel to ask for a status update on docs for the PHP extension api (as opposed to the PEAR all-script api, whose auto-generated docs are broken). Turns out my suspicions were right. Documentation is a high priority and none currently exists for the api in question. However... I was informed that the classes support reflection... so

A quick grep of the source for the extension tells me that I am looking at four classes: GearmanClient, GearmanWorker, GearmanJob, and GearmanTask. A ridiculously short php script later...

<?
Reflection::export( new ReflectionClass('GearmanWorker') );
Reflection::export( new ReflectionClass('GearmanClient') );
Reflection::export( new ReflectionClass('GearmanJob') );
Reflection::export( new ReflectionClass('GearmanTask') );
?>

And I can at least try to make a human readable list of available methods.

GearmanWorker

__construct()
clone()
error()
returnCode()
setOptions( $option, $data )
addServer( $host, $port ) - both args optional, examples say defaults are localhost on port 4730.
addFunction( $function_name, $function, $data, $timeout ) - data and timeout optional
work()

GearmanClient

__construct()
clone()
error()
setOptions( $option, $data )
addServer( $host, $port ) - reflection says REQUIRED, however the provided examples and personal experience says otherwise
do( $function_name, $workload, $unique ) - unique is optional
doHigh( $function_name, $workload, $unique ) - unique is optional
doLow( $function_name, $workload, $unique ) - unique is optional
doJobHandle()
doStatus()
doBackground( $function_name, $workload, $unique ) - unique is optional
doHighBackground( $function_name, $workload, $unique ) - unique is optional
doLowBackground( $function_name, $workload, $unique ) - unique is optional
jobStatus( $job_handle )
echo( $workload )
addTask( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskHigh( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskLow( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskBackground( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskHighBackground( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskLowBackground( $function_name, $workload, $data, $unique ) - data and unique are optional
addTaskStatus( $job_handle, $data ) - data is optional
setWorkloadCallback( $callback )
setCreatedCallback( $callback)
setClientCallback( $callback)
setWarningCallback( $callback)
setStatusCallback( $callback)
setCompleteCallback( $callback)
setExceptionCallback( $callback)
setFailCallback( $callback)
clearCallbacks()
data()
setData( $data )
runTasks()

GearmanJob

__construct()
returnCode()
workload()
workloadSize()
warning( $warning )
status( $numerator, $denominator )
handle()
unique()
data( $data )
complete( $result )
exception( $exception )
fail()
functionName()
setReturn( $gearman_return_t )

GearmanTask

__construct()
returnCode()
create()
free()
function()
uuid()
jobHandle()
isKnown()
isRunning()
taskNumerator()
taskDenominator()
data()
dataSize()
takeData( $task_object ) - optional
sendData( $data )
recvData( $data_len )

The extension also appears to expose all constants defined in the C api.

I have since added this to the official wiki - so there are at least SOME docs on the site now

Untitled

and still for good reason.