The recipe:

1. Compatible CPU with VT-x/AMD-V enabled in the BIOS
2. Innotek/Oracle/Sun VirtualBox (a current version) with hardware virtualization enabled
3. Profit!

The one downside to this? 64-bit VM’s running on 32-bit host OS can’t see multiple cpu’s. Boo. Hoo. I’ll just run more VM’s!

<?
$_profile_log = "/tmp/php-profile.log";

function _profile() {
    static $fh;
    if( !isset($fh) ) {
        global $_profile_log;
        if( !file_exists($_profile_log) ) {
            @touch( $_profile_log );
            @chmod( $_profile_log, 0664 );
        }
        $fh = @fopen( $_profile_log, "a" );
    }
    if( !$fh )
        return false;

    $stack = debug_backtrace();
    if( $stack[1] )
        $base = $stack[1];
    else
        $base = $stack[0];
    $buf = $base['file'].":".$base['line'].", ";
    if( $base['class'] )
        $buf .= $base['class'].$base['type'];
    $buf .= $base['function'];

    $buf = sprintf("[%s] %s\n",date("H:i:s"),$buf);
    return @fwrite( $fh, $buf );
}
?>

Sample use:

<?
require_once "profiler.php";

// setup
function func() {
    _profile();
}

class obj {
    function method() {
        _profile();
    }
    function other_method() {
        func();
    }
    function indirect_method() {
        $this->method();
    }
}

$obj = new obj();

// actual invocation cases
_profile();
func();
obj::method();
$obj->method();
$obj->other_method();
$obj->indirect_method();
?>

Output:

ammon@elzar:~$ cat /tmp/php-profile.log
[19:00:21] /home/ammon/test.php:20, _profile
[19:00:21] /home/ammon/test.php:21, func
[19:00:21] /home/ammon/test.php:22, obj::method
[19:00:21] /home/ammon/test.php:24, obj->method
[19:00:21] /home/ammon/test.php:13, func
[19:00:21] /home/ammon/test.php:16, obj->method

Last week, I needed to set up new replication of an 80gb database. This should have been routine by now, but when I attempted to prepare the backup this time, it whined and complained and failed. I was kind of frazzled by the time I gave up on the issue and declared it a fluke of one sort or another.

Last night, I tried again from Sunday’s full backup, and it happened again:

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=75546624, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 75546624
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818  2:54:34  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
090818  2:54:34  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//tmp/#sql6e1e_8cce1_0.ibd
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

I gave up after poking a few things.

This morning’s fresh look turned up this bug report.

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo mkdir tmp
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
090818 12:27:41  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=84983808, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 84983808
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818 12:27:41  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 86 1298333184 (6 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1303576064 (13 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1308818944 (20 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1314061824 (27 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1319304704 (34 %)
090818 12:27:44  InnoDB: Starting an apply batch of log records to the database...
... snip ...

That’s right. There’s a bug in innodb restoration that interprets location of /tmp (configurable in my.cnf) to be relative in stead of absolute.

So, if you have problems while trying to restore from an xtrabackup/ibbackup snapshot (or if you’re trying to recover innodb after a crash), just creating the offending tmp directory appears to work.

It works under the assumption that your server daemon has a unique name and only ever runs a single instance – this also means that the binary and the init.d script cannot share a name – otherwise strange things happen

Actual invocation logic may need to be updated on a per-service basis and chkconfig style headers would have to be added manually, but it works well for what it is.

#!/bin/bash

DIR=''	# path to the daemon executable
CMD=''	# name of the command itself
ARG=''	# optional. any arguments to pass when starting
NAM=''	# descriptive name of the daemon so it shows up pretty

function get_ps {
	ps --no-header -C${CMD}
}

function do_start {
	echo -n "Starting ${NAM}... "
	cd ${DIR}
	nohup ./${CMD} ${ARG} &
	SUCC=`get_ps | wc -l`
	if [ "1" == "$SUCC" ]; then
		echo "[SUCCESS]"
	else
		echo "[FAILURE]"
	fi
}

function do_stop {
	echo -n "Stopping ${NAM}... "
	PID=`get_ps | awk '{print $1}'`
	kill $PID
	SUCC=`get_ps | wc -l`
	if [ "0" == "$SUCC" ]; then
		echo "[SUCCESS]"
	else
		echo "[FAILURE]"
	fi
}

case "${1:-''}" in
	'start')
		do_start
		;;
	'stop')
		do_stop
		;;
	'restart')
		do_stop
		do_start
		;;
	*)
		#echo "Usage: $SELF start|stop|restart|reload|force-reload|status"
		echo "Usage: $SELF start|stop|restart"
		exit 1
		;;
esac

The documentation is lacking, but if the discussion group is any indication, real docs are a high priority for the project team. Today, I visited the IRC channel to ask for a status update on docs for the PHP extension api (as opposed to the PEAR all-script api, whose auto-generated docs are broken). Turns out my suspicions were right. Documentation is a high priority and none currently exists for the api in question. However… I was informed that the classes support reflection… so

A quick grep of the source for the extension tells me that I am looking at four classes: GearmanClient, GearmanWorker, GearmanJob, and GearmanTask. A ridiculously short php script later…

<?
Reflection::export( new ReflectionClass('GearmanWorker') );
Reflection::export( new ReflectionClass('GearmanClient') );
Reflection::export( new ReflectionClass('GearmanJob') );
Reflection::export( new ReflectionClass('GearmanTask') );
?>

And I can at least try to make a human readable list of available methods.

GearmanWorker

• __construct()
• clone()
• error()
• returnCode()
• setOptions( $option, $data )
• addServer( $host, $port ) – both args optional, examples say defaults are localhost on port 4730.
• addFunction( $function_name, $function, $data, $timeout ) – data and timeout optional
• work()

GearmanClient

• __construct()
• clone()
• error()
• setOptions( $option, $data )
• addServer( $host, $port ) – reflection says REQUIRED, however the provided examples and personal experience says otherwise
• do( $function_name, $workload, $unique ) – unique is optional
• doHigh( $function_name, $workload, $unique ) – unique is optional
• doLow( $function_name, $workload, $unique ) – unique is optional
• doJobHandle()
• doStatus()
• doBackground( $function_name, $workload, $unique ) – unique is optional
• doHighBackground( $function_name, $workload, $unique ) – unique is optional
• doLowBackground( $function_name, $workload, $unique ) – unique is optional
• jobStatus( $job_handle )
• echo( $workload )
• addTask( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskHigh( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskLow( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskHighBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskLowBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
• addTaskStatus( $job_handle, $data ) – data is optional
• setWorkloadCallback( $callback )
• setCreatedCallback( $callback)
• setClientCallback( $callback)
• setWarningCallback( $callback)
• setStatusCallback( $callback)
• setCompleteCallback( $callback)
• setExceptionCallback( $callback)
• setFailCallback( $callback)
• clearCallbacks()
• data()
• setData( $data )
• runTasks()

GearmanJob

• __construct()
• returnCode()
• workload()
• workloadSize()
• warning( $warning )
• status( $numerator, $denominator )
• handle()
• unique()
• data( $data )
• complete( $result )
• exception( $exception )
• fail()
• functionName()
• setReturn( $gearman_return_t )

GearmanTask

• __construct()
• returnCode()
• create()
• free()
• function()
• uuid()
• jobHandle()
• isKnown()
• isRunning()
• taskNumerator()
• taskDenominator()
• data()
• dataSize()
• takeData( $task_object ) – optional
• sendData( $data )
• recvData( $data_len )

The extension also appears to expose all constants defined in the C api.

I have since added this to the official wiki – so there are at least SOME docs on the site now

There is a performance hit for actually having to call the __autoload() method, but if you’re in a situation where the hit for executing a few extra comparison calls is unacceptable… you probably aren’t developing in PHP in the first place

Almost all of the php I’ve written in the last 2-3 years uses autoloading, and it has probably saved me hundreds of hours of aggravation.

In most of my projects, the first line of any script or class usually looks something like this:

require_once "/var/www/common/lib.php";

Then lib.php usually reads something like this:

<?
function __autoload( $class ) {
    include_once( "$class.php" );
}
?>

And that is all that is strictly required to make the magic happen. It is fast, it is easy to understand, it is easy to use. You can use require_once() or include_once() and there is very little meaningful difference.

I’ve looked around the net and found several other attempts at improving on this simple mechanism. But they invariably overcomplicate things. They attempt to recurse source directories, cache filename->class differences to the filesystem, and otherwise turn what should be a simple filesystem operation that the php environment supports natively into a mess of exception handling and wheel reinvention.

There are obviously theoretical instances where you might want to have more than the one require_once/include_once line… but I’ve honestly never encountered one myself.

I mean, you could try to throw an exception if the file didn’t exist or otherwise failed to load… but nothing will happen. Failure to instantiate a nonexistant class is a fatal error in PHP, and will be handled as such with or without you – preempting any attempt at throwing an exception.

The only thing you can add is a bit of extra diagnostics or maybe logging to a separate location.

Assume that we have a file ‘test.php’:

<?
require_once "autoload.php";
$frog = new Frog();
?>

If autoload.php contains a simple simple autoload function that uses require_once(), and Frog.php doesn’t exist anywhere in your include path, the results will look something like this:

ammon@kif:~$ php test.php 

Warning: require_once(Frog.php): failed to open stream: No such file or directory in /home/ammon/autoload.php on line 3

Fatal error: require_once(): Failed opening required 'Frog.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/ammon/autoload.php on line 3

If we had used an include_once() call, the output is similar, but slightly more informative:

ammon@kif:~$ php test.php 

Warning: include_once(Frog.php): failed to open stream: No such file or directory in /home/ammon/autoload.php on line 3

Warning: include_once(): Failed opening 'Frog.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /home/ammon/autoload.php on line 3

Fatal error: Class 'Frog' not found in /home/ammon/test.php on line 4

So that’s probably a bit more useful in tracking down the error. Require calls don’t return anything – they throw a fatal error on failure. Include calls, however, return FALSE on failure and TRUE if the file is (or, in the case of include_once, has already been) successfully included. So you can include_once() and write to a separate logfile (or to the output stream…) if you need more information than the fatal error already provides you.

<rant>

To those who insist on giving your classes and their containing files different names… umm. Wow.

If I have a class called DatabaseConnection, I’m going to put it in a file called DatabaseConnection.php. If I’m working with strange people who somehow don’t think that is explicit enough, I might call it DatabaseConnection.class.php and tweak the autoload method ever so slightly to compensate. There’s no good reason to put it in a file called projx-database_connection.incl or something. No. There isn’t.

If you want to organize your classes into a meaningful directory structure… good for you. Use PHP’s built-in include_path ini option. Don’t waste time trying to cascade down a directory structure searching for the classes – just make sure your includes are all in a set of reliable locations. You don’t actually have to edit the php.ini file and bounce Apache or your php-cgi processes, just define the additional include paths in the same file where you define your autoloader:

set_include_path(
    get_include_path() . PATH_SEPARATOR .
    "/var/www/includes" . PATH_SEPARATOR .
    "/var/www/includes/apple" . PATH_SEPARATOR .
    "/var/www/includes/banana"
);

Naturally, you could turn that into some function calls to dynamically register and unregister directories, etc… but at that point, you’re probably hurting yourself again. If your codebase is being reorganized enough to make maintenance of the list of include dirs onerous without full time intervention, something else has probably already gone very wrong. At best, the code probably doesn’t work anyway, so any brief delay in updating the list can’t hurt any more than whatever else is happening.

</rant>

But seriously. __autoload() is your friend. It will help clean up your code if you let it. It can help enforce naming conventions. It can even improve performance… so long as you refrain from using it to shoot yourself in the foot.

The bug has been reported here, where it was ignored for a few months before being shot down and ignored some more as per php dev team regulations.

Sample code given by the reporter of the bug is markedly similar to the situations I’ve encountered the problem:

pcntl_signal(SIGINT, "sig_handler");
$sock = socket_create_listen($port);
$read_socks = array($sock);
$n = NULL;
$foo = socket_select($read_socks, $n, $n, NULL);

By filling in his blanks, my first test case looks something like this:

<?
function sig_handler($signo) {
        echo "received sig #$signo\\\n";
}
pcntl_signal( SIGINT, "sig_handler" );

$socket = socket_create_listen( 1234 );
$r = array( $socket );
$n = NULL;
while( true ) {
        $foo = socket_select( $r, $n, $n, NULL );
        echo "select returned '$foo'\\\n";
}
?>

When executing the script and pressing ^C (which sends SIGINT), the following occurs:

ammon@morbo:~$ php sigtest.php
PHP Warning:  socket_select(): unable to select [4]: Interrupted system call in /home/ammon/sigtest.php on line 13
select returned ''

Ok, so the warning is to be expected, and we can easily squelch that.

The real problem is that the signal handler never runs.

However… for the first time in my life, a response to a php bug report proves enlightening. The dev who answered this ticket provides his sample code and says he can’t duplicate the bug. Upon looking at the differences between their code, only one difference stands out:

declare(ticks=1);

The declare(ticks) directive is deprecated as of php 5.3 and will not be with us in php 6.0. Ticks are an unreliable, unpredictable, and generally bad thing in php. I’ve neither successfully used them nor seen a successful and justified use.

That being said… turning the tick on but not telling it to do anything appears to address the problem of discarded interrupts:

<?
declare(ticks=1);

function sig_handler($signo) {
        echo "received sig #$signo\\\n";
}
pcntl_signal( SIGINT, "sig_handler" );

$socket = socket_create_listen( 1234 );
$r = array( $socket );
$n = NULL;
while( true ) {
        $foo = @socket_select( $r, $n, $n, NULL );
        echo "select returned '$foo'\\\n";
}
?>

And execution:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

Which is precisely the desired behavior.

I don’t know what the performance hit for turning ticks on is, I haven’t had time to research this. But I can confirm that by declaring ticks globally, it does work in an OO environment as well:

<?
declare(ticks=1);

class signal_tester {
    function __construct() {
        pcntl_signal( SIGINT, array(&$this,"sig_handler") );
        $this->start();
    }

    function sig_handler($signo) {
        echo "received sig #$signo\\\n";
    }

    function start() {
        $socket = socket_create_listen( 1234 );
        $r = array( $socket );
        $n = NULL;
        while( true ) {
            $foo = @socket_select( $r, $n, $n, NULL );
            echo "select returned '$foo'\\\n";
        }
    }
}

$test = new signal_tester();
?>

Executing and hitting ^C:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

After a few minutes of largely unscientific testing, it appears that turning ticks on globally costs a whopping 4 bytes of ram and causes the script to occasionally consume more cpu than the top process I used to monitor it. So… at first glance the cost is pretty negligible and all I can say is that if you ever need to handle signals (SIGTERM, SIGHUP, etc…) from within a blocking select call in php, it looks like declare ticks is the only option for now.

I did the initial tests in 5.1.6, but can confirm the same behavior in 5.2.5. I don’t know how the behavior is going to be in 5.3, since I don’t run alpha releases on my servers but my gut likes to think that it will continue to work the same for now… and will hopefully not break until 6.0 (when everything else will explode for a few years). Shrug.

Originally, the files in question never exceeded more than a few thousand lines. Unfortunately, I am encountering cases now where the files are now occasionally 50,000 lines or longer. This causes PHP’s memory consumption to explode.

Note: Code snippets provided here are not fully functional standalone shell scripts. The scripts I ran to benchmark the algorithms contain some rudimentary setup logic that is not important here, so has not been included.

My original method:

// tail-file.php
$arr = @file( $fname, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$arr = array_slice($arr, -$lines);
$buf = implode("\\\n",$arr);

This is easy to understand and is pretty fast, all things considered. Unfortunately, the memory footprint for loading a file into an array is obscene. Loading a 4400 line log file with this method could consume more than 17mb of ram. 50,000 line files easily stressed the 256mb limit I am able to provide the process.

So, the obvious solution to the memory consumption is to avoid loading the entire file at once. What if we kept a rotating list of lines in the file?

// tail-array.php
$arr = array_fill( 0, $lines+1, "\\\n" );

$fp = fopen($fname, "r");
while( !feof($fp) ) {
    $line = fgets($fp, 4096);
    $arr[] = $line; // faster than array_push()
    array_shift($arr);
}
fclose($fp);
$buf = implode("",$arr);

This method works by keeping the $lines-many most recent lines of the file in an array. Memory consumption remains sane, but the performance hit for performing so many array pushes and shifts is bad. Really bad. With small files, I can’t notice any difference between this method and the file() method… but with longer files, it adds up quickly.

Given a 51 line, 4kb file, an average execution ($lines = 20) might look like this:

ammon@zapp:~$ time ./tail-file.php a.log >/dev/null

real    0m0.015s
user    0m0.009s
sys     0m0.007s

ammon@zapp:~$ time ./tail-array.php a.log >/dev/null

real    0m0.016s
user    0m0.010s
sys     0m0.006s

Comparable enough. But given a 50,004 line (3.3mb) log file:

ammon@zapp:~$ time ./tail-file.php b.log >/dev/null                  

real    0m0.079s
user    0m0.058s
sys     0m0.021s

ammon@zapp:~$ time ./tail-array.php b.log >/dev/null                 

real    0m0.119s
user    0m0.112s
sys     0m0.007s

The difference becomes quite clear. However… what if my log file grows obscenely large? I’ve got a 9 million line log file (1.6gb) lying around to test with…

ammon@zapp:~$ time ./tail-file.php c.log >/dev/null

real    0m0.015s
user    0m0.008s
sys     0m0.008s

ammon@zapp:~$ time ./tail-array.php c.log >/dev/null                 

real    0m19.351s
user    0m18.545s
sys     0m0.803s

The file() method crashes because it can’t allocate enough ram to hold a 9 million element array and the array method takes almost 20 seconds to execute. It’s slow… but at least it works.

Of course, there are other methods. The one I finally settled on is this:

// tail-seek.php
$fp = fopen($fname, "r");
$lines_read = 0;
if( $fp !== FALSE ) {
    fseek( $fp, 0, SEEK_END );
    $pos = $eof = ftell($fp);
    do {
        --$pos;
        fseek($fp, $pos);
        $c = fgetc($fp);
        if( $c == "\\\n" )
            $lines_read++;
    } while( $pos > 0 && $lines_read <= $lines );
    $buf = fread($fp, $eof-$pos);
}
fclose($fp);

This method doesn’t waste time reading the bulk of the file. It jumps to the end and scans backward until enough newlines have been located. The only problem here is that your average filesystem isn’t optimized for reading backwards… but since we’re not really reading very much data, it doesn’t much matter.

ammon@zapp:~$ time ./tail-seek.php a.log >/dev/null

real    0m0.017s
user    0m0.009s
sys     0m0.008s

ammon@zapp:~$ time ./tail-seek.php b.log >/dev/null                  

real    0m0.017s
user    0m0.008s
sys     0m0.010s

ammon@zapp:~$ time ./tail-seek.php c.log >/dev/null                  

real    0m0.023s
user    0m0.015s
sys     0m0.008s

Performance is a trifle slower on small files, but it’s astronomically better on long ones. This is similar to the method used by most unix ‘tail’ commands, and is the clear winner for actual use in my application.

Of course, it needs a bit of cleanup from the state I’ve provided it in, and isn’t appropriate for all environments… but it’s a trifle better than requiring 20 seconds and 20gb of ram to execute

Sometimes it’s the spaces vs tabs issue. Sometimes it’s file line endings (silly Windows-only editors and their CRLF). And sometimes it’s just people adjusting whitespace arbitrarily on lines (like adding spaces around parens or leaving spaces at the end of lines, etc…).

Regardless of the individual manifestation, it’s a silly problem, but one that causes more than its share of tears among developers everywhere.

Perhaps the easiest and smartest solution is to browbeat your co-developers into compliance. Force people to use editors that preserve line endings, force them to strip trailing whitespace and conform to a universal standard of indentation, etc… but it’s not always the nicest or most reliable solution. People will make mistakes, even if it’s only once a month… going over that diff might cost you an hour to figure out what had actually changed.

There are a few other solutions out there. They’re not new, and they’re not for everyone… but they can be phenomenally helpful at times. I’ll go over the two simplest ones.

dos2unix

Ever gotten a diff that reads like this?

ammon@binky:~/test$ svn diff one
Index: one
===================================================================
--- one (revision 2)
+++ one (working copy)
@@ -1,11 +1,11 @@
-One is the loneliest number that you'll ever do
-Two can be as bad as one
-It's the loneliest number since the number one
-
-No is the saddest experience you'll ever know
-Yes, it's the saddest experience you'll ever know
-`Cause one is the loneliest number that you'll ever do
-One is the loneliest number, worse than two
-
-It's just no good anymore since she went away
-Now I spend my time just making rhymes of yesterday
+One is the loneliest number that you'll ever do
+Two can be as bad as one
+It's the loneliest number since the number one
+
+No is the saddest experience you'll ever know
+Yes, it's the saddest experience you'll ever know
+'cause one is the loneliest number that you'll ever do
+One is the loneliest number, worse than two
+
+It's just no good anymore since she went away
+Now I spend my time just making rhymes of yesterday

This is what happens when something changes the line endings of a file. In this case, the original file was created with LF endings and was then edited slightly by an application that converted them to CRLF.

Now… if this were a 1000 line perl script in stead of an 11 line lyrics snippet… it would be soulcrushingly difficult to find the one actual change in the file.

Most unix distros have at their disposal the dos2unix / unix2dos utilities. On Red Hat, you can yum install dos2unix to get them. On Debian/Ubuntu, you can apt-get install tofrodos. I don’t have any other unices lying around at present to check on, but you can always just get the source at http://www.thefreecountry.com/tofrodos.

ammon@binky:~/test$ dos2unix one
ammon@binky:~/test$ svn diff one
Index: one
===================================================================
--- one (revision 2)
+++ one (working copy)
@@ -4,7 +4,7 @@

 No is the saddest experience you'll ever know
 Yes, it's the saddest experience you'll ever know
-`Cause one is the loneliest number that you'll ever do
+`cause one is the loneliest number that you'll ever do
 One is the loneliest number, worse than two

 It's just no good anymore since she went away

Much easier to figure out what has changed this way.

For extra credit, look into the svn:eol-style property. Set this on files as you commit them – or just use autoprops to do the dirty work for you…

diff-cmd

Of course, some times it’s not line endings. Sometimes the problem is random meaningless whitespace changes. Maybe somebody used an editor that auto-indents with spaces when the file was already indented with tabs, etc…

Subversion allows you to specify an alternate command to use to generate your diffs (in stead of relying on svn’s internal diff generation).

ammon@binky:~/test$ svn diff ----diff-cmd /usr/bin/diff -x -w one
Index: one
===================================================================
7c7
< `Cause one is the loneliest number that you'll ever do
---
> `cause one is the loneliest number that you'll ever do

But what if (for some bizarre reason) you don’t care about the case of letters?

ammon@binky:~/test$ svn diff ----diff-cmd /usr/bin/diff -x -iw one
Index: one
===================================================================

If you always want to use your custom diff utility you can set it in your runtime config to save yourself the hassle of having to type it manually each time.

For those using TortoiseSVN, you can always just specify graphical diff/merge utils to use in stead of Tortoise’s builtin ones. Personally, I’m a big fan of WinMerge, but there are several other good ones out there.

Yes, I know you can do this via cURL. This was explicitly written for a case when that particular luxury was not available to me.

So… it’s been a while since I’ve written a post, and I’m feeling the withdrawals. I’ve got 16 half-written posts waiting in the queue and two more that I want to write but haven’t even started. It’s also time for another round of anime reviews; I’ve got two video games I want to review…

And I am having to play the “too busy with RL” card.

In stead, I present a function that I wrote this morning. It’s been done countless times before, and you can probably find incredibly similar code out there already, but it’s the first time I’ve ever actually needed something quite like this.

/**
 * Send post data somewhere 
 *
 * @param $host The domain name of the host to send the data to.
 * @param $path The path of the script to receive the post.
 * @param $data An array of key/value pairs to send.
 * @return       The HTTP response, headers and all.
 */
function http_post( $host, $path, $data ) {
	// build a query string from the data array
	$arr = array();
	foreach( $data as $key => $val )
		array_push( $arr, "$key=".urlencode($val) );
	$data = implode( "&", $arr );
	// send that post
	$fh = fsockopen( $host, 80 );
	fwrite( $fh, "POST $path HTTP/1.1\\\r\\\n" );
	fwrite( $fh, "Host: $host\\\r\\\n" );
	fwrite( $fh, "Content-type: application/x-www-form-urlencoded\\\r\\\n" );
	fwrite( $fh, "Content-length: ".strlen($data)."\\\r\\\n" );
	fwrite( $fh, "Connection: close\\\r\\\n\\\r\\\n" );
	fwrite( $fh, $data );
	// get the response
	while( !feof($fh) )
		$buf .= fgets( $fh );
	fclose( $fh );
	return $buf;
}// end: http_post

This method could easily be modified to send GET data or to talk over a different port. It could also probably actually do something with the response buffer (check for 200, etc…). It could be a bit more fault tolerant, etc… but that’s not what I need for the application at hand.

For other examples of how people have done similar/related things, take a look at the comments on the fsockopen() documentation page.