Untitled » php

dead simple traditional style rotation

Ammon — Fri, 05 Feb 2010 17:51:26 +0000

In response to my two-step rotation post earlier this week, I figure I may as well share the logic I use for a more traditional logfile rotation scheme.

I think this is as simple as I can possibly make it:

= MAX_COPIES ) {
    $fname = array_shift($copies);
    trace( "deleting ".$fname );
}
$next = count($copies);
while( $fname = array_shift($copies) ) {
    --$next;
    trace( "rotating $fname -> $next" );
    rename( $fname, "$back_fname.$next" );
}

trace( "creating $back_fname" );
touch( $back_fname );
?>

A sample series of executions might look like this:

ammon@wernstrom:/path/to/log/file$ touch abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
- rotating /path/to/log/file/abc.log -> 0
- creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
- rotating /path/to/log/file/abc.log.0 -> 1
- rotating /path/to/log/file/abc.log -> 0
- creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
- rotating /path/to/log/file/abc.log.1 -> 2
- rotating /path/to/log/file/abc.log.0 -> 1
- rotating /path/to/log/file/abc.log -> 0
- creating /path/to/log/file/abc.log
ammon@wernstrom:/path/to/log/file$ php rotate.php
- deleting /path/to/log/file/abc.log.2
- rotating /path/to/log/file/abc.log.1 -> 2
- rotating /path/to/log/file/abc.log.0 -> 1
- rotating /path/to/log/file/abc.log -> 0
- creating /path/to/log/file/abc.log

This doesn’t have any failsafes, doesn’t compress anything, depends on an external call to ‘ls’, and it actually deletes old files in stead of overwriting them… but it is the shortest, simplest method I’ve come up with to get the job done.

If I feel like making this a full-fledged series, I might actually post a more thorough implementation later

simple two-step logfile rotation

Ammon — Wed, 03 Feb 2010 19:23:08 +0000

This is the result of 10 minutes of pounding on the keyboard after yet another disappointing experience with trying to get logrotate to do something vaguely more flexible.

This simple script scans all normal files in a log directory, and if they are older than a certain cutoff, moves them into a holding directory for old logs. Future passes will check files in the old directory for another age setting and will delete them. That’s all there is to it.

Configure your cutoffs, directories of interest, and optionally plug in a better logging mechanism and you’re set. (Oh, and change the #! if necessary, of course).

#!/usr/bin/php

lazy php profiler

Ammon — Wed, 13 Jan 2010 01:06:56 +0000

Caveman profiling with a side of “where were you at 9pm on the night in question?” As always, season to taste.

Sample use:

method();
    }
}

$obj = new obj();

// actual invocation cases
_profile();
func();
obj::method();
$obj->method();
$obj->other_method();
$obj->indirect_method();
?>

Output:

ammon@elzar:~$ cat /tmp/php-profile.log
[19:00:21] /home/ammon/test.php:20, _profile
[19:00:21] /home/ammon/test.php:21, func
[19:00:21] /home/ammon/test.php:22, obj::method
[19:00:21] /home/ammon/test.php:24, obj->method
[19:00:21] /home/ammon/test.php:13, func
[19:00:21] /home/ammon/test.php:16, obj->method

gearman 0.3 php extension api

Ammon — Tue, 26 May 2009 21:10:17 +0000

No real preamble to be made here. Gearman is a distributed job queuing system by the fine folks who brought us memcached. It is nicer than anything else I’ve looked at. I am attempting to switch one of my projects over to it (replacing a crufty curl + unix sockets + memcached monstrosity that attempted to do the same job).

The documentation is lacking, but if the discussion group is any indication, real docs are a high priority for the project team. Today, I visited the IRC channel to ask for a status update on docs for the PHP extension api (as opposed to the PEAR all-script api, whose auto-generated docs are broken). Turns out my suspicions were right. Documentation is a high priority and none currently exists for the api in question. However… I was informed that the classes support reflection… so

A quick grep of the source for the extension tells me that I am looking at four classes: GearmanClient, GearmanWorker, GearmanJob, and GearmanTask. A ridiculously short php script later…

And I can at least try to make a human readable list of available methods.

GearmanWorker

__construct()
clone()
error()
returnCode()
setOptions( $option, $data )
addServer( $host, $port ) – both args optional, examples say defaults are localhost on port 4730.
addFunction( $function_name, $function, $data, $timeout ) – data and timeout optional
work()

GearmanClient

__construct()
clone()
error()
setOptions( $option, $data )
addServer( $host, $port ) – reflection says REQUIRED, however the provided examples and personal experience says otherwise
do( $function_name, $workload, $unique ) – unique is optional
doHigh( $function_name, $workload, $unique ) – unique is optional
doLow( $function_name, $workload, $unique ) – unique is optional
doJobHandle()
doStatus()
doBackground( $function_name, $workload, $unique ) – unique is optional
doHighBackground( $function_name, $workload, $unique ) – unique is optional
doLowBackground( $function_name, $workload, $unique ) – unique is optional
jobStatus( $job_handle )
echo( $workload )
addTask( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskHigh( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskLow( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskHighBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskLowBackground( $function_name, $workload, $data, $unique ) – data and unique are optional
addTaskStatus( $job_handle, $data ) – data is optional
setWorkloadCallback( $callback )
setCreatedCallback( $callback)
setClientCallback( $callback)
setWarningCallback( $callback)
setStatusCallback( $callback)
setCompleteCallback( $callback)
setExceptionCallback( $callback)
setFailCallback( $callback)
clearCallbacks()
data()
setData( $data )
runTasks()

GearmanJob

__construct()
returnCode()
workload()
workloadSize()
warning( $warning )
status( $numerator, $denominator )
handle()
unique()
data( $data )
complete( $result )
exception( $exception )
fail()
functionName()
setReturn( $gearman_return_t )

GearmanTask

__construct()
returnCode()
create()
free()
function()
uuid()
jobHandle()
isKnown()
isRunning()
taskNumerator()
taskDenominator()
data()
dataSize()
takeData( $task_object ) – optional
sendData( $data )
recvData( $data_len )

The extension also appears to expose all constants defined in the C api.

I have since added this to the official wiki – so there are at least SOME docs on the site now

php autoload

Ammon — Tue, 28 Oct 2008 23:06:40 +0000

As of version 5.0, PHP has had the ability to dynamically include required classes as needed – without requiring the developer to manually include all possible dependencies beforehand. This means that in cases where your code execution never touches 39 of the 40 classes in the project, it loads, parses, and runs that much faster.

There is a performance hit for actually having to call the __autoload() method, but if you’re in a situation where the hit for executing a few extra comparison calls is unacceptable… you probably aren’t developing in PHP in the first place

Almost all of the php I’ve written in the last 2-3 years uses autoloading, and it has probably saved me hundreds of hours of aggravation.

In most of my projects, the first line of any script or class usually looks something like this:

require_once "/var/www/common/lib.php";

Then lib.php usually reads something like this:

And that is all that is strictly required to make the magic happen. It is fast, it is easy to understand, it is easy to use. You can use require_once() or include_once() and there is very little meaningful difference.

I’ve looked around the net and found several other attempts at improving on this simple mechanism. But they invariably overcomplicate things. They attempt to recurse source directories, cache filename->class differences to the filesystem, and otherwise turn what should be a simple filesystem operation that the php environment supports natively into a mess of exception handling and wheel reinvention.

There are obviously theoretical instances where you might want to have more than the one require_once/include_once line… but I’ve honestly never encountered one myself.

I mean, you could try to throw an exception if the file didn’t exist or otherwise failed to load… but nothing will happen. Failure to instantiate a nonexistant class is a fatal error in PHP, and will be handled as such with or without you – preempting any attempt at throwing an exception.

The only thing you can add is a bit of extra diagnostics or maybe logging to a separate location.

Assume that we have a file ‘test.php’:

If autoload.php contains a simple simple autoload function that uses require_once(), and Frog.php doesn’t exist anywhere in your include path, the results will look something like this:

ammon@kif:~$ php test.php 

Warning: require_once(Frog.php): failed to open stream: No such file or directory in /home/ammon/autoload.php on line 3

Fatal error: require_once(): Failed opening required 'Frog.php' (include_path='.:/usr/share/php:/usr/share/pear') in /home/ammon/autoload.php on line 3

If we had used an include_once() call, the output is similar, but slightly more informative:

ammon@kif:~$ php test.php 

Warning: include_once(Frog.php): failed to open stream: No such file or directory in /home/ammon/autoload.php on line 3

Warning: include_once(): Failed opening 'Frog.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /home/ammon/autoload.php on line 3

Fatal error: Class 'Frog' not found in /home/ammon/test.php on line 4

So that’s probably a bit more useful in tracking down the error. Require calls don’t return anything – they throw a fatal error on failure. Include calls, however, return FALSE on failure and TRUE if the file is (or, in the case of include_once, has already been) successfully included. So you can include_once() and write to a separate logfile (or to the output stream…) if you need more information than the fatal error already provides you.

To those who insist on giving your classes and their containing files different names… umm. Wow.

If I have a class called DatabaseConnection, I’m going to put it in a file called DatabaseConnection.php. If I’m working with strange people who somehow don’t think that is explicit enough, I might call it DatabaseConnection.class.php and tweak the autoload method ever so slightly to compensate. There’s no good reason to put it in a file called projx-database_connection.incl or something. No. There isn’t.

If you want to organize your classes into a meaningful directory structure… good for you. Use PHP’s built-in include_path ini option. Don’t waste time trying to cascade down a directory structure searching for the classes – just make sure your includes are all in a set of reliable locations. You don’t actually have to edit the php.ini file and bounce Apache or your php-cgi processes, just define the additional include paths in the same file where you define your autoloader:

set_include_path(
    get_include_path() . PATH_SEPARATOR .
    "/var/www/includes" . PATH_SEPARATOR .
    "/var/www/includes/apple" . PATH_SEPARATOR .
    "/var/www/includes/banana"
);

Naturally, you could turn that into some function calls to dynamically register and unregister directories, etc… but at that point, you’re probably hurting yourself again. If your codebase is being reorganized enough to make maintenance of the list of include dirs onerous without full time intervention, something else has probably already gone very wrong. At best, the code probably doesn’t work anyway, so any brief delay in updating the list can’t hurt any more than whatever else is happening.

But seriously. __autoload() is your friend. It will help clean up your code if you let it. It can help enforce naming conventions. It can even improve performance… so long as you refrain from using it to shoot yourself in the foot.

php signals while selecting

Ammon — Mon, 27 Oct 2008 20:26:38 +0000

So a fairly longstanding gripe of mine has been that PHP fails to execute registered signal handlers when it receives a signal in the middle of a blocking select call. Today, I finally bumped into a situation where I couldn’t just change the spec to avoid the situation… and I’ve finally figured out how to make it work.

The bug has been reported here, where it was ignored for a few months before being shot down and ignored some more as per php dev team regulations.

Sample code given by the reporter of the bug is markedly similar to the situations I’ve encountered the problem:

pcntl_signal(SIGINT, "sig_handler");
$sock = socket_create_listen($port);
$read_socks = array($sock);
$n = NULL;
$foo = socket_select($read_socks, $n, $n, NULL);

By filling in his blanks, my first test case looks something like this:

When executing the script and pressing ^C (which sends SIGINT), the following occurs:

ammon@morbo:~$ php sigtest.php
PHP Warning:  socket_select(): unable to select [4]: Interrupted system call in /home/ammon/sigtest.php on line 13
select returned ''

Ok, so the warning is to be expected, and we can easily squelch that.

The real problem is that the signal handler never runs.

However… for the first time in my life, a response to a php bug report proves enlightening. The dev who answered this ticket provides his sample code and says he can’t duplicate the bug. Upon looking at the differences between their code, only one difference stands out:

declare(ticks=1);

The declare(ticks) directive is deprecated as of php 5.3 and will not be with us in php 6.0. Ticks are an unreliable, unpredictable, and generally bad thing in php. I’ve neither successfully used them nor seen a successful and justified use.

That being said… turning the tick on but not telling it to do anything appears to address the problem of discarded interrupts:

And execution:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

Which is precisely the desired behavior.

I don’t know what the performance hit for turning ticks on is, I haven’t had time to research this. But I can confirm that by declaring ticks globally, it does work in an OO environment as well:

start();
    }

    function sig_handler($signo) {
        echo "received sig #$signo\\\n";
    }

    function start() {
        $socket = socket_create_listen( 1234 );
        $r = array( $socket );
        $n = NULL;
        while( true ) {
            $foo = @socket_select( $r, $n, $n, NULL );
            echo "select returned '$foo'\\\n";
        }
    }
}

$test = new signal_tester();
?>

Executing and hitting ^C:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

After a few minutes of largely unscientific testing, it appears that turning ticks on globally costs a whopping 4 bytes of ram and causes the script to occasionally consume more cpu than the top process I used to monitor it. So… at first glance the cost is pretty negligible and all I can say is that if you ever need to handle signals (SIGTERM, SIGHUP, etc…) from within a blocking select call in php, it looks like declare ticks is the only option for now.

I did the initial tests in 5.1.6, but can confirm the same behavior in 5.2.5. I don’t know how the behavior is going to be in 5.3, since I don’t run alpha releases on my servers but my gut likes to think that it will continue to work the same for now… and will hopefully not break until 6.0 (when everything else will explode for a few years). Shrug.

php tail

Ammon — Tue, 27 May 2008 19:08:21 +0000

I have a php script that frequently needs to email me the last few lines of a log file. I can’t afford to exec() a binary tail process, so the solution has to be in pure php.

Originally, the files in question never exceeded more than a few thousand lines. Unfortunately, I am encountering cases now where the files are now occasionally 50,000 lines or longer. This causes PHP’s memory consumption to explode.

Note: Code snippets provided here are not fully functional standalone shell scripts. The scripts I ran to benchmark the algorithms contain some rudimentary setup logic that is not important here, so has not been included.

My original method:

// tail-file.php
$arr = @file( $fname, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$arr = array_slice($arr, -$lines);
$buf = implode("\\\n",$arr);

This is easy to understand and is pretty fast, all things considered. Unfortunately, the memory footprint for loading a file into an array is obscene. Loading a 4400 line log file with this method could consume more than 17mb of ram. 50,000 line files easily stressed the 256mb limit I am able to provide the process.

So, the obvious solution to the memory consumption is to avoid loading the entire file at once. What if we kept a rotating list of lines in the file?

// tail-array.php
$arr = array_fill( 0, $lines+1, "\\\n" );

$fp = fopen($fname, "r");
while( !feof($fp) ) {
    $line = fgets($fp, 4096);
    $arr[] = $line; // faster than array_push()
    array_shift($arr);
}
fclose($fp);
$buf = implode("",$arr);

This method works by keeping the $lines-many most recent lines of the file in an array. Memory consumption remains sane, but the performance hit for performing so many array pushes and shifts is bad. Really bad. With small files, I can’t notice any difference between this method and the file() method… but with longer files, it adds up quickly.

Given a 51 line, 4kb file, an average execution ($lines = 20) might look like this:

ammon@zapp:~$ time ./tail-file.php a.log >/dev/null

real    0m0.015s
user    0m0.009s
sys     0m0.007s

ammon@zapp:~$ time ./tail-array.php a.log >/dev/null

real    0m0.016s
user    0m0.010s
sys     0m0.006s

Comparable enough. But given a 50,004 line (3.3mb) log file:

ammon@zapp:~$ time ./tail-file.php b.log >/dev/null                  

real    0m0.079s
user    0m0.058s
sys     0m0.021s

ammon@zapp:~$ time ./tail-array.php b.log >/dev/null                 

real    0m0.119s
user    0m0.112s
sys     0m0.007s

The difference becomes quite clear. However… what if my log file grows obscenely large? I’ve got a 9 million line log file (1.6gb) lying around to test with…

ammon@zapp:~$ time ./tail-file.php c.log >/dev/null

real    0m0.015s
user    0m0.008s
sys     0m0.008s

ammon@zapp:~$ time ./tail-array.php c.log >/dev/null                 

real    0m19.351s
user    0m18.545s
sys     0m0.803s

The file() method crashes because it can’t allocate enough ram to hold a 9 million element array and the array method takes almost 20 seconds to execute. It’s slow… but at least it works.

Of course, there are other methods. The one I finally settled on is this:

// tail-seek.php
$fp = fopen($fname, "r");
$lines_read = 0;
if( $fp !== FALSE ) {
    fseek( $fp, 0, SEEK_END );
    $pos = $eof = ftell($fp);
    do {
        --$pos;
        fseek($fp, $pos);
        $c = fgetc($fp);
        if( $c == "\\\n" )
            $lines_read++;
    } while( $pos > 0 && $lines_read <= $lines );
    $buf = fread($fp, $eof-$pos);
}
fclose($fp);

This method doesn’t waste time reading the bulk of the file. It jumps to the end and scans backward until enough newlines have been located. The only problem here is that your average filesystem isn’t optimized for reading backwards… but since we’re not really reading very much data, it doesn’t much matter.

ammon@zapp:~$ time ./tail-seek.php a.log >/dev/null

real    0m0.017s
user    0m0.009s
sys     0m0.008s

ammon@zapp:~$ time ./tail-seek.php b.log >/dev/null                  

real    0m0.017s
user    0m0.008s
sys     0m0.010s

ammon@zapp:~$ time ./tail-seek.php c.log >/dev/null                  

real    0m0.023s
user    0m0.015s
sys     0m0.008s

Performance is a trifle slower on small files, but it’s astronomically better on long ones. This is similar to the method used by most unix ‘tail’ commands, and is the clear winner for actual use in my application.

Of course, it needs a bit of cleanup from the state I’ve provided it in, and isn’t appropriate for all environments… but it’s a trifle better than requiring 20 seconds and 20gb of ram to execute

flash policy service daemon

Ammon — Tue, 22 Apr 2008 09:05:10 +0000

Sorry it took me so long to post this, but Wordpress 2.5 doesn’t seem to like me trying to upload gz/zip files, so I had to upload the source manually.

Well, it’s been months since I promised to post some usable socket policy service code, so I will.

The script here is meant to serve as a good starting point for people whose servers need to allow flash clients to make socket connections. I have not actually used this exact code in a production environment, but I have been using code that is 99% identical for a while now. I am confident that any blatant flaws are the result of simple copy-paste errors as I compiled the package. Please let me know if you find any.

I have however, stress tested the heck out of this service. One instance successfully served up over 16000 policy file requests fed into it as rapidly as I could send them. The same networking code has also handled requests from at least 100 different hosts at roughly the same time.

Everything has been combined into a single cli php script that requires no special installation. Just plop it down on the server and run it as root. It will take care of the rest. The config defaults should be safe, but you probably want to specify them more clearly – just to be safe.

The daemon is made of three classes:

Logger – A rudimentary log file management class that I copy from project to project in one form or another. The included version is stripped down from some of the other versions I’ve written, and I’m planning on releasing a more feature-rich version in the future.
Daemon – A simple class for daemonizing a process. Adapted and re-adapted countless times from an original php4 class I found on the net a few years ago by some guy named Seth (whose email domain no longer exists).
FlashPolicyService – The meat and potatoes, a child of Daemon. Mostly, this is just the requisite networking code and glue to make everything work together.

As with any of my other code, this is licensed under CC Attribution 3.0.

Download:

FlashPolicyService-09c.zip (4.4kb)

Source code after the jump.

#!/usr/local/bin/php
 on port 843 and serves up
 * an xml crossdomain policy file. This sort of service is necessary if any
 * flash content is going to connect to sockets on the running host.
 *
 * I have made every effort to package everything you need into a single
 * file here, even though it could easily have been split into 3 or more
 * scripts.
 *
 * Requirements:
 *  - PHP 5 with sockets, pcntl, and posix extensions
 *  - Root access (to bind to a <1024 port)
 *
 * Use:
 *  Simply execute the script from the command line as root:
 *    # ./FlashPolicyService.php
 *  You can enable debug mode by invoking the script with '-d' as a parameter:
 *    # ./FlashPolicyService.php -d
 *  To stop the daemon, simply send it a SIGTERM and it should attempt to
 *  exit cleanly.
 *
 *  If you get 'bad interpreter' errors or the like, change the #! line to
 *  reflect the actual installed location of your php cli.
 *
 * Configuration:
 *  At present, there aren't very many config options. Simply edit the
 *  section immediately following this header. Options are commented.
 *
 * License:
 *  This code is made available under a Creative Commons Attribution 3.0
 *  License. Basically, you can use it however you like, but I would
 *  appreciate some credit when you do.
 *
 * Disclaimer:
 *  I make no guarantees that this code won't make your server explode in a
 *  shower of blue flames. However, I don't expect that it will. I am actually
 *  confident that this code will be helpful.
 *
 *  That said, this is still my beta release. If you find any bugs, please
 *  let me know so I can fix them.
 *
 * - Ammon Lauritzen [Apr 21, '08]
 *   http://ammonlauritzen.com/blog/2008/04/22/flash-policy-service-daemon/
 *
 * Changelog
 * 0.9.c
 *   - Fixed some typoes that were the result of how I combined everything
 *     into a single file. Thanks Alex!
 * 0.9.b
 *   - Original version to be posted at this url.
 * 0.9.a
 *   - Original proof of concept code posted online.
 */

/*** Config ***/

// where should we save the log output?
$log_filename = "/tmp/flash-policy.log";

// uncomment these lines to choose which user to run the daemon as
// default behavior is to look up and attempt to run as 'nobody'
# $daemon_uid = 99;
# $daemon_gid = 99;

// set this if you want to use an external file in stead of the default
$xml_filename = "";
$default_xml =
	'<'.'?xml version="1.0" encoding="UTF-8"?'.'>'.
    ''.
        ''.
        ''.
    '';

/*** You shouldn't have to edit anything below here ***/

/////////////////////////////////////////////////////////////////////////////
class Logger {
	public function __construct( $logfile ) {
        $this->logfile = $logfile;
        $this->open_log();
    }// end: constructor

    public function __destruct() {
        @fflush( $this->fh );
        @fclose( $this->fh );
    }// end: destructor

    public function log( $msg ) {
		// deal with redundant log spam
		if( $msg == $this->last_msg ) {
			$this->last_msg_count++;
			return;
		} else if( $this->last_msg_count ) {
			$this->write( "Last message repeated ".$this->last_msg_count." times." );
		}
		$this->lasg_msg = $msg;
		$this->last_msg_count = 0;

		// actually write the log message out
		$this->write( $msg );
	}// end: log

	private function write( $msg ) {
        $msg = sprintf("[%s] %s\\n",date("y-m-d H:i:s"),$msg);
        $succ = @fwrite( $this->fh, $msg );
        if( $succ === FALSE ) {
            echo $msg;
        }
    }// end: write

	private function open_log() {
        if( !file_exists($this->logfile) ) {
            touch( $this->logfile );
            chmod( $this->logfile, 0664 );
        }
        $this->fh = @fopen( $this->logfile, "a" );
    }
}// end: logger class

/////////////////////////////////////////////////////////////////////////////
class Daemon {
	public function __construct() {
		error_reporting(0);
		set_time_limit(0);

		global $log_filename, $pid_filename;
		$this->logger = new Logger( $log_filename );
		$this->pid_filename = $pid_filename;

		$this->log_tag = $this->daemon_tag = "launcher";
		$this->debug( "constructed", $this->daemon_tag );
	}// end: constructor

	public function __destruct() {
		$this->debug( "destructing", $this->daemon_tag );
	}// end: destructor

	protected function log( $msg, $tag = false ) {
		if( $tag == false )
			$tag = $this->log_tag;
		$this->logger->log( $tag.": ".$msg );
	}// end: log
	protected function debug( $msg, $tag = false ) {
		if( DEBUG )
			$this->log( $msg, $tag );
	}// end: debug

	protected function main() {
		$this->log( "override main() in daemon subclass", $this->daemon_tag );
		$this->stop();
	}// end: main

	public function start() {
		$this->log( "starting daemon", $this->daemon_tag );
		if( !$this->_start() ) {
			$this->log( "unable to start daemon", $this->daemon_tag );
			return;
		}

		// report execution details
		$this->log( "uid = ".posix_getuid().", gid = ".posix_getgid() );
		$this->log( "cwd = ".getcwd() );

		// invoke main loop
		$this->running = true;
		while( $this->running ) {
			$this->main();
		}
	}// end: start
	private function _start() {
		if( !$this->_fork() ) {
			return false;
		}// end: try to fork

		if( !posix_setsid() ) {
			$this->log( "unable to setsid()", $this->daemon_tag );
			return false;
		}// end: try to set sid

		if( !$this->_suid() ) {
			return false;
		}// end: try to set uid

		// register signal handler
		declare(ticks = 1);
		pcntl_signal( SIGTERM, array(&$this, "on_sigterm") );
		// chdir somewhere moderately safe by default
		chdir( '/tmp' );
		return true;
	}// end: _start
	private function _fork() {
		$this->log( "forking...", $this->daemon_tag );
		$pid = pcntl_fork();
		if( $pid == -1 ) {
			// error
			$this->log( "unable to fork", $this->daemon_tag );
			return false;
		} else if( $pid ) {
			// parent
			$this->debug( "done with parent", $this->daemon_tag );
			exit( 0 );
		} else {
			// child
			$this->daemon_tag = "daemon";
			$this->child = true;
			$this->pid = posix_getpid();
			$this->debug( "child pid = ".$this->pid );
			return true;
		}
	}// end: _fork
	private function _suid() {
		global $daemon_uid, $daemon_gid;
		if( !isset($daemon_uid) || !isset($daemon_gid) ) {
			// we didn't get a uid/gid, try to make one up
			$this->debug( "searching for info on 'nobody'", $this->daemon_tag );
			$res = posix_getpwnam("nobody");
			if( $res !== FALSE ) {
				$uid = $res['uid'];
				$gid = $res['gid'];
			} else {
				// the 'nobody' user doesn't exist on this system, refuse
				// to daemonize as root
				$this->log( "unable to find info on 'nobody' user", $this->daemon_tag );
				return false;
			}
		} else {
			$this->debug( "got uid/gid of $daemon_uid/$daemon_gid", $this->daemon_tag );
			$uid = $daemon_uid;
			$gid = $daemon_gid;
		}
		// actually try to switch now
		if( !posix_setgid($gid) ) {
			$this->log( "unable to set gid to ".$gid, $this->daemon_tag );
			return false;
		} else if( !posix_setuid($uid) ) {
			$this->log( "unable to set uid to ".$uid, $this->daemon_tag );
			return false;
		} else {
			return true;
		}
	}// end: _suid

	public function stop() {
		$this->log( "stopping daemon", $this->daemon_tag );
		$this->running = false;
	}// end: stop

	protected function on_sigterm( $sig ) {
		if( $sig == SIGTERM ) {
			$this->log( "got SIGTERM", $this->daemon_tag );
			$this->stop();
			exit( 0 );
		}
	}// end: on_sigterm
}// end: daemon class

/////////////////////////////////////////////////////////////////////////////
class FlashPolicyService extends Daemon {
	public function __construct() {
		parent::__construct();
		$this->log_tag = "fps";
		$this->debug( "constructing" );

		$this->connections = array();
		$this->request_str = "";
		$this->port = 843;

		// get our xml
		global $xml_filename, $default_xml;
		if( strlen($xml_filename) == 0 || !file_exists($xml_filename) ) {
			$this->log( "unable to read xml from '$xml_filename', using default" );
			$this->xml = $default_xml;
		} else {
			$this->log( "reading policy xml from $xml_filename" );
			$this->xml = file_get_contents( $xml_filename );
		}
		$this->debug( "policy xml: ".$this->xml );
		// make sure we're null terminated
		$this->xml = trim($this->xml)."\\n\\0";

		// and get going
		$this->init();
	}// end: constructor

	public function __destruct() {
		parent::__destruct();
		if( $this->sock )
			@fclose($this->sock);
	}// end: destructor

	private function init() {
		$this->debug( "init..." );
		if( $this->check_socket() ) {
			parent::start();
		} else {
			$this->log( "not starting daemon" );
		}
	}// end: init

	protected function main() {
		if( $this->sock ) {
			while( true ) {
				$this->accept_socket();
			}
		} else {
			$this->log( "server socket is closed?!" );
			parent::stop();
		}
		// paranoia to keep from absolutely hosing cpu if something goes wrong
		usleep( 10000 );	// 10ms
	}// end: main

	private function check_socket() {
		$this->sock = @socket_create( AF_INET, SOCK_STREAM, SOL_TCP );
		if( !$this->sock ) {
			$this->log( "unable to create socket" );
		} else {
			$succ = @socket_bind( $this->sock, "0.0.0.0", $this->port );
			if( !$succ ) {
				$this->log( "unable to bind to port ".$this->port );
			} else {
				$backlog = 100;
				$succ = @socket_listen( $this->sock, $backlog );
				if( !$succ ) {
					$this->log( "unable to listen with backlog of $backlog" );
				} else {
					// everything's good
					return true;
				}
			}// end: able to bind
		}// end: able to create socket

		// if we got here, it's an error. abort.
		$errno = socket_last_error( $this->sock );
		$errstr = socket_strerror( $errno );
		$this->log( "socket error: $errno: $errstr" );
		return false;
	}// end: check_socket

	private function accept_socket() {
		$r_socks = array_merge( array($this->sock), $this->connections );
		$this->debug( "selecting on ".count($r_socks)." sockets" );

		// block until something interesting happens
		$select = @socket_select( $r_socks, $w_socks = NULL, $e_socks = NULL, NULL );
		if( !$select )
			return;

		// did we get a new connection?
		if( in_array($this->sock, $r_socks) ) {
			$conn = @socket_accept( $this->sock );
			if( $conn !== false )
				@socket_getpeername( $conn, $addr );
			$this->log( "connection accepted from $addr, $conn" );
			$this->connections[] = $conn;
		}// end: got a new connection

		// check for policy requests
		foreach( $r_socks as $conn ) {
			// ignore the server socket
			if( $conn == $this->sock )
				continue;

			// read from the client
			$data = @socket_read( $conn, 1024 );
			if( $data === FALSE ) {
				$this->log( "got disconnect from $conn" );
			}// end: client closed connection
			else {
				$this->debug( "read '".trim($data)."' from $conn" );
				if( strpos($data, $this->request_str) !== FALSE ) {
					$this->log( "sending policy xml to $conn" );
					@socket_write( $conn, $this->xml );
				} else {
					$this->log( "got invalid request from $conn" );
				}
			}// end: got data from the client

			// and always disconnect after having read something, whether
			// it was a valid request or not - especially if it wasn't 
			@socket_close( $conn );
			$key = array_search( $conn, $this->connections );
			if( $key !== FALSE )
				unset( $this->connections[$key] );
		}// end: foreach socket
	}// end: accept_socket

}// end: flash policy service class

/////////////////////////////////////////////////////////////////////////////
/**
 * Actual execution code here. This checks if the php install has all of our
 * requisite extensions, makes sure we're launching as root, and checks if
 * debug mode was requested before actually starting the daemon.
 */

// make sure we have required extensions
$required_extensions = array( "sockets", "posix", "pcntl" );
$missing_extension = false;
foreach( $required_extensions as $ext ) {
	if( !extension_loaded($ext) ) {
		echo "Missing required php extension '$ext'.\n";
		$missing_extension = true;
	}
}
if( $missing_extension ) {
	exit( 1 );
}

// make sure we launch as root, otherwise we can't bind to 843
if( posix_getuid() != 0 || posix_getgid() != 0 ) {
	echo "Policy service must be started as root.\n";
	exit( 1 );
}

// see if we're in debug mode
define( "DEBUG", in_array('-d',$argv) );

// start the daemon
$fps = new FlashPolicyService();
?>

lazy image browser

Ammon — Fri, 21 Mar 2008 17:26:50 +0000

The other day, I heard a few people talking about needing an easy way to browse images on a remote Apache server that has Indexes disabled.

They had a ~20 line php script that they were dropping into each directory in order to generate indices. The problem came when they started organizing the images into subdirectories. Eventually, it became necessary to copy the new script into a mind-bogglingly large number of directories. Inevitably, dirs were missed, etc…

I interjected that I could probably fix their problem in 30 minutes.

So I did.

$subdir\\\n";
$dirs = "";
$imgs = "\\\n";

if( file_exists($dir) && is_dir($dir) ) {
	$dh = opendir($dir);
	while( false !== ($file = readdir($dh)) ) {
		if( $file == "." || $file == ".." || $file == ".svn" || substr($file,-4) == ".php" )
			continue;
		if( is_dir("$base/$subdir/$file") ) {
			$dirs .= "|$file|\\\n";
		} else {
			$imgs .= "\\\n";
		}
	}
}

echo $dirs;
echo $imgs;
?>

It’s not elegant. It’s not pretty. It has plenty of room for improvement – it’ll generate links to Windows explorer thumbnail db’s, etc… But it is fast and should be moderately secure. Just drop it in the root directory of your image structure and you’re good.

new flash security policies

Ammon — Thu, 13 Dec 2007 22:14:20 +0000

So… I am not happy with Adobe right now. With the push of Flash Player 9,0,115,0 “moviestar”, which included such awesome features as H.264 and AAC codec support and improvements to fullscreen mode, they kind of ambushed me with some sweeping changes to their security policy.

I’d been running pre-release nightly builds of the player since 9,0,60,x… and had noticed some strange warnings. Mysterious “Socket Security Error #2048″ exceptions that were being thrown at random – even though I was serving an appropriate (for the time) crossdomain.xml file, unexplained timeouts attempting to talk to an xml socket server when I was very clearly not attempting to do any such thing, etc… My regularly repeated attempts to find documentation on what the warnings actually meant proved fruitless. I believe that is because the appropriate document was not actually released to the public until 9,0,115,0 was released.

Now, the bit where they improved the format for crossdomain.xml files doesn’t really affect me one way or the other. I approve of the improvements but could really care less in this case. They don’t really affect anything I’m doing.

The part that really chaps my hide is the fact that they’ve completely redone the way that socket security policies are handled. The important parts:

A SWF file may no longer make a socket connection to its own domain without a socket policy file. Prior to version 9,0,115,0, a SWF file was permitted to make socket connections to ports 1024 or greater in its own domain without a policy file.

HTTP policy files may no longer be used to authorize socket connections. Prior to version 9,0,115,0, an HTTP policy file, served from the master location of /crossdomain.xml on port 80, could be used to authorize a socket connection to any port 1024 or greater on the same host.

That’s right. Your socket policy data can’t live in the sitewide crossdomain.xml file that Apache serves any more.

Flash Player 9,0,115,0 introduces a concept of socket master policy files, which are served from the fixed TCP port number 843.

Socket policy files may be obtained from the same port as a main connection (the socket connection being made by ActionScript, which is authorized by a socket policy file), or from a different port, separate from the main connection. If you opt to serve a socket policy file from the same port as a main connection, the server listening on that port must understand socket policy file requests (which are indicated by a transmission of from Flash Player), and must respond differently for policy file requests and main connection requests.

When a SWF file attempts to make a socket connection, even to its own domain, Flash Player will first attempt to contact port 843 to see if the host is serving a socket master policy file.

So… regardless of whether you’re even using a custom port 843 client, the Flash Player is going to try to hit it. What if your firewall doesn’t allow/route traffic to sub-1024 ports w/o special configuration? What if you don’t have the access to bind to a sub-1024 port and can’t rewrite your other server process to serve the policy data on its port?

Socket meta-policies can only be declared in a socket master policy file. The syntax is the same as for declaring a meta-policy in an URL master policy file, using the tag. Socket meta-policies cannot be declared in HTTP response headers, as HTTP is not involved.

This implies that you can’t even tell apache to listen to port 843 and serve up the data. You HAVE to either maintain a separate server process specifically for the purpose of serving this policy data, or you have to edit the process that SWF’s are connecting to and make them serve the data..

As of the time of this writing (10 days after moviestar’s release), they have yet to release promised help on how to deploy a solution to these new changes. Granted, the one article they did release explains what needs to be done in high level terms. It was sufficient to help me out. I wrote a server that simply listens on port 843 and spews the required xml. But… I’d have really appreciated specific examples, and I suspect plenty of people would appreciate drop-in solutions to the issue.

A 5-minute skeleton implementation (not recommended for production use by any means) written as a PHP cli script might look something like this:

#!/usr/bin/php
'.
	''.
        ''.
        ''.
	'';

// make sure everything launches correctly
if( posix_getuid() != 0 )
	die( "You must run this script as root.\\\n" );

$sock = @socket_create( AF_INET, SOCK_STREAM, SOL_TCP );
if( !$sock )
	die( "Unable to create socket.\\\n" );

$succ = @socket_bind( $sock, "0.0.0.0", 843 );
if( !$succ )
	die( "Unable to bind to port 843.\\\n" );

$succ = @socket_listen( $sock );
if( !$succ )
	die( "Unable to start listening.\\\n" );

// start serving policies
while( true ) {
	$r = $w = $e = array( $sock );
	if( @socket_select( $r, $w, $e, null ) !== false ) {
		$conn = @socket_accept( $sock );
		if( $conn !== false ) {
			// somebody connected, just dump the xml and close
			socket_write( $conn, $policy_file );
			socket_close( $conn );
		} else {
			echo "socket_accept() failed?\\\n";
			break;
		}
	} else {
		echo "socket_select() failed?\\\n";
		break;
	}
}// end: listen forever

// clean up
socket_close( $sock );
?>

I’ll try to make my production version of this a bit more suitable for public consumption and release it as soon as I can.

The random #2048 security errors continue, despite having deployed my port 843 policy xml server. Granted, they happen less than before… but they still happen. And even when my policy server isn’t running, the errors aren’t thrown 100% of the time. This just baffles me. If they were consistent, that would be one thing. But when you get a security error 1 time in 20… that’s not security, that’s not even a lame deterrent. It’s just incentive to hammer the same port over and over again until something finally gives.

Now, I admit that I could be wrong here… but I’ve re-re-read the documentation on these policies a few times now, and cannot find any reason for the behaviors I’m seeing.

update

On April 22nd, 2008, I released a much better, much more reliable version of this daemon. Head over there for more details and source code.