Last week, I needed to set up new replication of an 80gb database. This should have been routine by now, but when I attempted to prepare the backup this time, it whined and complained and failed. I was kind of frazzled by the time I gave up on the issue and declared it a fluke of one sort or another.

Last night, I tried again from Sunday’s full backup, and it happened again:

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=75546624, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 75546624
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818  2:54:34  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
090818  2:54:34  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
InnoDB: File name .//tmp/#sql6e1e_8cce1_0.ibd
InnoDB: File operation call: 'create'.
InnoDB: Cannot continue operation.

I gave up after poking a few things.

This morning’s fresh look turned up this bug report.

ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo mkdir tmp
ammon@amy:/var/lib/2009-08-16_04-02-17$ sudo xtrabackup --prepare --target-dir=.
xtrabackup  Ver 0.8.1rc Rev 78 for 5.0.83 unknown-linux-gnu (x86_64)
xtrabackup: cd to .
xtrabackup: This target seems to be not prepared yet.
090818 12:27:41  InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=84983808, start_lsn=(86 1293090752)
xtrabackup: Temporary instance for recovery is set as followings.
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:512M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 84983808
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Log scan progressed past the checkpoint lsn 86 1293090752
090818 12:27:41  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Doing recovery: scanned up to log sequence number 86 1298333184 (6 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1303576064 (13 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1308818944 (20 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1314061824 (27 %)
InnoDB: Doing recovery: scanned up to log sequence number 86 1319304704 (34 %)
090818 12:27:44  InnoDB: Starting an apply batch of log records to the database...
... snip ...

That’s right. There’s a bug in innodb restoration that interprets location of /tmp (configurable in my.cnf) to be relative in stead of absolute.

So, if you have problems while trying to restore from an xtrabackup/ibbackup snapshot (or if you’re trying to recover innodb after a crash), just creating the offending tmp directory appears to work.

The bug has been reported here, where it was ignored for a few months before being shot down and ignored some more as per php dev team regulations.

Sample code given by the reporter of the bug is markedly similar to the situations I’ve encountered the problem:

pcntl_signal(SIGINT, "sig_handler");
$sock = socket_create_listen($port);
$read_socks = array($sock);
$n = NULL;
$foo = socket_select($read_socks, $n, $n, NULL);

By filling in his blanks, my first test case looks something like this:

<?
function sig_handler($signo) {
        echo "received sig #$signo\\\n";
}
pcntl_signal( SIGINT, "sig_handler" );

$socket = socket_create_listen( 1234 );
$r = array( $socket );
$n = NULL;
while( true ) {
        $foo = socket_select( $r, $n, $n, NULL );
        echo "select returned '$foo'\\\n";
}
?>

When executing the script and pressing ^C (which sends SIGINT), the following occurs:

ammon@morbo:~$ php sigtest.php
PHP Warning:  socket_select(): unable to select [4]: Interrupted system call in /home/ammon/sigtest.php on line 13
select returned ''

Ok, so the warning is to be expected, and we can easily squelch that.

The real problem is that the signal handler never runs.

However… for the first time in my life, a response to a php bug report proves enlightening. The dev who answered this ticket provides his sample code and says he can’t duplicate the bug. Upon looking at the differences between their code, only one difference stands out:

declare(ticks=1);

The declare(ticks) directive is deprecated as of php 5.3 and will not be with us in php 6.0. Ticks are an unreliable, unpredictable, and generally bad thing in php. I’ve neither successfully used them nor seen a successful and justified use.

That being said… turning the tick on but not telling it to do anything appears to address the problem of discarded interrupts:

<?
declare(ticks=1);

function sig_handler($signo) {
        echo "received sig #$signo\\\n";
}
pcntl_signal( SIGINT, "sig_handler" );

$socket = socket_create_listen( 1234 );
$r = array( $socket );
$n = NULL;
while( true ) {
        $foo = @socket_select( $r, $n, $n, NULL );
        echo "select returned '$foo'\\\n";
}
?>

And execution:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

Which is precisely the desired behavior.

I don’t know what the performance hit for turning ticks on is, I haven’t had time to research this. But I can confirm that by declaring ticks globally, it does work in an OO environment as well:

<?
declare(ticks=1);

class signal_tester {
    function __construct() {
        pcntl_signal( SIGINT, array(&$this,"sig_handler") );
        $this->start();
    }

    function sig_handler($signo) {
        echo "received sig #$signo\\\n";
    }

    function start() {
        $socket = socket_create_listen( 1234 );
        $r = array( $socket );
        $n = NULL;
        while( true ) {
            $foo = @socket_select( $r, $n, $n, NULL );
            echo "select returned '$foo'\\\n";
        }
    }
}

$test = new signal_tester();
?>

Executing and hitting ^C:

ammon@morbo:~$ php sigtest.php
received sig #2
select returned ''

After a few minutes of largely unscientific testing, it appears that turning ticks on globally costs a whopping 4 bytes of ram and causes the script to occasionally consume more cpu than the top process I used to monitor it. So… at first glance the cost is pretty negligible and all I can say is that if you ever need to handle signals (SIGTERM, SIGHUP, etc…) from within a blocking select call in php, it looks like declare ticks is the only option for now.

I did the initial tests in 5.1.6, but can confirm the same behavior in 5.2.5. I don’t know how the behavior is going to be in 5.3, since I don’t run alpha releases on my servers but my gut likes to think that it will continue to work the same for now… and will hopefully not break until 6.0 (when everything else will explode for a few years). Shrug.

When you are setting up public-key authentication on OpenSSH, you must be very careful of file ownerships and permissions. In many stock unix setups, this isn’t a problem. But in any environment where you are relying on a lot of group access to files, it is easy to slip up and earn yourself a system that will silently fail to authenticate (unless you turn on debug level verbosity).

1. The private key must be readable only by the user initiating the connection.
2. The authorized_hosts file must be writable only by the account accepting the connection.

Sounds simple enough, ne?

The real trick is that group write permission anywhere up the directory tree can render these precautions meaningless. Who cares if I can’t see into .ssh in your home directory if I can manipulate your home dir itself?

3. $HOME and $HOME/.ssh must be locked down on the destination host.

A general good rule of thumb for permissions might be something like this:

ammon@farnsworth:~$ chmod 755 .
ammon@farnsworth:~$ chmod 700 .ssh
ammon@farnsworth:~$ chmod 600 .ssh/authorized_keys

Obviously, this gets kind of tricksy if you want to do something like allow SCP file transfers to the Apache user on a system… and their home dir is /var/www… and your web developers have group write access to this dir.

In situations like that, you have two options. First, you could disable the permissions checks (by turning off StrictModes in the sshd_config), but that’s not advisable. Second, you could make a separate home dir for the apache user with the restrictions in a place where they won’t interfere with anyone’s work.

Sometimes it’s the spaces vs tabs issue. Sometimes it’s file line endings (silly Windows-only editors and their CRLF). And sometimes it’s just people adjusting whitespace arbitrarily on lines (like adding spaces around parens or leaving spaces at the end of lines, etc…).

Regardless of the individual manifestation, it’s a silly problem, but one that causes more than its share of tears among developers everywhere.

Perhaps the easiest and smartest solution is to browbeat your co-developers into compliance. Force people to use editors that preserve line endings, force them to strip trailing whitespace and conform to a universal standard of indentation, etc… but it’s not always the nicest or most reliable solution. People will make mistakes, even if it’s only once a month… going over that diff might cost you an hour to figure out what had actually changed.

There are a few other solutions out there. They’re not new, and they’re not for everyone… but they can be phenomenally helpful at times. I’ll go over the two simplest ones.

dos2unix

Ever gotten a diff that reads like this?

ammon@binky:~/test$ svn diff one
Index: one
===================================================================
--- one (revision 2)
+++ one (working copy)
@@ -1,11 +1,11 @@
-One is the loneliest number that you'll ever do
-Two can be as bad as one
-It's the loneliest number since the number one
-
-No is the saddest experience you'll ever know
-Yes, it's the saddest experience you'll ever know
-`Cause one is the loneliest number that you'll ever do
-One is the loneliest number, worse than two
-
-It's just no good anymore since she went away
-Now I spend my time just making rhymes of yesterday
+One is the loneliest number that you'll ever do
+Two can be as bad as one
+It's the loneliest number since the number one
+
+No is the saddest experience you'll ever know
+Yes, it's the saddest experience you'll ever know
+'cause one is the loneliest number that you'll ever do
+One is the loneliest number, worse than two
+
+It's just no good anymore since she went away
+Now I spend my time just making rhymes of yesterday

This is what happens when something changes the line endings of a file. In this case, the original file was created with LF endings and was then edited slightly by an application that converted them to CRLF.

Now… if this were a 1000 line perl script in stead of an 11 line lyrics snippet… it would be soulcrushingly difficult to find the one actual change in the file.

Most unix distros have at their disposal the dos2unix / unix2dos utilities. On Red Hat, you can yum install dos2unix to get them. On Debian/Ubuntu, you can apt-get install tofrodos. I don’t have any other unices lying around at present to check on, but you can always just get the source at http://www.thefreecountry.com/tofrodos.

ammon@binky:~/test$ dos2unix one
ammon@binky:~/test$ svn diff one
Index: one
===================================================================
--- one (revision 2)
+++ one (working copy)
@@ -4,7 +4,7 @@

 No is the saddest experience you'll ever know
 Yes, it's the saddest experience you'll ever know
-`Cause one is the loneliest number that you'll ever do
+`cause one is the loneliest number that you'll ever do
 One is the loneliest number, worse than two

 It's just no good anymore since she went away

Much easier to figure out what has changed this way.

For extra credit, look into the svn:eol-style property. Set this on files as you commit them – or just use autoprops to do the dirty work for you…

diff-cmd

Of course, some times it’s not line endings. Sometimes the problem is random meaningless whitespace changes. Maybe somebody used an editor that auto-indents with spaces when the file was already indented with tabs, etc…

Subversion allows you to specify an alternate command to use to generate your diffs (in stead of relying on svn’s internal diff generation).

ammon@binky:~/test$ svn diff ----diff-cmd /usr/bin/diff -x -w one
Index: one
===================================================================
7c7
< `Cause one is the loneliest number that you'll ever do
---
> `cause one is the loneliest number that you'll ever do

But what if (for some bizarre reason) you don’t care about the case of letters?

ammon@binky:~/test$ svn diff ----diff-cmd /usr/bin/diff -x -iw one
Index: one
===================================================================

If you always want to use your custom diff utility you can set it in your runtime config to save yourself the hassle of having to type it manually each time.

For those using TortoiseSVN, you can always just specify graphical diff/merge utils to use in stead of Tortoise’s builtin ones. Personally, I’m a big fan of WinMerge, but there are several other good ones out there.

I have a swf that is being loaded off of an https server. As it fires up, it attempts to call a simple authentication service that lives on the same host. This works fine under Firefox, Opera, and Safari.

However, under IE, it throws an exception:

[IOErrorEvent type="ioError" bubbles=false cancelable=false eventPhase=2 text="Error #2032: Stream Error. URL: https://host/path/script.php?username=allaryin&passwd=hash"]

Obviously, if I just load the url directly into IE, it loads fine. This problem only occurs when flash tries to load the url for me.

When I monitor the query with Charles, it shows that the request is being made correctly and that the information I am expecting is successfully being returned. However, Flash is apparently ignoring the response.

This behavior has been observed on different machines, running both XP and Vista.

Thoughts?

A few days after sending this email, I’d received no response other than an IM from a friend on the list who didn’t know the answer either. So… I resumed consultation of the Google. I dug through ML archives. I read IRC logs. Eventually, I stumbled across mention of a blog post back in 2005 that had addressed a similar problem under IE6. Unfortunately, the site hosting this old blog has ceased to exist/function. So, I found it on the Internet Archive:

http://web.archive.org/web/20070521185428/http://www.gmrweb.net/2005/08/18/flash-remoting-https-internet-explorer/

The post mentions a few potential solutions to the problem such as doing some http header management in Apache. I tried the suggested changes (in the PHP, I didn’t have the access/desire to tweak Apache at the time):

header("Expires: " . date("D, d M Y H:i:s", 0) . " GMT");
header("Last-Modified: " . gmdate("D, d M Y H:i:s") . " GMT");
header("Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0");

This didn’t have any effect on my particular problem. Charles showed that the requested headers were being sent correctly, but Flash + IE7 + HTTPS still failed to talk to my web service.

So, I poked around the net some more without coming up with any helpful solutions to the problem. Returning to the archived blog post, I read the comments and saw another solution proposed. Someone said that simply sending an empty pragma header seemed to help in cases where IE was having difficulty dealing with PHP sessions over HTTPS. Specifically, the pragma header needed to be flushed after the session had started.

And we were using a session variable…

session_start();
header("Pragma: ");

And it works now.

So… yeah. Hooray for obscure comments on wayback machine archived blog posts addressing a similar problem

Big application we’re working on over here (a Flex/AS3 rewrite and feature upgrade for the CMYK Books layout application) works fine in every browser we’ve thrown at it. With one interesting caveat.

It doesn’t play nice with Opera 9 (haven’t tried older versions, come to think of it…).

The application loads just fine. If you point your browser directly at the SWF file, it does everything it’s supposed to.

But, if you load the SWF from within an HTML page (ie, the normal way), it starts to be problematic. Specifically, it starts sending bogus HTTP headers. These headers confuse the web services (since IIS detects that the’re being sent an ill-formed request and refuses to pass it on for soap parsage). What you wind up with is a page that loads normally and takes you to a login form that doesn’t do anything when you click the button.

Specifically, what happens is the application sends two Referer headers. One containing the URL of the HTML file and one containing the URL of the SWF. If the SWF isn’t loaded from within an HTML file, it only sends the one referrer (the SWF’s location), and IIS recognizes the valid request and everything works.

So… how to fix it? Not entirely sure yet. None of the options that I’ve been able to see/change seem to have any effect. But this is a pretty big strike against Opera IMO. I’m a fan of the application, it’s always been good to me when I’ve used it. I’m excited to play with it on my Wii (if I ever get one… mumble). But for it to do such an odd thing… confuses me.

I’ve currently got a doozy of an issue in a project that I thought I was done with over a month ago

There are two and a half ways to manipulate sound in Flash.

1. You can just drop audio tracks onto the timeline and hope for the best.
2. You can instantiate Sound objects in, attach the audio tracks, and play them.
3. A hybrid method where sounds are loaded onto the timeline but are then manipulated with a Sound object.

In 90% of cases, my game uses the second method. In the other 10%, I used the third. I have a simple flag (stored at _global.audio) that is set to true if people want sound playing and false otherwise. Any of my sound handling code monitors changes in the flag and sets volume on the appropriate sound objects accordingly. Nothing too fancy.

However, we’ve found an interesting fringe case when trying to control timeline sounds via actionscript. To duplicate the problem:

• Take one stage with a 2 frame timeline and one movie clip.
• On the movie clip, load a sound onto the timeline.
• Put the clip onto frame 2 of the stage, leaving frame 1 empty.
• Put a stop() action on the second frame of the main timeline.

This works as intended – ie, the sound plays exctly once.

• Now, add some code to the movie clip’s timeline to control the volume:

  var snd:Sound = new Sound(this);
  snd.setVolume( 0 );
  

This also works as intended – ie, the sound does not play at all.

• Replace the stop() action on the main timeline with a gotoAndStop(1) call.

Now if you play the swf… the sound will happen… even though we’re explicitly telling it not to – and you’re jumping to a state where the movie clip containing the sound should not be loaded.

I suspect that what is happening is that when you leave the portion of the timeline where the Sound object exists, there is a bit of confusion in the VM. In stead of discarding the movie clip in its entirety as makes sense, it’s discarding the sound object first. The timeline then continues playing the audio where it left off (after being muted for only one frame).

So… if you want reliable control over your audio assets, the solution might then be to put your Sound object responsible for playing them in a location where it won’t be destroyed when the timeline updates. Unfortunately, that doesn’t seem to work either. I’ve tried locating my Sound object in _global or _root, but that doesn’t solve the problem either.

What appears to happen is that sounds on the timeline are attached to the stage when their individual timelines disappear. Telling my Sound object to listen to the stage in stead of to the movie clip works.

I’m investigating this more… maybe… but shrug. More likely, I will simply find a way to stop using sounds on the timeline at all in this one problem spot in the game.

I have a Trac install on the server that’s protected with generic http auth:

AuthType basic
AuthName "Repository Name"
AuthUserFile "/dir/to/htpasswd"
Require valid-user

This sort of config has worked for forever. It worked fine under 2.0. It works fine under 2.2. This is not the problem.

When they changed the version number to 2.2 they renamed a whole bunch of the auth modules. They also split a bunch of the behaviors out into multiple separate modules.

So, in order to get the behavior I got by including just mod_auth, I now had to include several different mods. No problem there. The docs tell you this. No problem there.

Thus, my config got a section that looked like this:

LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authz_host_module modules/mod_authz_host.so

Mod_auth_basic for the fundamental behaviors. Mod_authn_file to allow me to read from an htpasswd file. Mod_authz_host because they renamed mod_access (the “allow from xxx.yyy.zzz.com” type directives).

That got apache loading and understanding all of my config file. Then I tried to go to the trac install and blam. 500 internal nastiness error of kabloomitude.

Apache log file said:

[Mon Nov 20 11:47:12 2006] [crit] [client my.ip.address] configuration error:  couldn't check access.  No groups file?: /projects, referer: http://hostname/

Yeah. I was pretty sure I never mentioned anything about auth groups. Checking the entire config file proved me right. 30 minutes of quality time with Google proved entirely unhelpful. Just a bunch of cries for help that were either unanswered or eventually resolved for reasons unrelated to my problem.

I did discover that including mod_authn_default at least prevents the 500’s – just turning them into auth denied errors.

Turns out in my case that in order to actually auth against usernames you must include mod_authz_user now. This wasn’t mentioned in any of the docs I dug through.

So what was happening?

The “no groups file?” error message provided is erroneous. It’s generated as part of some fallthrough code in Apache itself that happens when no existing auth mechanism is able to assume responsibility.

It requested the password and then had nothing to do with it – it didn’t know how to “Require valid-user” or something so it just bombed through, hoping another mod would answer… which of course didn’t happen since I didn’t specify a second authentication method for these url’s.

Moral of the story?

Starting from scratch with Apache config files is painful. Avoid it if possible. If upgrading versions, grab the default config file and merge your changes into it in stead of dropping your file in place and hoping it works as is.

I would have done this but I was working on a Windows server (yeah, I know, but some people in the .NET group are scared of progress – more on this particular saga later) and the windows apache installer doesn’t apparently generate a default config file if it detects one already in place. On unix, I always compile critical server applications by hand, so I always have the defaults to work from.

I’m currently involved in some very heavy manipulation of XML data in a Flash application. For such purposes, I am compelled to use XPath. It’s a decent enough language. Ok, I lied. It’s good. Saves mounds of time that would otherwise be spent iterating over xml tree structures and searching for elements.

When using XPath in Flash, one has two options. First, Macromedia Adobe has a first party implementation. Apparently it’s not that wonderful and nobody who knows better uses it. Second, there’s the XFactor Studio API. This is what I use. It’s what a lot of people use. It is nice. It has never failed me before.

Until today.

I have a very simple favor to ask of the library. I want to select all of a node’s siblings. Now, granted, I am doing some searching to determine which node’s siblings I need here, but it’s really basic.

The XML doc I’m working with looks something like this:

<configurator>
    <group>
        <products>
            <p>1</p>
            <p>2</p>
        </products>
        <choice id="a"/>
        <choice id="b"/>
    </group>
    <group>
        <products>
            <p>3</p>
        </products>
        <choice id="c"/>
        <choice id="d"/>
    </group>
</configurator>

What I want is all <choice>’s in the same <group> as a <product> with a given <p>. Sound simple enough? Well, it is. The XPath statement to do this might look something like this:

//group[products/p='3']/choice
or
//group/choice[../products/p='3']

Both search paths are equally valid and return the same results when I try them out on other xpath implementations.

The XFactor version returns nothing. I know the lib works, and I know I’m using it correctly since my program continues to function up to this point (and I’m making 4 or 5 selections before this point).

So, I’m mildly peeved here. I’ve tried breaking it up into a pair of sequential queries (first select the appropriate group, then select the choices within that group), but that doesn’t do anything either.

So far, this problem has cost me 2 hours this morning and it doesn’t look to be clearing up any time soon. Sigh.

update

After another hour, I have determined that the XFactor XPath implementation is broken when it comes to searching based on text node values. Chris (coworker) swears up and down that it isn’t broken, so he’s trying to prove me wrong now

In the mean time, I have implemented an ugly brute-force solution to the problem. So, in stead of:

var path:String = "//group/choice[../products/p='"+pid+"']";
var list:Array = XPath.selectNodes( configurator.firstChild, path );

I have had to settle for something like this:

// grab all products
var path:String = "//group/products";
var pNodes:Array = XPath.selectNodes( configurator.firstChild, path );
var gNode:XMLNode = null;

// iterate over products until we find the one we want, YICK
for( var k = 0; k < pNodes.length; k++ ) {
	// look at all <p> children
	var tmp:Array = pNodes[k].childNodes;
	for( var j = 0; j < tmp.length; j++ ) {
		if( tmp[j].firstChild.nodeValue == pid ) {
			// parent.parent 
			gNode = tmp[j].parentNode.parentNode;
			break;
		}
	}
	if( gNode != null ) break;
}

// grab the appropriate wad of choices, finally
var list:Array = XPath.selectNodes( gNode, "//choice" );

update

After about 20 minutes of headscratching, Chris seems to have given up as well

Interestingly enough… the first party api doesn’t do it either:

var path:String = "//group/choice[../products/p='"+pid+"']";
var list:Array = mx.xpath.XPathAPI.selectNodeList( configurator.firstChild, path );

So… I’m very curious if anybody actually does these kinds of searches in Flash? They’re valid according to the W3C spec and they work in Java… for a language like Flash (which is currently handcuffed to XML as its only reasonable means of communication with the outside world) to have a broken XPath implementation is… not good. Sigh.

We have a total of four computers currently on the network, and in regular use. Of them, Rincewind (the old server turned secondary windows desktop) and Vimes (“my” laptop that Penny monopolizes) generally shun the torrents, letting Hedwig (linux server) and Tarma (windows media center type bedroom computer) handle all the dirty work. Which is fine by them, since they’re the machines with the big processors, big ram, and big hard drives (though Rincewind does have a gig of ram and 300gb drive of his own now, shrug).

Traditionally, I have always preferred the command line python scripts to bulky gui applications. So… Hedwig has had a long and only slightly besmirched career as our primary download and redistrobution center.

When Hedwig moved home from the office after I quit my last job, I expected some slowdown – after all, we only have a 5 mbit downstream and 512 kbit upstream. The results of my first batch of downloads led me to believe that the torrent itself was somehow bugged – they were all healthy files with 10 or more seeds and 100 or more general users, yet even after a few hours running, I couldn’t manage to eke out more than 5kbps per torrent.

I let this slide for a while in the face of a new problem. One particular studio whose shows I was interested in watching was having problems with their single-episode torrents and recommended people to just use the batch torrents in stead. This is usually fine by me, but in this case, the batches were 25 episodes huge (~4.5gb) and I only needed one file out of the mix.

So, I hunted around for a client that would allow me to selectively download individual files from batch torrents, and was pleased to find BitComet. I installed it on Tarma and fired it up.

Instantly, I noticed a huge difference. BitComet was actually able to utilize my bandwidth and get me transfers at the rates my connection would actually support. It was great. But, I was still running the mud from home and didn’t want to lock out players, so I throttled things way back – leaving plenty of bandwidth open for no less than 10 users to do their bidding w/o any problems.

Yet… even with a dedicated portion of the pipe, users suddenly began to complain about inability to log in to the game. Once an active socket was opened, there were no problems, but people were generally incapable of opening said sockets…

About the same time, Penny started complaining from the couch that the laptop was having problems browsing web pages. I noticed that I too began to have problems opening up outgoing connections from any of our machines. I throttled the torrents waay back – leaving 3/4 of our bandwidth unused. And still, the problem did not go away.

I quit BitComet. The problem did not go away. I rebooted Tarma. No such luck there either.

Finally, I gave our router a violent kick to the head and rebooted it forcibly. The network was immediately cleansed and back to normal.

This problem has repeated itself numerous times since that first event. It became so reliable – if I so much as looked at BitComet, my network would sieze up – that I stopped downloading files all together until the mud had left the building. And then, I got into the habit of stopping torrents during the day so I could work, and then turning them back on when I went to sleep. I had assumed that BitComet was somehow opening up so many connections that it managed to exhaust the router’s capacity to handle them all.

Until today. [Cue dramatic music and lighting]

The answer (I hope) to all of my woes and problems finally hit me. BitComet was not leaving stale connections around. It has been overheating the router. See… they’re both fairly new (<1 year old) Linksys models, and this particular line of networking hardware is designed to stack together, so the router was sitting directly on top of the modem.

I figure the high number of different sockets created when added to the high amount of wan communication and the fact that Tarma is connected to the network via encrypted wifi (high amounts of radio energy plus extra computational power spent on every packet to maintain security) probably generated more heat than anything else I’ve done to this network.

Since, previously, my high traffic transactions had been mostly one-way communication (sending a file between two machines) and my torrents had been on Hedwig (who is plugged into the network via good old fashioned cat6 twisted pair)… yeah.

When I checked on my torrents this morning (Sousei no Aquarion for Penny), I noticed that they were acting unusually sluggish. Like really really slow. So, since I needed to get onto the web anyways to work on things, I went in to physically reboot the router. It was almost painfully hot to the touch. The modem underneath it was also practically a puddle of molten plastic.

I smacked myself in the head and proceeded to fashion a cardboard stand for the router from the packaging of some Atari computer game or another. There is now at least an inch of airflow between the two devices and I’ve had torrents running all day w/o interrupting my work.

I can even post rants on here and stuff