Ok, a while back when TekTonic (A’tuin’s ill-favored hosting service) crashed and had all kinds of problems in general, this caused a reboot of my system. Quite understandable that. Unfortunately, the box had been up for a while (probably 3 months or so of uptime) and they must have applied some kernel patches or something to their system because it has been unstable ever since.

Most notable among these problems is a heinous memory management problem - new processes segfault in stead of making something else swap out. Only slightly less notable is the fact that TeamSpeak ceased to work at the same time.

ammon@atuin:/usr/local/tss$ ./teamspeak2-server_startscript start
starting the teamspeak2 server
Runtime error   0 at BFFFFBB8
./teamspeak2-server_startscript: line 92: 27874 Segmentation fault      ./server_linux -PID=tsserver2.pid

Yeah. Fun stuff. Over time, the only difference in the output tends to be the memory address at which it borks out on me Didn’t really bother me much since I wasn’t actually hosting any chat services - I had actually only installed it on a whim in the first place anyways.

I’m also not the only one with this problem. Before they mysteriously vanished, the TekTonic support forums had at least 3 or 4 threads complaining about this exact problem.

Well, earlier this week, I had a discussion with guild leader on CoV where we talked about possibilities for voice chat. I brought up A’tuin and its currently functional Ventrilo setup - but the Vent license only allows 8 clients to connect to a non-pro account… and they’re very picky about with whom they will do business (pro accounts supporting a minimum of 1000 clients at once spread across multiple servers and stuff).

Our SG has more than 8 members now, and it would be a shame if people were getting rejected just because Vent refuses to sell me a license. So, I started looking back into TS again today. It still crashed.

Using the trick taught to me back during my first Google tech interview, I applied strace to the binary. Lo and behold, the trace reveals that the crash occurs whilst trying to open /usr/lib/locale/en/LC_CTYPE. I checked, and sure enough… my machine seemed to have a broken installation of the locales package… and along with it, a rather old version of libc.

So… patching these up gives me progress. I’m going to see what else needs to be done to bring things up to speed, but I might actually get TS running on this monster tonight.

Update - 12:30am, Feb 22

After updating glibc and locales (and practically every other system library), I am still getting the following output from strace:

...snip...
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 4
fstat64(4, {st_mode=S_IFREG|0644, st_size=1583760, ...}) = 0
mmap2(NULL, 1583760, PROT_READ, MAP_PRIVATE, 4, 0) = 0x40170000
close(4)                                = 0
pipe([4, 5])                            = 0
--- SIGSEGV (Segmentation fault) @ 0 (0) ---
rt_sigreturn(0)                         = 136425499
...snip...

Blech. So… it’s closing the locale archive document and then trying to pipe it somewhere else? Don’t know if this is actually the problem though. I really need to learn how to use this utility better. Mumble, grumble. I’m gonna dig through ML and forums and stuff to see if I can come up with anything further.

Update - 7:30am, Feb 23

Well, fooey. I looked at man pages online (since I can’t seem to locate which Debian package actually installs the syscall man pages for me…) and aside from discovering that there is apparently nothing fishy about the pipe call, noticed something else in the sigreturn man page:

sigreturn never returns.

Well, not terribly interesting. If the machine is segfaulting, the kernel is well within its rights to make whatever syscalls are necessary to recover and report error messages to me and stuff, no? So that might be a dead end.

Looking further up through the entire stack trace, I noticed that attempts to read from this /etc/ld.so.nohwcap file that I’d never heard of were all failing. Google turns up this blog entry. So, I created the file and nothing really changed except that the program doesn’t fail on those particular reads any more. I’m assuming that the access to /etc/ld.so.preload is also harmless, so will ignore this one.

So, I figured I’d try to force a segfault of my own to compare the two straces for any similarities. Not much there, eh? But when I do an ltrace on my file - it clearly specifies what’s crashing.

The ltrace of the TS binary is way too verbose to put online anywhere, so yeah. It looks like it’s dying on some assortment of pthreads mutex calls, which I guess is entirely possible and realistic.

Update - 8:50am, Feb 23

The TeamSpeak forums seem to have a handfull of threads dealing with this exact sort of problem - which seems to have affected people on multiple distros (including Debian). A German thread is the longest discussion of the problem and seems to have an answer.

They use setarch to fool the machine into thinking that it is something it is not… Erm, ok, I guess. It sounds like problems with architecture-specific behaviors failing when run on the wrong kind of box. That makes some sort of sense.

It especially makes sense on my machine - a VPS on dual Opterons. Not the most normal architecture out there. When you run uname -m, it just says i686… so shrug.

Of course, setarch isn’t available for Debian. The solution given in the german thread is to use alien to turn an RPM into a DPKG for installation I’ll try that out later and see what it does to me.

Posted by Ammon in howto, sysadmin, teamspeak, ventrilo, work