Recently, I’ve encountered the need to set up fully redundant nagios servers in a typical pair setup. However, reading the documentation [here], the solutions seemed lacking. The official method is to simply run two different machines with the same configuration. The master should do everything a normal nagios server should do, but the slave should have its host & service checks and notifications disabled. Then, a separate mechanism is set up so that the slave can “watch” the master, and enable the aforementioned features if the master goes down. Then, still checking, the slave will disable those features again when the master comes back alive.

Well, this solution sounds just fine in theory, but in practice it really creates several more problems than it solves. For instance, acknowledgements, scheduled downtimes, and comments made do NOT synchronize with the official method. Their mechanism does not allow for this, as it uses the obsessive-compulsive service and host parameters, which can be executed after every single check is run. It therefore has no access to the comment/acknowledgement data, so it simply cannot synchronize.

So can this data be synchronized? The short answer is yes, but I’ll explain the game plan before we dive in. Nagios can (unknowingly) provide us with another synchronization method, its own internal retention.dat file. This file is, by default, written to every 90 minutes by the nagios process, and contains all of the information necessary to restore nagios to the state it was in when it exited. Sounds like exactly what we need! So we’re going to now stop thinking about running nagios, and start thinking about how we can take this blob of ASCII data, and ensure it never gets corrupted and is as frequently as possible being updated. This is, after all, the true goal of the situation.

First and foremost, we will need a nagios installation! You can follow their documentation on this one and set one up for yourself. It’s ok, I can wait.

Second, we need another nagios installation on a second server! Hop to it.

Third, all of the nagios configuration files need to be constantly synchronized between these two hosts. I use puppet to synchronize my config files over the servers I administer, so unfortunately my implementation is highly specific. You may need to find other ways to synchronize your config files, but this should not be terribly difficult (perhaps a Makefile with a versioning repository and ssh keys?). This is needed in the official nagios failover deployment as well. Anyway, one “gotcha” I faced was the need to change the configuration parameter “retain_state_information=90” to “retain_state_information=1”. Do not forget to do this or else synchronizations will only occur once every 90 minutes.

Fourth, you will need to deploy this script on both hosts, and configure the requirements. You will see embedded ERB syntax in this script, that is because puppet allows me to configure discrepancies in my deployment inline, as the final configuration files are generated on-the-fly, then pushed to the clients.

[root@puppet ~]# cat /usr/bin/nagios-watchdog.sh.erb
#!/bin/bash

# Executable variables. Useful.
RM=”/bin/rm -f”
MV=”/bin/mv”
ECHO=”/bin/echo -e”
FIXFILES=”/sbin/fixfiles”
MAILER=/usr/sbin/sendmail
SUBJECT=”URGENT: nagios master process switch has taken place.”
RECIPIENT=”sysadmin@example.com”
SERVICE=/etc/init.d/nagios
RETENTIONFILE=/var/log/nagios/retention.dat

# This is where we point the servers at each-other (configure this properly in your deployment!)…
<% if fqdn == “nagios1.example.com” %>
MASTERHOST=192.168.1.2
<% else %>
MASTERHOST=192.168.1.1
<% end %>

# Ensure only one copy can run at a time…
PIDFILE=/var/run/nagios-watchdog.pid
if [ -e ${PIDFILE} ]; then
exit 1;
else
touch ${PIDFILE};
fi

# Checks the actual daemon status on the other host…
su nagios -c “ssh ${MASTERHOST} \”/etc/init.d/nagios status\” >/dev/null 2>&1″

# Is the other host doing all the work?
if [ $? -eq 0 ]; then
# Stop what I’m doing…
${SERVICE} stop >/dev/null 2>&1

# Copy the retention data from the other nagios process…
su nagios -c “scp ${MASTERHOST}:${RETENTIONFILE} /tmp/”;

# Verify that we didn’t get a corrupted copy…
if [ `grep “{” /tmp/retention.dat | wc -l` -eq `grep “}” /tmp/retention.dat | wc -l` ]; then
${MV} /tmp/retention.dat ${RETENTIONFILE};
else
${RM} /tmp/retention.dat;
fi
${FIXFILES} restore /var/log/nagios
else
${SERVICE} status >/dev/null 2>&1
if [ $? -ne 0 ]; then
${ECHO} “From: nagios-watchdog@`hostname`\nSubject: ${SUBJECT}\nTo: ${RECIPIENT}\nNow running on host: `hostname`” | ${MAILER} ${RECIPIENT};
${SERVICE} start >/dev/null 2>&1;
fi
fi

${RM} ${PIDFILE}

exit 0;

There is a single requirement to this script, you must give no-password ssh keys to the nagios accounts on each host, but you can use those securely by using the allowed commands directives of the authorized_keys file.

Fifth, and finally, we must implement a mutex operation around running nagios processes. Recall that we are synchronizing copies of nagios internal state data, and having a running nagios process is just a luxury. If you look at the script above, it simply ensures that nagios is running one, but not both servers, and ensures that the newest retention.dat file always has priority. The mutex operation doesn’t need to be infinetely accurate, I used the following relatively barbaric solution:

[root@nagios1 ~]# crontab -l
1,5,9,13,17,21,25,29,33,37,41,45,49,53,57 * * * * /usr/bin/watchdog-nagios.sh
0 6 * * * /usr/bin/nagios-reports.sh
0 12 * * * /usr/bin/nagios-reports.sh

[root@nagios2 ~]# crontab -l
3,7,11,15,19,23,27,31,35,39,43,47,51,55,59 * * * * /usr/bin/watchdog-nagios.sh
0 6 * * * /usr/bin/nagios-reports.sh
0 12 * * * /usr/bin/nagios-reports.sh

Sixth, and optionally, as you can see above, I’ve also set up redundant reporting. We do a similar test to ensure that at a maximum, only one email report is dispatched for the given timeframe. In this solution, reports could be theoretically lost forever if a specific set of circumstances is met, but that was deemed acceptable in this deployment. To see the real magic behind that script:

[root@puppet ~]# cat /var/lib/puppet/files/nagios/usr/bin/nagios-reports.sh.erb
#!/bin/bash

/etc/init.d/nagios status >/dev/null 2>&1

if [ $? -eq 0 ]; then
/usr/bin/nagios-reporter.pl –type=24 –embedcss –warnings
fi

And voila, of course nagios-reporter.pl could be any report generation tool you wish, just be sure to call it in the method that suits your reporting needs.

Seventh, and convenient to have, I also wrote these two quick PHP scripts. Throw them in /var/www/html on each nagios box and do not redirect straight to nagios. Then setup DNS in a round-robin multiple A-record fashion. That is,

[root@puppet ~]# dig +short nagios.example.com
192.168.1.1
192.168.1.2

Once you get that set up, insert these two files into the aforementioned directories:

[root@puppet ~]# cat /var/lib/puppet/files/nagios/var/www/html/index.php.erb | sed ‘s/</\</g’
<HTML>
<HEAD>
<TITLE>INTERNal Redirect</TITLE>
</HEAD>
<FRAMESET ROWS=”30,100%” BORDER=”1″ STYLE=”border-style:solid” noresize>
<FRAME SRC=”switcher.php” NAME=”switcher”/>
<?php

// This will set the $me and $you variables correctly…
$me = “<%= fqdn %>”;
if($me == “nagios1.example.com”)
{ $you = “nagios2.example.com”; }
else
{ $you = “nagios1.example.com”; }

# Test whether or not nagios is running locally.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, “https://localhost/nagios/cgi-bin/status.cgi”);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
$output = curl_exec($ch);
curl_close($ch);
$pos = strpos($output, “Whoops!”);

if($pos === false)
{ echo(“<FRAME SRC=\”https://$me/nagios/\” NAME=\”activenode\”/>”); }
else
{ echo(“<FRAME SRC=\”https://$you/nagios/\” NAME=\”activenode\”/>”); }

?>

</FRAMESET>
</HTML>

[root@puppet ~]# cat /var/lib/puppet/files/nagios/var/www/html/switcher.php.erb | sed ‘s/</\</g’
<HTML>
<HEAD>
<TITLE>Switcher</TITLE>
</HEAD>
<BODY>
<CENTER>
<FONT SIZE=”-1″>

<?php
$me = “<%= fqdn %>”;
if($me == “nagios1.example.com”)
{ $you = “nagios2.example.com”; }
else
{ $you = “nagios1.example.com”; }

# Test whether or not nagios is running locally.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, “https://localhost/nagios/cgi-bin/status.cgi”);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
$output = curl_exec($ch);
curl_close($ch);
$pos = strpos($output, “Whoops!”);

if($pos === false)
{ $current = $me; }
else
{ $current = $you; }

echo(”
Currently using: $current [<a href=\”javascript:parent.document.location.reload(true)\”>Redetect active node</a>]”);
?>

</FONT>
</CENTER>
</BODY>
</HTML>

So now, you can simply visit http://nagios.example.com and nagios will always be displayed, unless a particularly bizarre set of circumstances has occurred. If this is the case, don’t panic! Remember, we set our minds correctly in the beginning and the integrity of the retention.dat files is not in question. The scipts may just take a minute or two to adjust themselves properly. For those that worry the DNS failover wouldn’t work, I’ve verified that it does on some of the popular browsers. There is no 90-second timeout delay, either, as in all but the rarest circumstances. I verified that a timeout can occur if the first connect() call’s SYN packets are dropped completely, but this is the aforementioned rare circumstance. Most testing on this is done at the iptables level, but be sure to REJECT the SYN packets (not DROP!) if you want an accurate account of the speed of the failover in your web-browser during a real-life outage. Also ensure that your router will send proper ICMP Host Unreachable responses if one of the addressed hosts is offline.

I think that’s pretty much all you need to get going. This deployment has been running for a little while now in a production environment, and has been rock-solid. It’s a bit more work than the official solution, but it solved my monitoring needs and extensive testing, both real-world and artificial, has not yet revealed any issue with this solution.

xmodmap -e “remove mod4 = Super_L”

Then go back into the Keyboard Shortcuts list and try again. Super_L should now show up in the window when you hit the key.

1) Get the frame on the network. I really can’t help you too much with this, just use the picture frame’s menu system to do it. The menu is fairly intuitive. If possible, try to give the picture frame a static IP address, whether through the frame’s menu system or via the DHCP server in your private network.

2) The frame sends a lot of, well, odd traffic. It uses multicast and unicast traffic, so, and you might be disappointed in me over this, I recommend you just allow all traffic from that source. A rule such as “-A INPUT –src 192.168.100.100 -j ACCEPT” in /etc/sysconfig/iptables will work perfectly here. Just change the IP address in the example to the static IP address you gave the frame when you got it on the network in part 1.

3) Install [mediatomb] and start it. Check it’s configuration file for what port it runs on, and navigate to that port on your local computer. For example, point firefox at “http://127.0.0.1:thatport”. From there, configure what pictures and stuff you would like to share with the frame.

4) Reboot the frame. At the main screen, an entry labelled “Network Computer” should appear. Click on that one, navigate to the folder with all the pictures in it, and press play. Congrats, you’re now sharing your photos.

5) You may want to make mediatomb start automatically on system boot with “chkconfig –levels 345 mediatomb on”.

6) You don’t need to restart the frame or mediatomb to make changes to the pictures that are shared. Just make the changes, and they’ll happen in real time.

Hope this helps someone out there.

[brose@allmybase-demo]$ convert -resize ‘1024×768’ in.jpg out.jpg

your image would be resized such that neither dimension exceeded the bounds, but unless the aspect ratio is dead on (convert honours and preserves aspect ratios), your image will not be the exact size you specified.

But, there is a simple fix. Append an exclamation point to the end of the size, like this:

[brose@allmybase-demo]$ convert -resize ‘1024×768!’ in.jpg out.jpg

and convert will do exactly what you ask, even if it means distorting the image by destroying the aspect ratio. This could be pretty handy to someone, so I decided to blog it. Heck, it’ll probably be useful to me again soon when I say, “hmmm… how did I do that again?”. Enjoy.

Anyway, this leaves us with a couple of options. You could do no video chatting, or you could also use another utility to chat, such as gnome-meeting. I’m going to assume, however, that these aren’t good options for you. Below we’ll embark on the quest of getting skype to behave properly on a 64-bit linux system. For me, this happened to be on Fedora 11. Your mileage may vary.

First, go grab the RPM from the skype website and install it. Pull in the dependencies as well via yum. Then, make sure the packages “libv4l-0.5.9-1.fc11.i586”, “pulseaudio-libs-0.9.15-14.fc11.i586”, and “alsa-plugins-pulseaudio-1.0.20-2.fc11.i586” (or whatever versions you want) are installed.

[root@allmybase-demo ~]# yum -y install libv4l-0.5.9-1.fc11.i586 pulseaudio-libs-0.9.15-14.fc11.i586 alsa-plugins-pulseaudio-1.0.20-2.fc11.i586

First thing’s first, we’ll need to make sure that this library will load properly when skype launches, otherwise there will be no video for skype to send to your chatting partners. To do this, mimic what I’ve done here:

[root@allmybase-demo ~]# mv /usr/bin/skype /usr/bin/skype.proper
[root@allmybase-demo ~]# cat << EOF > /usr/bin/skype
#!/bin/bash
LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so
skype.proper

EOF

What we’ve done here is ensure that skype will be able to communicate properly with the webcam by preloading the 32-bit library file before the execution of the next ELF binary. At this point, fire up skype. Go into the options, and set all of the audio dropboxes to “pulse”. If you’re lucky, sound will work out of the box. If not, you’ll probably get something like this, however:

[brose@allmybase-demo ~]$ skype
RtApiAlsa: underrun detected.
RtApiAlsa: underrun detected.
RtApiAlsa: underrun detected.
RtApiAlsa: underrun detected.
RtApiAlsa: underrun detected.

This seems to be a well-known bug. There have been a variety of “hacks” to make the problem bearable, and some of them do work a bit, but I’m going to present another option here that works quite well. Traditional hacks usually involve changing the nice level of pulse audio such that it runs at a different rate than the fork()’d process sending audio data. By adjusting this finely, one can minimize the impact of the buffer underrun by ensuring that data will ALWAYS be written to the buffer before it’s played.

However, I’m going to make one assumption here, no matter how incorrect it may be. I’m going to say that you’re going to try to keep the computer as quiet as possible while chatting on skype. First, mute or turn off any programs that are actively using the sound on your system. A simple test of this is whether or not there are sounds coming out of your speakers. Now in skype, change the three “sound devices” dropbox settings to the underlying hardware on your system. For me, this was labelled as, “HDA Intel (hw:Intel,0)”. Apply the settings and see if you can now get proper sound on skype by calling the echo123 test number. If so, you’re done. Just remember to always stop your music and audio applications before making a call.

If at this point, however, you get “Problem with audio playback”, it means that you didn’t kill all services using sound. It’s okay, there’s a simple solution. Quit skype, and, from the command line, run it again as:

[brose@allmybase-demo ~]$ pasuspender skype >/dev/null 2>/dev/null&

This will suspend pulseaudio from using the soundcard while the application skype is running. You now can ONLY make sound with skype. To get other programs to make sound after this, you will need to quit skype. But, you can rest assured that your sound will work 100% and that there will be no audible interruptions coming from your computer. This fix does work, I’ve done it and so has Thomas (his link is on the right), and we’ve both made calls, having no problems whatsoever. Hope this helps someone, perhaps leave a comment if it did? Have a good one.

“First, the loading screen is a simple blue loading bar. Second, the boot process is extremely quick. Does the first have anything to do with the second? It sure does. The Fedora 11 development team has decided to go with speed instead of looks during the boot process and I think Fedora 11 users will be thrilled with this small change to speed up the LiveCD boot time.”

HE’S CORRECT, I’M SO THRILLED!!! One main reason I find this funny — he assumes he knows exactly what the Fedora development team had in mind when designing this blue bar loading screen, yet he’s not a developer on the team. Nor is he correct. Nor did he even read the release notes or improvements list. Nope, not a single piece of knowledge, data, or documentation to back up this claim.

The truth? The dev team did work very hard on the boot process, and the fruit of their labors is the plymouth codebase. The blue/light-blue/white loading bar? Yeah, that’s actually a fallback when the graphical plymouth booting mechanism fails. Great job doing your homework! So we’ve established two things here, that there’s at least one page on the internet with incorrect information, and that, if you are experiencing the fedora 11 blue bar boot (say THAT one three times fast), you can actually attain a prettier boot process.

This how-to is written both from memory, and specifically for my laptop, which I was attempting to spruce up with a nice graphical boot. If it’s inaccurate, sorry about that, but you can certainly use the base I’m giving you here as a launching point for further research.

First, my laptop has an NVidia GPU in it, so it was necessary to download the proprietary NVidia drivers and install them. I did not install the 32-bit compatibility libraries, since if I do, the NVidia installer says something to the effect of, “Could not check random component #9662, assuming successful installation”. Well NVidia, you know what happens when you assume things… Anyway, once you’ve gotten the nvidia kernel module installed, edit /etc/modprobe.d/blacklist.conf and blacklist the modules “nv” and “nouveau” to prevent them from being loaded. We only want the proprietary driver to be loaded here. Make sure you update /etc/X11/xorg.conf as well and tell it to use the nvidia module.

Next, we need to make the framebuffer have the optimal settings, that way plymouth won’t die and fall back to the blue bar loading screen. Append to the end of the kernel line in /boot/grub/grub.conf the text “vga=ask” and reboot. The kernel will prompt you for a video mode. I recommend typing “scan” here and striking return abruptly, as we want to make sure all available modes are listed. And yes, the abrupt manner with which the enter key is pressed is important. So yeah, pick which mode fits your screen the best, for me it was 1920x1200x32, which had a hex code of like 37D or something. Like I said, this is all from memory. Now, write down this hex code somewhere, because you’ll need it again shortly. After the machine boots, find a calculator and convert the hex code you just entered into the kernel into decimal, then place the decimal value in the /boot/grub/grub.conf file, replacing the “ask” with your number. Mine resolved to decimal “893”, so I ended up with “vga=893” on the kernel line. Congrats, you have now made a hospitable environment with which plymouth can bedazzle you.

Okay, now it’s time to configure the workhorse. We’ll need the appropriate utilities, so run this:

# yum install plymouth plymouth-gdm-hooks plymouth-libs plymouth-plugin-label plymouth-plugin-two-step plymouth-scripts plymouth-system-theme plymouth-theme-charge plymouth-utils

Honestly, most of these were probably already installed, but it never hurts to make absolutely sure. Now, recall before where we compiled the NVidia driver? We’ll need that to be loaded during the boot process so that plymouth can come up. This means that we’ll need a new initrd containing the NVidia module. This is also the purpose of blacklisting the nv and nouveau modules, as they now won’t get included into the initrd. I’ve become Canadian lazy lately, so you can run this command to see how to build a new initrd:

# rpm -aq –scripts kernel | grep initrd

A list of 2 or 4 commands will show up from that. Now find the appropriate one and run it. For me, that was:

# /sbin/new-kernel-pkg –package kernel –mkinitrd –depmod –install 2.6.29.5-191.fc11.x86_64

Voila, one new shiny initrd. But the initrd isn’t quite done yet. We need to update plymouth’s hooks into it. First, we’ll want to select a theme for initrd to use. I’ve made sure, at the very least, you’ve installed the “charge” theme in the yum command above. There’s plenty more in the repos, feel free to search for more, install them, and play around. Some are very neat. Anyway, to see which themes are available, do something like this:

# ls -1 /usr/share/plymouth/themes/
charge
default.plymouth
details
text

As you can see, I have a couple of themes installed, but let’s focus on the charge theme. Set it as the default theme, and then update it’s initrd hooks by running, in order:

# plymouth-set-default-theme charge
# /usr/libexec/plymouth/plymouth-update-initrd

Note: This entire abovesaid process needs to be redone for each NVidia driver or kernel update, otherwise it’ll be broken and fall back to the blue bars again.

Now, reboot and enjoy the pretty loading screen. Now there’s only one issue you’ll need to figure out from here on out that I can’t seem to work out, and that’s when to reboot in order to enjoy the loading screen anyway. I mean, if you followed this tutorial, you’re clearly running Linux, which means you’ll probably not need to reboot any time soon. Effectively, you may have just wasted a good 15 minutes setting up a boot screen you’ll never really enjoy too much. Thanks for reading my blog, though, I do appreciate it. I promise next time I’ll include a spoiler warning at the top of the article. Have a nice day!

[Native Linux 64-bit Flash Player from Adobe Labs]

It works very well. No longer do you need to get nspluginwrapper working, and then configure pulseaudio (if that’s your distro) to listen to 32-bit apps, blah blah blah. Simply unpackage, throw the .so file into the firefox plugins directory, and restart firefox. Great stuff. I’ve also noticed there’s a performance gain to be seen, my flash videos used to lag and flicker on youtube when I made them fullscreen, and this doesn’t happen anymore. Sound worked out of the box with pulse.

And the best part, although it’s supposedly an alpha build, it crashes a heck of a lot less than it used to. No more npviewer.bin segfaults in the /var/log/messages file! My guess is that this is because of the lack of a hacked wrapper. But, that being said, if it crashes on you a lot after trying it, don’t be too upset. At least they’re trying.

From the dawn of my days with Linux, I have been told to heed warnings that this new system gives. This isn’t Windows any more, the warnings that you receive are no longer benign. My first experience with this was when I had a hard drive that was dying, so I decided to do a fsck. The only problem was that I did a fsck on the root partition – while it was still mounted — read/write. In all fairness, it was my own arrogance that made me add the flag to manually override that protection system. I figured, “what could get worse? The drive is already dying”. I did this with no backups. Three days and 20-something manual superblock hex-edits later, I finally got my data back. The valuable thing from that incident is the experience, though. I now knew how to do manual superblock recreations, which always comes in super handy. I also had learned to heed the warnings of the core developers very well.

Until Saturday. My friend had told me before about how funny the new anaconda argument is: “icantbelieveitsnotbtr”. And now who says these hardcore developers don’t have a sense of humor? Well, I’ve learned a heck of a lot over these last few years, and sometimes I even feel guilty that I don’t give back as much as I should to the open source community. So I figured I’d give their hard work a shot. I’ve heard nothing but glowing reviews about ZFS, and btrfs is supposed to be the up-and-coming big thing in the filesystem world. I figured, at the very least, I could do some benchmarks or something.

So here goes nothing…

I fired up the installer, adding the appropriate flags. I had /dev/sda1 already formatted 50 gig ntfs for Windows. This was to be a dual-booting system. I added a 512-meg /boot, sda2, and a 2-gig swap, sda3. sda4 was to be a pointer to an extended partition table, where sda5 resides, filling up all of the remaining space of 59 gigs. I picked my packages and started the installer. Uh oh, the format of sda5 as btrfs failed. Oh well. Reboot, repeat. Same results, but this time it just died with some non-helpful error code 1. Uhhh. So I tried to boot again. My partitioning table was simply gone. WTF.

I remade the partition table, and figured, “third time’s the charm!”. I selected btrfs as the filesystem for /. If this didn’t work, I was going to give up and use the (now-default) ext4 filesystem. Alas, it worked. I was actually looking forward to using the new filesystem. The install finished normally, and I booted into my new Linux system.

First thing’s first, I ran a full yum upgrade. I noticed something just then. The seek times with btrfs were amazing, read speeds were okay, but the write speeds were flat-out terrible. I’m guessing that’s simply because of all of the metadata btrfs stores, but I honestly have no idea why that is. Oh well, I do far more reads than writes, so this works out well. I then proceeded to use the system for a while. After my used disk space started to get above 25%, I started to get really unexpected results – kernel failures, corrupted filesystem blocks, half-missing extent entries, system bus errors, the whole nine yards.

I did the next logical thing, I strace’d the processes that were dying unexpectedly on me. Worst of all was yum, it would just die with a “Bus Error”. Strace revealed it was hitting massive ENOSPC (no space left) errors trying to open files on the hard drive. df reported the filesystem as exactly 27% full. A google search revealed this as a known bug, but no one else has really had the problem until their filesystem reached over 75% full. So why was mine being the opposite? I theorized that it may have been because of loads and loads of useless metadata across the hard drive. I rebooted (the only thing that fixed the ENOSPC errors) and installed the btrfs-progs package. This package includes userspace utilities needed to create, check, modify and correct any inconsistencies in the btrfs filesystem. I ran a btrfsck, which reported no problems. I read that it may be useful to re-examine my metadata by forcing a balance of it across all drives, even though it only existed on one partition. This is done with the command “btrfs-vol -b /”. Kicker – this operation, after about 5 minutes of heavily accessing the drive, segfaulted. I was left with a half-balanced btrfs system, ENOSPC errors everywhere, and a feeling like this was the beginning of the end.

I didn’t quite want to give up yet, but I also wanted a usable machine. I had read somewhere that kernel 2.6.30 had an insane number of btrfs improvements. On a personal wish, I downloaded the kernel source for 2.6.31 and compiled. Booting into the new kernel was like a breath of fresh air.

SIDE NOTE: kernel > 2.6.30 is awesome. readahead is now merged into the kernel source for MUCH faster booting. There have been a ludicrous amount of improvements, check the changelog on kernel.org for a good list of these changes.

Anyway, the new kernel loaded up just fine. The errors held off for a bit, long enough to make me think they were gone. But then they came back full-force. The balance segfaulted again. The fsck said nothing was wrong. No one had any idea what was going on. I do like to help the open source community, but I simply don’t have time at the present to dig through hundreds of thousands of C code source lines in order to fix a bug this serious. After filing 4 bug reports, submitting probably close to 50 kernel errors, and genuinely giving it my best college try, I was done with btrfs. I rebooted into anaconda and formatted my system as ext4 and went along my way. Yes, I know you can *theoretically* convert a btrfs volume to ext4, but with the structure so far destroyed, I didn’t even bother wasting my time.

Moral? I guess just know what you’re getting into if you decide to give btrfs a try. When the kernel developers say that a feature is experimental, take that literally. There is no warning here, only some sound advice – proceed at your own risk, make plenty of backups, and just generally don’t expect too much of the filesystem at this early stage of it’s development.

Long-winded ideals speech beside, I want to monitor how long the rebuild is going to take in quasi-realtime. Details of the software raid can be found in the file /proc/mdstat. You can cat this file once in a while to see the progress. But that wasn’t ever good enough, too much of that whole keyboard-button-pushing thing in order to get the statistics to update. So I would usually type this in real quick:

while [ 1 ]; do clear; cat /proc/mdstat ; sleep 1; done

It’s just a simple loop that keeps the information fresh on the screen. Well boy did I just find out something today that helps greatly with that dilemma. Enter the Linux “watch” command, courtesy of the procps package. The watch command automatically updates the information on the screen in the command that follows the binary on the shell. For example, the command:

watch cat /proc/mdstat

will update the file exactly like the loop does. But watch does it even better, as it prints a clock in the corner and lets you know how often it is updating. Plus, watch doesn’t make the screen flicker for a millisecond like the shell clear does. The default delay in updating is 2.0 seconds, but you can change that easily via the -n flag. For example, to update every second, as in the custom loop above, try the command:

watch -n 1 cat /proc/mdstat

and be amazed. It would seem at first glance that the delay can be set as low as 0.1 seconds. Pretty neat. It’s an interesting tool, and certainly one that is very useful for these sorts of things. Check out the man page for more information on the command.

Silence the PC Speaker with “rmmod pcspkr”. Make it loud again with “modprobe pcspkr”.

Your roommate will thank me for this… whether he/she knows it or not