Icon Rufen Sie uns an
+49 441.309197-69 +49 441.309197-69

Confidence 2008 Day Two

Posted by Felix Kronlage on Sunday, May 18, 2008

The day started with a nice walk in the sun from our apartment in the jewish quarter to the conference center just on the other side of the river.

Today one of my favourite speakers, Dinis Cruz, gave his talk on OWASP. His talk was actually splitted in two subjects. First on OWASP, then on “ounce F1” which is an enhancement from him to the Ounce Labs Product. Dinis is an amazing speaker, full of energy jumping around on stage.

The afternoon was filled with hanging out on the lawn outside in the sun, talking to various people about binpatchng, infosec in general as well as giving my second interview with Tyrel from Sites Collide. We mostly covered topics like flashable devices, the loss in quality in Software because of easy-updateable devices and of course binpatchng. I will post a link, once Tyrel has published the interview.

The interview I made with Tyrel at last years Confidence conference.

One of the nice things about Conferences like confidence is that you have certain people, like Dinis, that you meet every now and then at conferences and you have these amazing discussions. For me this is one of the best ways to gather new input, collect thoughts and be inspired about playing with and hacking new technology.

The evening was filled with going for food with all the speakers and of course having the obligatory beer!

’till next year at Confidence ’09!


First day at Confidence 2008

Posted by Felix Kronlage on Saturday, May 17, 2008

Back in Krakow!

Like last year, I was invited to Confidence again. I’ve arrived on Thursday (after spending hours on the rollfield of Warsaw airport) and experienced the very warm welcome of the organizers. At the airport I’ve met with Alesio and we were immediately taken to a pub to “recover” from the flight stress. ;)

The first day of the conference was filled with talking to many of the people that you meet at these places, it’s always really nice meeting your friends all over the place. Being at Confidence as a speaker is incredible, since the organizers do whatever they can to make you happy.

Hey, I even got a plate of fruits (thanks Anna), when I did not want the bbq. :)

My talk was the last one, so I was keeping the people from getting to their beer. I talked about binpatchng, which is an enhanced variant of openbsd binpatch.

Stay tuned for more. Both on Confidence second day as well as binpatchng. ;)


Debian openssl vulnerability

Posted by Bernd Ahlers on Wednesday, May 14, 2008

Yesterday the Debian Security Team annouced a serious security issue in Debian’s openssl package. You probably know it already because it’s been around on all news sites.

We just want to remind people on how important this is and how it can affect your systems.

The Debian annoucment1 reads:

It is strongly recommended that all cryptographic key material which has
been generated by OpenSSL versions starting with 0.9.8c-1 on Debian
systems is recreated from scratch. Furthermore, all DSA keys ever used
on affected Debian systems for signing or authentication purposes should
be considered compromised; the Digital Signature Algorithm relies on a
secret random value used during signature generation.

The first vulnerable version, 0.9.8c-1, was uploaded to the unstable
distribution on 2006-09-17, and has since propagated to the testing and
current stable (etch) distributions.

Affected keys include SSH keys, OpenVPN keys, DNSSEC keys, and key
material for use in X.509 certificates and session keys used in SSL/TLS

This actually means that all SSH, OpenVPN, DNSSEC, SSL/TLS session and X.509 key material which has been generated on a Debian machine after Sep 17 2006 is probably vulnerable.

This is quite bad and will create a lot of work for Debian sysadmins.

So if you’re using Debian on any of your systems we recommend reading of the actual security annoucement and follow-ups.

1 http://lists.debian.org/debian-security-announce/2008/msg00152.html


Adventures of buffer cache and I/O bottlenecks

Posted by Felix Kronlage on Thursday, April 10, 2008

Earlier this year, we noticed an increasing stability problem with our main mailserver, mail1.bytemine.net. First we blamed it on the increased load due to the ever increasing volume in spam. We deciced to move the biggest ressource hog, SpamAssassin away from this machine.

Doing mostly processing of text and binary content in e-mails SpamAssassin eats CPU for breakfast. For a bunch of weeks this seemed to have helped quite a bit, but then our problems returned even worse. Thanks to the nice systems data graphing suite symon Bernd noticed that our mailserver was constantly doing roughly 3 to 4 MB/s disk IO. While this does not seem that much at first, in average that meant, that the disks were always busy and data meant to go to disk was waiting around in buffer cache.

Additionally we were now watching the machine with systat(5), where you can watch the I/O stats very nice in realtime.

Around the same time we got a new machine with a LSI MegaRaid controller (MegaRAID SAS 8408, in OpenBSD driven by the mfi(4) driver) and while debugging performance issues with the raid, we figured out that it came with very safe, but slow default settings:

  • all disk caches were disabled
  • the controller cache was disabled
  • the controller was in Write-Through mode

Then we checked the manual for the LSI MegaRaid controller present in our mailserver (LSI MegaRAID SATA 150-4 driven by the ami(4) driver) and discovered, that the same options where present there and assumed they would have the very same defaults. The mailserver was initially deployed in 2005 and until november 2007 never showed any signs of having I/O congestion. Yet it definitely looked to us as if we were having problems with i/o bigtime.

By mid-february we were having resource starvation symptons every other day, with highly sluggish behavior during peek hours. Out of nowhere userland would freeze, the kernel yet still answering to icmp packets.

Once we went down to frankfurt to look at the controller and saw, that the very same defaults were used it became quite clear. Bernd reconfigured the controller to use more sane defaults and within minutes we saw a totally differeny I/O behavior. The following graph showing the days before and after the changed controller settings displays this very well (click images to see large version).

Looking at the week after we adjusted controller settings, the difference in I/O seems enormous. What happened was:

- with the default controller settings data could not get to the disks quick enough, so we constantly dragged pending I/O with us. With the adjusted settings, data gets to the data quickly, so that we don’t start to build up such a congested situation:

We then took some time to look at some of the recored data from last year. And suddenly the picture became all clear. Since November 2007 we had a n increase of I/O of roughly 700%. No wonder, that the “safe” (but slow) default settings didn’t hinder us before, the machine was bored, yet with more and more I/O coming it suddenly became a bottleneck. As the big increase of I/O is reading from the disk, we suspect that more and more of our customers have switched to using IMAP, which does cause a bigger perfomance hit that POP3. The following graph shows I/O over the last year:


thanks scan_ffs(8)

Posted by Felix Kronlage on Sunday, April 06, 2008

While updating machines at our colocation in Frankfurt last friday, Bernd and I hit a pretty harsh bug. Because of changes in the disklabel code, the disklabel of our main customer mailserver got updated by the OpenBSD kernel upon booting with the new OpenBSD kernel. After updating all packages and configurations on the machine we gave the server another reboot. Next the machine got stuck in the first-stage bootloader, complaining that there is no OS to boot.

Quite baffled, we booted bsd.rd kernel just to see that there is no disklabel present. Now, usually you have backups of /var/backups, where OpenBSD keeps informations like this around. Of course, these files were not in the backup (time to blush on my part). Thanks to a hint from our fellow OpenBSD developer Henning, we gave scan_ffs a try. Since scan_ffs is not present in the ramdisk kernels, we compiled it statically and placed in on a USB stick. scan_ffs scans a disk and outputs the partitions it finds. With a bit of thinking, we could then create a disklabel with the correct data. Once we had /var mounted, we could use the saved disklabel there to make sure the label is correct.

The manpage for scan_ffs contains the following advise:

The basic operation of this program is as follows:
1. Panic. You usually do so anyways, so you might as well get it over
with. Just don’t do anything stupid. Panic away from your machine.
Then relax, and see if the steps below won’t help you out. 

Very true. While we did not panic, the best thing to do in such situations is to think and evaluate your options and possible recovery methods and don’t be quick on trying to fix it, likely you will make it worse ;)

Thanks scan_ffs!

Older posts: 1 ... 22 23