bytemine has been a Zarafa Partner since late 2008. Since 2009 we’re a certified partner and have done quite a few Zarafa projects so far.

What is Zarafa?

Zarafa is an Open Source Collaboration, providing:

• Integration with your existing Linux mailserver
• Native mobile phone support
• Outlook “Look & Feel” webaccess
• Stable Outlook sharing (100% MAPI)

A lot more details about Zarafa can be found on their official website.

We’re not only offering various services such as conceptual work, deployment and integration, maintenance as well as 24/7 standby for Zarafa, but also offer Zarafa hosting on our Xen-Cluster located at our rack in Frankfurt connected to Kleyrex Internet Exchange.

Among the customers we’ve successfully migrated to Zarafa are:

• PSA Automation GmbH
• Banking Concepts AG
• Miniatur Wunderland Hamburg GmbH

While Miniatur Wunderland as well as Banking Concepts AG are running on their own infrastructure, taken care of by us, PSA Automation GmbH is taking advantage of our hosted Zarafa.

As I tweeted this morning, news.bytemine.net is shutting down. This is kind of the end of an era, I’ve been providing free usenet access for the last eleven years. The first six years through the usenet node news.hazardous.org, the last five years bytemine hosted the service on news.bytemine.net.

Admiddently news.bytemine.net was never a huge site. We had the Big8, most of the german-language hierarchies plus a few selected others. However non-binary traffic has decreased over the last years and – to be honest – the significance of usenet has decreased much as well. Added to that, news.bytemine.net has been running in auto-pilot mode for the last couple of years, so obviously my interest in usenet has decreased as well. These were a few arguments in the debate on whether to shutdown the service or not. The most significant issue was, that the machine that news.bytemine.net ran on simply uses too much power for the small benefit it brings to a few users (who can easily use one of the bigger sites for usenet), and because interest in usenet has decreased more and more, I did not want to invest the time in moving the service to a different maschine. Because news.bytemine.net ran on a Sun Ultra-10 workstation moving the service to a machine with different endianess, would’ve meant lots of work if the news spool was meant to be kept.

On a side note: Usenet is responsible for me getting into this whole Unix / Linux thing. Back in ‘97 I became interested in usenet and quickly noticed that if I wanted to run a usenet site, I would have to do it on Unix, since I did not want to run such a service on a Windows machine.

[11:43] fkr(news):~ %> sudo shutdown -hp now
Shutdown NOW!
shutdown: [pid 2700]
[11:44] fkr(news):~ %>                                                                                
*** FINAL System shutdown message from fkr@news.bytemine.net ***             
System going down IMMEDIATELY                                                  

System shutdown time has arrived
Connection to news closed by remote host.
Connection to news closed.

Farewell news.bytemine.net.

While we posted a lot about our development activities and our products lately, this is a bit of a different blog post, as I personally review our final move to our new colocation and try to reveal to you some of the things, ususally not visible.

Thursday night we moved the last machines from our old colocation at Nacamar (formery operated and owned by Tiscali), often refered to by us as colo “FFM”, to our new colocation at New Colo (internally named colocation “FRA”). Both are located in Frankfurt and roughly 10 km apart (as the bird flies).

We’ve been at the old colocation for more than five years; quite some time. The times at Nacamar / Tiscali had its ups and downs. When Nacamar took over the operations, qualitiy did go downhill. The colo itself was still nice (almost like a clinic so clean), but phone service and remote hands. doh. As Nacamar had their ups and downs with the colo, so were also my visit there, sometimes better, sometimes horrible. I recall visits, where I expected to spend a few hours there and ended up staying overnight, because a machine decided to trash various parts of the harddisk. Yes, these were the early days of bytemine, back when I was – on the unix side of things – mostly on my own. Things have changed since then by lot – for the good. The move thursday night involved three persons and has been prepared since april. So besides changing colocations and switching to a place that suits our requirements better, we’ve also switched from a rack that contained infrastructure that was in place before all the other byteminers joined to infrastructure that was built up jointly by the current bytemine team. A lot of things become a lot easier, if you have an experienced team around you and that become very visible throughout the last year.

As I blogged earlier this year, we’ve setup our own AS and run now in our own PI-Space. Roughly a year ago, I began looking for a new colocation. Since this april we’ve been evaluating and testing NewColo, but directly booking from Antilo, and while we’ve had a few problems at first, it turned out that Antilo was very good at resolving these and assuring we get the quality we want and need. Before moving the new colocation into production mode, we’ve got another uplink through GHOSTnet, so that we have two independent peers. Aside from that we now have a presence at the Kleyer-Rebstöcker Internet-Exchange.

The first servers we moved then to the new colocation were our Xen Servers, on which we do the rails- and webhosting (Bernd blogged about a while ago) as well as other things like running SpamAssassin instances for spamscanning.

On Thursday we then moved the rest of the infrastructure, like e-mail services etc., to the new colo. The move itself went so smoothly that I can hardly blog anything interesting about it. The whole combined move (taking down, driving to new colo, building up) took roughly 2 1/2 hours and so far we did not spot any fallout.

After finishing the first setup of the moved machines at NewColo I had a chat with Sebastian (Owner and Founder of NewColo) and it is quite clear that a place like NewColo and Antilo is so much more fitting to bytemine than Nacamar. Both are fairly small companies (at least compared to Nacamar and Tiscali), but that is exactly what we need, since we know that problems will be resolved right away and not escalated through different parts of a gigantic enterprise :)

Yesterday we took our own AS live. We’ve moved in with NewColo in Frankfurt, being directly serviced by Antilo. Our AS is AS48991.

We’re currently evaluating the connectivity there, since we need a replacement for one of our racks that is currently co-located at Nacamar, the colo being formerly owned by Tiscali.

Us moving to our own PI-space with our own AS is a rather big, yet exciting step for us, as we will be able to provide even better housing and hosting services as before. With our own AS, we can directly peer with others, having more control about our connectivity. If all works out, we will be moving our frankfurt based services within the next three months to the new colocation.

Stay tuned, we will likely have more news regarding the connectivity of our AS soon.

Today I’ve deployed a new piece of hardware (more on that later in detail ;), however, since I replaced the machine hanah.ol.bytemine.net with it, we have a small gap in our network latency graphs. sorry!

After several months of testing, we’ve now put our VoIP phone system into production! The new official phone numbers are:

Phone: +49-441-309197-0

Fax: +49-441-309197-99

Of course, each of us has a direct extension as well. The SIP-Connectivity comes from Antilo AG.

The whole system runs on a Soekris 5501-70 with Asterisk handling the SIP parts. For our desk we have Snom 370 hardphones, the mobile users use ekiga as a softphone. All phones connected to the main asterisk-server run over our VPN. Incoming calls that can not be handled due to all people being busy are submitted via e-mail to our ticketing system. That way nothing gets lost.

For the fax number we have an extension that hands over the call to IAXmodem which handles the incoming fax together with hylafax+. From there a PDF is submitted into our ticketing system, just like with our voicemail.

Aside the fact that we can talk inbetween our different locations over crypted lines, we now have a very high reachability with minimum effort!

Besides the help we got from Antilo in getting the initial SIP-connectivity running, the guys from Amooma gave us a quick hand, when I got stuck with handing over the incoming fax calls to hylafax. Of course it was a one-line change that got it working, but thanks to the professional support from Amooma it was quickly solved.

Want to listen to some OpenBSD tunes? Dial: +49-441-30919798!

Hooray! Since today irc.bytemine.net is linked to the Subcult IRC network again.

Around couple weeks ago (read this story) the AS the main hub is on, was not announced anymore, thus rendering us unable to reach it.

Since the AS that irc.swissix.ch is on, is still not being announced, us IRC operators have decided to build a new master hub, which is now irc.humppa.ch.

Some trivia along the news: we’re running the Hybrid ircd, and it is about time, I sit down and finish the OpenBSD port of it. One of the nice things with hybrid ircd is, that it is trivial to run the connects to the hub over SSL.

As such, we offer also SSL-based connections to our server. But keep in mind, even using irc with ssl, still allows the admin of the irc-server to peek into memory and read your stuff!

Happy ircing!

So, I come back from a business trip, look at my irc session and am suprised: irc.bytemine.net is not linked with irc.swissix.ch anymore. In the subcult irc network irc.swissix.ch is the primary hub.

First I thought, that our connection at the site where irc.bytemine.net is hosted, had temporary routing problems and tried to convince our hybrid-ircd to relink with irc.swissix.ch. After receiving many, many timeouts I looked a bit further and noticed that I wasn’t able to reach irc.swissix.ch from any german internet connection. Every packet that hit the first uplink router got dropped.

Today it was confirmed on #swinog that the announcements for the AS irc.swissix.ch is on have been stopped due to a Denial-of-Service attack. Once again it has been proofed that very few people (running DoS) are able to get free services killed. However stopping the AS from being announced is the only real solution to the problem. :(

It is not quite clear yet, when we will be able to relink irc.bytemine.net with the main hub.

Earlier this year, we noticed an increasing stability problem with our main mailserver, mail1.bytemine.net. First we blamed it on the increased load due to the ever increasing volume in spam. We deciced to move the biggest ressource hog, SpamAssassin away from this machine.

Doing mostly processing of text and binary content in e-mails SpamAssassin eats CPU for breakfast. For a bunch of weeks this seemed to have helped quite a bit, but then our problems returned even worse. Thanks to the nice systems data graphing suite symon Bernd noticed that our mailserver was constantly doing roughly 3 to 4 MB/s disk IO. While this does not seem that much at first, in average that meant, that the disks were always busy and data meant to go to disk was waiting around in buffer cache.

Additionally we were now watching the machine with systat(5), where you can watch the I/O stats very nice in realtime.

Around the same time we got a new machine with a LSI MegaRaid controller (MegaRAID SAS 8408, in OpenBSD driven by the mfi(4) driver) and while debugging performance issues with the raid, we figured out that it came with very safe, but slow default settings:

- all disk caches were disabled

- the controller cache was disabled

- the controller was in Write-Through mode

Then we checked the manual for the LSI MegaRaid controller present in our mailserver (LSI MegaRAID SATA 150-4 driven by the ami(4) driver) and discovered, that the same options where present there and assumed they would have the very same defaults. The mailserver was initially deployed in 2005 and until november 2007 never showed any signs of having I/O congestion. Yet it definitely looked to us as if we were having problems with i/o bigtime.

By mid-february we were having resource starvation symptons every other day, with highly sluggish behavior during peek hours. Out of nowhere userland would freeze, the kernel yet still answering to icmp packets.

Once we went down to frankfurt to look at the controller and saw, that the very same defaults were used it became quite clear. Bernd reconfigured the controller to use more sane defaults and within minutes we saw a totally differeny I/O behavior. The following graph showing the days before and after the changed controller settings displays this very well (click images to see large version).

Looking at the week after we adjusted controller settings, the difference in I/O seems enormous. What happened was:

- with the default controller settings data could not get to the disks quick enough, so we constantly dragged pending I/O with us. With the adjusted settings, data gets to the data quickly, so that we don’t start to build up such a congested situation:

We then took some time to look at some of the recored data from last year. And suddenly the picture became all clear. Since November 2007 we had a n increase of I/O of roughly 700%. No wonder, that the “safe” (but slow) default settings didn’t hinder us before, the machine was bored, yet with more and more I/O coming it suddenly became a bottleneck. As the big increase of I/O is reading from the disk, we suspect that more and more of our customers have switched to using IMAP, which does cause a bigger perfomance hit that POP3. The following graph shows I/O over the last year:

While updating machines at our colocation in Frankfurt last friday, Bernd and I hit a pretty harsh bug. Because of changes in the disklabel code, the disklabel of our main customer mailserver got updated by the OpenBSD kernel upon booting with the new OpenBSD kernel. After updating all packages and configurations on the machine we gave the server another reboot. Next the machine got stuck in the first-stage bootloader, complaining that there is no OS to boot.

Quite baffled, we booted bsd.rd kernel just to see that there is no disklabel present. Now, usually you have backups of /var/backups, where OpenBSD keeps informations like this around. Of course, these files were not in the backup (time to blush on my part). Thanks to a hint from our fellow OpenBSD developer Henning, we gave scan_ffs(8) a try. Since scan_ffs(8) is not present in the ramdisk kernels, we compiled it statically and placed in on a USB stick. scan_ffs(8) scans a disk and outputs the partitions it finds. With a bit of thinking, we could then create a disklabel with the correct data. Once we had /var mounted, we could use the saved disklabel there to make sure the label is correct.

The manpage for scan_ffs(8) contains the following advise:

The basic operation of this program is as follows:
1. Panic. You usually do so anyways, so you might as well get it over
with. Just don’t do anything stupid. Panic away from your machine.
Then relax, and see if the steps below won’t help you out.

Very true. While we did not panic, the best thing to do in such situations is to think and evaluate your options and possible recovery methods and don’t be quick on trying to fix it, likely you will make it worse ;)

Thanks scan_ffs(8)!