zeroxcool.net Things I like to talk, think or dream about, in every sense of the word

Verhuizing van servers

Volgende week staat er weer eens wat moois op de planning voor het werk. We zijn een beetje uit ons jasje gegroeid in het datacenter Schuberg Philis. Na vier racks vol te hebben gekregen hadden we een vijfde rack nodig. Aangezien dit niet geleverd kon worden in dezelfde ruimte werd ons een rack op de bovenverdieping aangeboden.

Logistiek gezien was dit sowieso een probleem. Een tweede probleem is de prijs van de stroom geworden. We moesten servers over meerdere racks gaan verspreiden, dus is er voor gekozen een volledige rij (11 racks) af te nemen. Dit houdt in dat we a.s. weekend de boel van onderen naar boven gaan verhuizen. En dat is eerlijk gezegd niet niets. Kortweg 130 servers van 'ons eigen' en 40 servers van klanten. Een hele operatie dus, die goed gepland en gecoördineerd moet worden. We gaan met vijf man sterk vrijdagnacht (23:00u) richting Amsterdam om het het geheel in gang te brengen en hopen zaterdagochtend rond een uur of 9:00u klaar te zijn.

Als m'n logje zaterdagochtend nog te bereiken is dan is het goed gegaan. Tot logs!

Murphy comes around Tiscom’s e-mail system

Pfff, finally an update. Have done a lot since last month. Let's talk about the split of our first e-mail server. That didn't went very well the first two times (saturday night and firday night). The first time we crossed by that about 15 domains had mailboxes bigger then 1Gb. Imagine that an average e-mail is about 2k, so that there were 1G / 2K = 536.870.912 files into one directory. And that these were also very fragmentated (because mails are not flushed to the disk beyond each other). So transferring of these directories took too much time (about one hour each). So we skipped the whole thing to a week later.

That friday night, with all the courage we had we tried again (in the meantime we had trashed those - big - e-mail boxes). Almost everything went well, only some particular domains didn't work well. The next day our support box was filled with lots of e-mail about slow POP connections etc. Also our Mail eXchangers (that are the servers in the MX records with SPAM- and virusscanning) were kind of overloaded. After seeking for the problem for almost three days we found it. The harddisk in the new POP-server was really slow (15MB/s at idle) so we had to go another night to replace the new POP-server with a good new server . This was planned for Saturday afternoon (there was good weather so the server weren't that busy). But when I finished setting up the new POP server, I got an SMS from my colleague Freek that the 'real' POP server was down. I logged in and saw that the filesystem was mounted read-only. Fuck, the harddisk crashed 16 hours too early .

So, we had to prepare ourselfs for another night datacenter. The harddisk was fully lost so we had to use the backup from Friday morning, so people who didn't fetched their e-mail this day have lost e-mail received that day. Nothing to do on that. After unpacking the backup we were able to restore the new POP-server (pop03) so that everything worked well. With solving this problem the load of the MXs also lowered. So the slow/damaged harddisk was the originator of this problem .

Enough about the Tiscom e-mail system . I'm back at school since the 31th of August. I still have to come into the 'one-hour-rhytm', but that will happen within two weeks. After that everything must go fast and the next vacation will arrive soon .

Also football is going fine, I'm playing in our fifth team and sitting on the bench at the first team. Speak you hopefully a bit sooner .

POP proxy project almost finished

Not very much to tell. The POP proxy project is almost finished. All the domain transfer scripts, the domain list transfer scripts etc. etc. are made. But we're still waiting for the new mailserver, where half of all the domains at the first mailserver will be moved to. But it should be there before saturday. That night we are going to replace everything. Move half of the accounts to the new server, move the servers fysically to a new rack and to install the POP proxy, that is. I'll share the 'transfer-domain-to-other-vpopmail-server' script with you on my qmail page, some days .

The DigiMatrix project isn't quite active yet, as I'm working on the e-mail change and Harmbox, an analystic tool I've created during my internship at Bronkhorst Travel. That all takes a lot of time. And there's also the training of my football team. We've had about 4 training sessions by now and in one hour the fifth one. My condition was really good, I wasn't the first one gapping after a run exercise . Also goalkeeping went very well, had lots of difficult balls, without even a keeper training session. That's quite some years ago that happened last . Well, everybody, have a nice vacation. Till the next log!

The Tiscom Hosting MX system

Not quite an active log, huh. Time to change, I'm not going to tell you what I've done and what I didn't, just anything I _would_ like to talk about. It was just annoying to tell everything I've done this week, so there wasn't a motivation to log. Well, now there is.

This week I'm having a week of vacation. Well, vacation, let's say, not going to school. Because I had to do lots of things for school this week. We're working on a GPS program (on PDA and desktop) to make it possible to set out walkings with a GPS device. Quite interesting project, but it takes lots of time. We're still running on schedule, I hope in a few weeks I can show the final results.

At work everything is also fine. We've introduced - after months of testing - finally our virus- and spamscanning - mail exchangers. Our mail system is now much more reliable to outages of one particulair server. I've ASCII-arted an impression of the current situation:

	MX1	= mx1.mail.tiscomhosting.nl
	MX2	= mx2.mail.tiscomhosting.nl
	POP01	= pop01.mail.tiscomhosting.nl

	 /----------------\				/----------------\
========\|      MX1       |				|     POP01      |/==========
inet in	||   virus and    |========\		/=======|  POP archiver  || inet out
========/|  spam scanner  |        |            |	|		 |\==========
	 `----------------'        |            |	`----------------'
				   \[ internal ]/
				   /[ network  ]\
	 /----------------\	   |		|	/----------------\
========\|      MX2       |	   |		|	|     POPxx      |/==========
inet in	||   virus and    |========/		\=======|  POP archiver  || inet out
========/|  spam scanner  |				|		 |\==========
	 `----------------'				`----------------'

As you can see, mail only comes in via the two MX servers. They scan all e-mail on SPAM and virusses. Virus e-mail is immediately deleted, SPAM e-mail otherwise is being tagged with an striking subject name. After that it is forwarded through the internal network to the POP-servers, they store all e-mail for clients and give them out on request. The internal forwarding is done by SMTP and the POP-servers only allow SMTP-traffic from the internal network.

Well, if one of the MX servers is down, there's no problem, the other one will still receive incoming e-mail. If one of the POP-servers fails, the mail is still being queued at the MX's. They will deliver everything when the POP-server is comming back up. The only remaining problem, when a POP-server is down, clients are unable to download their e-mail, that shouldn't be a big problem because the POP-server didn't have a higher downtime than a half an hour last year. Nice solution we still think.

Tomorrow again a football match.