Today, I had an interesting experience. I had to mail a banking form in paper due to local banking regulations. As there is no post office on my way to my work place and none nearby I had to wait until the weekend to do so.

I started around 11:05 AM to the nearest post office. I had to travel using LRT to the post office. I reached the post office at 11:20 AM. I was in the queue until 12:00 PM and was serviced in 3 minutes. The cost of the mailing was S$0.50 while the cost of the journey was (S$0.69 + S$0.69) to and fro. The efficiency of the transaction – wait until the end of the week, wait in queue for over 40 minutes, transport cast ~3 times the cost of actual transaction

Just to tickle our thoughts. Isn’t it time to go fully ‘e-*’ ?

I am writing this from India (GMT + 5.5) and roaming with my Singapore (GMT + 8.0) mobile phone. I noticed something very strange. In all my mobile phones, under “time and date” settings, I usually setup time zone as local time instead of GMT +/- X and Network operator time as auto-update. By this way, I hope to always have the current local time updated automatically. Everything seem to have worked fine until now…

In my Nokia E71, I noticed a time skew of around 2.5 hours (GMT+8-GMT-5.5 ?) when I switched on the phone after landing in the local airport. I thought the time synchronization hasn’t happened yet and waited. However, instead of correcting itself, the skew kept on becoming big and eventually it was 7 hours ahead of the local time. All my alarms, reminders would go off in the middle of the night waking us up even before we have actually slept

I turned off and on again my phone couple of times and still the problem would not go away. Now, it was showing time as per GMT+7 time zone. At this stage, I wondered if it could be the operator issue and looked at my other mobile (iPhone) which was also setup to auto-update from mobile operator. There were no issues and the phone synced well and it always showed the correct local time.

At this stage, I started to wonder what could be issue ? there were 4 combination possible – home mobile network, roaming mobile network, phone model / manufacturer and time zone differences. I did a look up table. My Nokia E71 is on Starhub Network in Singapore, and was connected to the Vodafone IN network automatically and was showing GMT+7 time zone time. My iPhone is on SingTel in Singapore and was connected to the Airtel network automatically and was showing the correct GMT+5.5 time zone time. My primary suspect was the mobile operators’ algorithm in sync roaming mobile phones. Initially I thought, perhaps my mobile phone is in GMT+8 so there was a bug in time synchronisation and the network was pushing GMT+8+localtime. However, this was not the case as well. It was pushing down GMT+7+localtime. This left me with the only possibility of Singapore being registered as GMT+7 in the network operator. Incidentally, the natural time zone of Singapore is also GMT+7 (see natural time zone of singapore). Any way, all is theories and I don’t have any concrete solutions.

If you have experienced such issues and know of an possibility, do drop an comment.

This is part 2 to my earlier post “A Linux crash, burn and recover experience“. I had mentioned in there that I plan to use a Virtual Machine (VM) strategy for easy backup and porting in future. However, the reality forced me to abandon the approach. It could also be that I don’t know the trade secrets read on…

As you know, I am using Red Hat Enterprise Linux (RHEL) for my server for the last several years. After recovery I decided to continue using the same OS. Therefore, the natural choice was to use xen as the virtual machine monitor. I used this server downtime to add more RAM totaling to 4GB and now the entire root partition is also protected by RAID 5.

During the OS installation, I selected virtualisation to be installed. Basically, this means xen-enabled linux kernel would be installed by default. Everything seemed to be ok and I continued on to firstboot. After completion, I logged into the system and from a comman prompt, I did cat /proc/meminfo and to my shock the first line was MemTotal: 3753984 KB (3.58 GB) a whopping half-a-gb gone missing. This is no-go as then the VM would be running much slower with fewer resources

In theory, 2^32 = 4096 so the entire 4GB of RAM should be addressable by the OS as physical memory. If there are any exception then it should be rounded-off to some big number but here 3.58 GB doesn’t sound like it either. With no choice, I decide to re-install with the 64-bit version of RHEL with the thoughts that 64 bit might recognize the entire 4GB. I was a bit hesitant about compatibility of the application software with a 64-bit OS. However, there was no other choice except to take the risk.

As with 32 bit installation I selected virtualisation and this installed the xen-enabled 64-bit RHEL kernel. After firstboot, I checked the total memory and got another shock. The memtotal now is 3708928 KB. This is 45,056 KB less than 32 bit xen-enabled RHEL kernel. I was confused as to why a 64 bit kernel recognise less RAM compared to 32 bit OS. I googled around and didn’t find a direct answer. At the same time, some one was suggesting that PAE kernels recognise 4GB or more RAM. Also, I remembered about my another system where I have 8GB of RAM and running Ubuntu 8.10 64 bit with kernel version 2.6.27. Cat /pro/meminfo showed 8181976 kB (7.8 GB). I use vmware workstation and not xen as VMM in that box. I then did a yum search pae to select an appropriate PAE kernel. I discovered that PAE is a 32-bit kernel feature and is not available available for 64-bit kernel. It made sense; why will we need physical pages extension when 2^64 is itself a huge number. Now, my second choice is to use continue using 64 bit kernel (risking any future software incompatibilities), lose ~50MB more than corresponding 32 bit kernel. It didn’t add up to be a good choice. At this juncture, I thought about PAE 32 kernel with Xen might do the trick. I started my 3rd install of RHEL for the day.

The 3rd choice was to install a PAE 32 bit kernel, verify the RAM is fully accessible and then turn on visualisation. The installation was smooth and after first boot, as with earlier attempts I did cat /proc/meminfo and hurray I got MemTotal: 4149928 kB. I then did a uname -a and inferred that by default (without virtualisation selected at install) the anaconda system installer installs PAE enabled kernel if it detects 4GB or more RAM. This was a welcome note. Now, I again did a yum-search for xen and installed it and rebooted it. On the grub bootup screen I became wiser and learned the following

• 32 bit kernel + xen = okay
• 32 bit kernel + PAE = okay
• 32 bit kernel + PAE + xen = NOT POSSIBLE

The above summarises it all. If you want PAE, forget XEN; if you want XEN forget PAE. Now, I have to make a hard decision whether to stick with my VM strategy and be constrained with only 3.5GB of RAM which could mean that the guest OS have only a maximum of 3GB RAM OR forget virtualisation and use all 4GB of RAM. I then factored in my hardware features (not to be generalized). The server is more than 4 years old and does not have vTX (hardware virtualisation) feature. I learned that this would make the VM run slower and compound this with the fact that 0.5GB is not there, this might result in a very slow VM or may be it won’t be noticeable. I was already tired for the day and wasn’t interested in doing another combinatorial experiment this time with VM variable. Therefore, I decided to abandon my VM strategy and be happy with the increased resources. As a take-away from this experience I have compiled who would have made the memory missing. I actually don’t know the answer, if you know please drop a reply, thanks !

There are several suspects in my list

1. Linux kernel ? RHEL version 5 uses 2.6.18.x as the base kernel. I am wondering if the Linux kernel has been fixed some time after the above kernel release. The reasoning being, Ubuntu 8.10 is able to see all the 8GB RAM (of course it was without xen)
2. Xen ? Is it that xen-enabling the Linux kernel makes the memory vanish ? If so, what do we conclude of xen technology in general ? do the peers face the same missing memory issues ?
3. Xen + PAE could have solved my problem but they were not compatible. Why ?

I had a chance to beta test the upcoming release of Windows OS, version 7. I didn’t have any expectations for the Windows Vista successor; I have downgraded all my factory-installed Vista systems to Windows XP within 15 minutes … I was pleasantly surprised with what Microsoft had done this time though :O

With no expectations and in fact negative expectations, I didn’t want to destroy any of my trusted Windows XP systems . I had a 7 year old laptop (P4 and 1GB RAM) which I came with Windows XP and now serving as test machine for anything that came new (Fedora 10, Ubuntu 8.10, Open BSD among others). I decided to lose my Fedora 10 (it rocks btw) and risk with that Windows 7 will turn out to be.

The installation was very smooth. Very few questions being asked and the time it took was also reasonable given that the hardware is very old. When I rebooted for the first time after installation, I noticed that the startup time is very fast (< 20 secs ?). You should read this number taking my hardware into picture. Therefore, I would assume that it would be much faster in modern systems. Once I logged in I noticed that the system was much more smoother and more responsive that Windows Vistas. I would even say that on the same hardware, it was probably more responsive that the original OS, Windows XP.

I didn’t spend much time with the OS but in the quick 2 hours or so, I concluded that Windows 7 would be good product and worth a try. Let us wait and see. In the mean time, for more information click on Windows 7 .

One of my trustworthy RHEL servers which was up (virtually) for 4+ years and served thousands of users over-the-web recently crashed. As data in the server was critical, we had to undertake a salvage operation before re-install. This post is my experience of recovering the server’s data and also some of the lessons learned and new strategies evolved for configuring future resilient server…

Since 2002, I have been volunteering some of my free time to help out the local red cross unit in Singapore. The story begins somewhere in 2004 when we decided to procure a server for running some of the internal IT services used for blood donor recruitment, recognition, retention and recall. Over time, the server got loaded with more and more services and it was beginning to look like a crash could mean a lot of downtime.

It is not the case that we didn’t have any backup. With the IT budget that was available, I had configured a soft raid of level 1(replication) for two of the critical partitions of the server – /home and /var. All was well until last year when we began taking regular backups of important configuration files and storing them remotely conforming to business continuity best practices. Late last year, I decided to take local full backups.

The first sign of trouble seem to have been visible last year. As part of the backup process, we wanted to use an external usb hard disk. However, the server did not recognize the hard disk and create the respective /dev/ entries although dmesg was showing the usb device as being recognized. with no /dev/ entry the hard disk could not be mounted. I concluded at that time that it could be because of the mismatch between old usb1.x port and the new usb2 hard disk. I wasn’t very convinced with that hypothesis but I didn’t have any alternate theories as well. Now in hind sight I know this hypothesis is not right.

One fine day, the system console stopped responding normally and there was huge delays between echos for key strokes. I suspected a run-away process with potential memory leaks so just hit the reboot. On POST screen, BIOS reported that one of the hard disks is reporting a future failure (a SMART feature of new generation hard disks). Following this, the kernel refused to boot and dropped me into the maintenance prompt. After running fsck for well over an hour with hundreds of inodes being orphaned and added to the lost+found list, finally the server started booting…

During the startup, two daemons – httpd and mysqld did not come up owning to some missing files. These two services are key to all our web apps and thus the server came to a stand still. Coupled with this was the fact that the server stopped at console-login and did not start up in X although the runlevel listed the server being in 5. I decided it was not worthy to continue using this server as what all was broken could not be cumulatively listed let alone being fixed.

I didn’t even want to reboot since my intuition suspected that the server might never come up again. With the local external hard disk not recognizable the only option was to do a remote backup. Luckily, ssh was up and I was able to connect from outside to the server. However, I soon found out that scp was not found in the PATH and not on the filesystem. I did a which sftp and it was found. With a sigh of relief, I started to connect from outside. The relief was short lived because although the control connection could be established the data connection was not being created. There was no firewall and no other network blockage. I thought the missing scp could be the cause. Therefore, it seems that the only option was to dismantle the server and recover the data.

Luckily, one of my friends (Kumar c/o thoughtsware) accidentally discovered that although pull technique was not working, push technique using ftp after ssh-in was working. Armed with this, I scrambled to setup with a ftp server to be the receiving end. By setup, I mean it, as I don’t usually use ftp because it is insecure. I started to move data one file a time from the server to this ftp site. The data was flowing but at a very slow rate. After a few days of data transfer, I decided it was time for the surgical data recovery on site. Another good friend and a mentor, Harish from Red Hat, offered to help out with the recovery.

First, I wanted to test out if a flash drive was readable on the server. To my surprise, the crippled server was able to recognise the fat32 filesystem drive and created the necessary /dev/ entries. At this stage, I concluded that there was nothing wrong with the server hardware and something was not okay in the software when I earlier tried a ext3 formatted disk. I started copying the data into the thumb drive but very soon the data transfer halted with the process becoming non-responsive as well. We guessed the process may have hit a bad sector. We decided it was time to shutdown the broken OS and try our last alternative.

Our plan was that we boot the server using a live cd, Fedora 10 in this case. Then mount an external hard disk and the internal hard disks and dd out the partitions and attempt a system recovery using the binary img files. The only unknown factor being the partition was soft raid partitions and we haven’t done a recovery before on this filesystem type.

We booted the server using Fedora 10 live cd and successfully mounted the internal hard disks. Next, when we inserted the same external hard disk into the usb port, Fedora mounted it without any fuss. At this point, I infered that earlier attempts to mount it may have failed because root filesystem may have developed bad sectors at exactly the same spot where the entries are created.

Without any delay, using dd we copied all the 4 raw partitions (2 raw partitions per raid 1 partition) to the external hard disk. Our attempts to copy out the boot and root partitions did not succeed as dd complained of missing superblocks info. After copying the raid partitions out, we decided to try our luck with booting the server back into the original, now broken OS and try out copy out configuration files by mounted a thumb drive as earlier. However, on reboot the server didn’t go anywhere beyond grub prompt. The server was totalled.

Armed with the dd created partition binary files, I mounted the hard disk containing them in another Fedora 10 machine. Then I used losetup to mount the binary files as a loop back device. I then used mdadm to combine the respective component partitions and mounted them as a md0 device. Following this, I mounted this md0 device under a directory and was able to see the files. Finally, I performed a recursive copy of the data in these mounted partitions into a clean ext3 partition for restoring. In the end, we could not recover a few configuration files like apache, networking and such. However, majority of the data was recovered.

I am now awaiting the hardware vendor to replace the parts and hopefully the server would be online in a few days.

Based on this experience, new knowledge and technologies, our new server would be designed to be more reliable than the current one and in case of another crash, should be easier to recover as well. Our idea is to use virtualisation and run our current server in a VM. By this way, we can easily backup the entire server and also restore from backups more easily. Secondly, we can enforce security policies on the host OS and also on the guest OS while monitoring all the network traffic flowing to VM at the host level. The host will be responsible for providing a abstracted and redundant hardware layer to the VM taking care of the raid and other hardware specific features.

Feel free to comment on our recovery experience and the new server design strategy as well

Posted in Computer Science ]]> http://sptkrishnan.wordpress.com/2009/04/02/a-linux-crash-burn-and-recover-experience/feed/ 3 sptkrishnan http://sptkrishnan.wordpress.com/2009/03/30/did-my-20th-blood-donation-on-saturday/ http://sptkrishnan.wordpress.com/2009/03/30/did-my-20th-blood-donation-on-saturday/#comments Mon, 30 Mar 2009 05:06:09 +0000 S P T Krishnan http://sptkrishnan.wordpress.com/?p=140 ]]>

the heading says it all…

I started donating blood in April 2002 and until March 2009 I have donated 20 times. I am not sure if I should feel proud about it because in theory I could have donated up to 28 times (4 years a year is allowed in Singapore for whole blood donation). However, My target was 21 (or 3 times a year). So, I guess I am short of 1 time now and that amounts to 95% of goal.

I am hoping that I will be able to donate 5 more times this and next year to reach 25 in 2010 and get a silver medal

Exactly, 8 months ago to this date I had a chance to visit the Museum of Science and Industry in Chicago, United States. The exhibition was one of the largest I have ever been to. It featured exhibits from multiple disciplines of Science and was lively, interactive and very well maintained. Even with a few stop-by’s it took us ~6 hours to cover it. I recommend a visit to the museum if you are in Chicago.

One of the exhibits that captured my interest was on computer criminals…

…the museum has used the opportunity to educate the public on the intricate difference between the words “Hackers” and “Crackers”. While, even to this date, many IT publications refer to computer criminals as hackers, I am happy to note that the exhibition has done its part to help to clarify the  difference. The following tells it all, thanks msichicago .