Recently in Behind the Curtains Category

Certainly not, if you store credit card information or passwords in clear text on the servers. Recent data theft disasters have shown, that it is not enough to operate a "secure server" and leave all customer's information unencrypted on this server. Because if you think your secure server is invincible, all your customer's data is at risk, the moment it turns out that the secure server is not as secure as you thought.

What's even worse, your customers have entrusted you with their data believing that operating a secure data center will be sufficient to protect their personal data from falling into the wrong hands. It's time to destroy this false belief.

Almost everything you'll learn to know about why you can trust an online shop or an online service provider boils down to the fact, that they make every possible attempt to secure their servers in the data center with all available bells and whistles of modern technology. But there is very little information - if any - about how they treat your data when it is stored on their secure servers online.

Of course I honestly value every effort to make online servers as secure as possible, but on the other hand I am convinced, that securing your online servers alone is not good enough. Let me explain.

A secure server is not invincible

Today most online services use a scripting language and a database server (think PHP plus MySQL for instance) and sensible information goes into a database. Usually the access to this database is given to every program that knows about the database password and normally runs on the same computer.

   It's a complete miracle to me, why after a default installation of some online shop applications the database password is stored in a file that could be read by any user on the system. It seems to be a commonplace belief that the server running the online shop is invincible, and therefore it poses no problem to store a database password in clear text without any additional protection. As we should know by now, secure servers are not always invincible and storing customer data unencrypted is a very bad idea. Not every online service is that careless about database passwords and it's easy to restrict access further, but today it seems to be the norm to dump the responsibility for the protection of customer data onto the administration personell in the online data center.

I wish to make the point, that we have to create online services in a way that if something goes wrong - despite the care to prevent this - customer data inside the online service is still protected against exploitation by intruders. And I'm convinced that such a protection is not only possible but absolutely essential if customers want to use online services securely.

Security and Convenience - Choose one.

As a consequence I'm sure that we have to correct another common misconception. You cannot have both, security and convenience in the online world. Forget it, you have to dump your notion of an easy, hassle-free, automatic and secure online service. Choose one or the other, you cannot have both.

That does not mean that secure online services have to be a pain in the neck but it certainly means that some procedures that have been introduced to make a customer's online experience "smooth", "seamless" and "easy" have to go if an online service that stores credit card information will ever be secure. Without knowing why, online customers will always think that some (unknown) data center professional is the right one to fight security problems, because after all they are in charge to make sure everything runs well. So it's vital that the ordinary online customer knows why some features of online services cannot remain the way they are today.

In their efforts towards making the online customer's life as uncomplicated as possible some vendors have started to encrypt their customer's sensible information. That's fine. But if you store the encryption key in a file next to the encrypted database, it's a little like locking your door and hiding the key under the door mat. This clever idea was born out of necessity, of course. As online customers demanded automatic payment from their online services there had to be a way that a program on a secure server could access the encrypted data (CC info for instance) without the intervention of a human being. The idea of storing the encryption key in some way on the server had been invented originally to ease the pain of the online customer.

If a customer wants to sign up to a service in the middle of the night (very convenient) the customer's data can either be stored in clear text (a bad idea) or, if it has to be encrypted, the key must be available to the web server program on the secure server (another bad idea). Surely, if you go fighting a bad idea with another bad idea, you'll never end up with a secure online solution.

Before you come to the conclusion that there cannot be a secure online service, let me tell you what I've learned during the last couple of months, while I tried to learn something from the data theft disasters and created a secure online service for small businesses, the secure online bills.

Secure Online Bills

An online service has to protect a customer's data even if there is an intrusion into the secure server, this is the fundamental principle that determined the design of my online service. It's really not easy to follow this principle at all times, because while I coded the system, I tried not to burden the user with avoidable inconveniences. But writing every single line of code myself made it clear to me that there have to be some decisions that will make the user's life a little bit more complicated. And it is inevitable, if the system should be secure.

I know, the last thing a customers wants is complications, "easy" is the marketing word not "complicated". But it became clear to me that there had to be more human intervention in the process if it should ever become more secure. So it is not complication that is inevitable. It's the introduction of the human factor into the process that makes it more reliable and secure.

What we need is more human intervention and less automation. I'm sure that if you begin to see, why it is necessary to rely on the informed decision of a human being instead of an automated web server process, the loss of convenience will become totally irrelevant. Making online services more human is the way to go, that's what I've learned from coding a secure online solution.

For instance, if you sign up for an online service and expect your login details to arrive via email within the next few minutes, your data cannot be secure, because the encryption key must be somewhere under the door mat. This is fine for demonstration purposes, and I use it myself for the secure booking service demo, but it's not good enough for the real thing, where customers rightly expect their information to be protected.

An online service cannot be secure when signing up is a matter of seconds. In fact, the setup of a secure online service requires manual work (of a human) on a server and of course some communication between the interested customer and the responsible person at the online service provider. The reason is simple, if the encryption key cannot be hidden under the door mat, it must be entered into the system from the outside in the moment when the encrypted information is needed. Using the system in a secure way requires preparation that cannot entirely be automated or, let me say, should not be automated at all, if you don't wish to encounter unpleasant surprises along the way.

What about lost passwords ?

Storing passwords in a database with the key under the door mat, no! Once the system is secure the database can only be re-encrypted with a key coming from outside the system. If you think the secure server should be able to reset your account easily, you're wrong. This would only be possible with a second emergency key stored on the server under a second door mat. No, the solution is to contact the service provider and make him use his emergency key which is safely stored outside the system to re-encrypt the database. That means, you'll have to wait until a human being does something useful for you. Is that too much of an inconvenience? I don't think so. Don't expect a server to do it instead. Get rid of this belief.

And if that takes time and costs money, be happy that your problem is being taken care of securely. And stop searching for a dirt-cheap, automatic way to have your negligence corrected without having to talk to a real human being.

After having installed an open source online-shop software on a VPS I had to suffer a hefty delay following the login as administrator until eventually the control panel appeared on the screen. Despite of this admin login problem the software ran fast and responsive, but the admin login, which would normally only take 2 seconds, took more than two minutes to complete.

Possible causes for this problem are manifold, some misplaced configuration option, a software bug, a missing software component, many things are conceivable. What raised my suspicion was the fact that this problem seems to occur only at the administrator login, well, I hadn't created new users by now. It is likely that the problem occurs for new users as well.

The Hunt

Actually there were two separate delays that cropped up after I typed the admin password, with a little bit of activity in between. It looked like a time-out, so I shut down the VPN's firewall and the problem was gone. At this point it was clear that some network activity took place which normally was blocked by the firewall. To find out what was going on I engaged a wonderful network analysing tool called wireshark or tshark to capture the network packets after login. It turned out that the VPN initiated a http and a second https connection to a server within the domain of the online-shop's original vendor.

I know it's only me who tends to think of a possible backdoor, a software "calling home" to report something, in such a case. But at this point I wanted to find out what was going on behind the scenes. Out of sheer curiosity I started to poke into the source code to find more informative evidence. As you may imagine, looking for "http" in the source code revealed tons of references that were mostly inactive links to the vendor's homepage. More extensive filtering brought a function "load_xml_file" to light that was used to download a file that contained only innocent version information in XML format that could as well be part of the distribution and stored locally.

Benefits of Open Source

The vendor had decided to download this file to make sure that the online-shop software will automatically become aware of a new version once it is released. Of course this is a legitimate intention, but it would force the shop user to open outgoing connections on the server machine to avoid the timeout penalty which could open up another can of worms for other applications. I decided to change the source code to load the information from local files instead of the vendor's homepage and turned on my restrictive firewall again.

This is exactly the flexibility and reliability one gets with using open source software which would never, ever be possible if you used proprietary solutions instead. People often say, nobody looks at the source code, which is true for many open source programs, but with proprietary products you would not even have the chance to take the approach described above, because you are at the vendor's mercy to accept what the program is actually doing.

The freedom to change the code is a benefit that could possibly not be overestimated.

On the last day of the year my email stopped coming in. You may have read about my approach to fetch my email using a secure tunnel that uses SSL certificates in addition to a password to access my email. Well, on the last day of the year my ROOT CERTIFICATE, which I use for Kerry Linux, had expired after five years. Time flies by.

As I had other plans for the days ahead I thought just to renew the root certificate to buy time, but it seemed that my attempts to renew my root certificate did not result in a new usable certificate to replace the old one. My user certs, which are not up for expiration yet could not be reanimated with a quick fix like that.

After a while I thought, there is a reason for that and I began to think about root certificates more thoroughly. In the past five years we've definitively seen the crackdown of MD5 and SHA-1 is not invincible, too. Would it not be prudent to increase the key length and to use a more secure (i.e longer) hash and go through the trouble of creating a new root key and issue new user certs? I decided to go along that route and created a fresh new CA root key with 4096 bits for the Kerry Linux Certification Center. Although my openssl software does only permit using SHA-1, which is a pity, I felt content and everything was up and running for me in an hour or so.

Re-Animation of the old ROOT KEY

After a while I began to wonder if it was possible to reanimate the old key and out of curiosity tried to explore the way to do it in more detail. I googled and found this nice posting from Arsen Hayrapetyan which led me to success. My former attempts to recreate the old certificate always led me to the following error message when I tried to verify a user's certificate::

Unable to get the issuer certificate? I supplied it in the command, but it didn't work out as planned.

So I followed Arsen's hints and created a testbed for an experiment, where I set the serial number back to 00 and emptied the file "index.txt" so that my new certificate could inherit the properties of the old one including its serial number. Then I created a new certificate request based on the old root certificate "cacert.cert" and used this new request to sign it with the old key.

The result was a new root certificate "cacert_renewed.pem" that verified my old user certs perfectly.

It's good to have an alternative, isn't it?

The XEN hypervisor uses big files (a couple of gigabytes) as filesystem images for virtual machines. Unlike other virtualisation solutions XEN does not impose its own internal structure on the image file. The big file simply has to contain an ordinary ext3 filesystem and, optionally, a partition table just as if it were a real hard disk. The ability to use big files as hard disks comes in handy if you are running short of space on your main hard disk. With an external hard disk you should be well prepared to run a number of virtual machines as big files. However, having the filesystem of a virtual machine in a big file raises the question of how to boot the virtual machine. Essentially there are two options to do that:

1. Provide the VM's kernel and the init-ramdisk, which are usually stored inside the filesystem (in the /boot directory), as separate files together with the big file, and modify the VM's configuration to use them.

2. leave the kernel and the init-ramdisk in the big file and provide a working boot sector that accesses the kernel inside the big file, using the native XEN pygrub bootloader to start the virtual machine.

Both options require that the big file must be associated with a real, special device file (i.e /dev/loop0) in order to create a filesystem on the big file. While for the first option it is sufficient to simply connect the big file with the loop device, using the "losetup /dev/loop0 bigfile" command, the second option is much more complex, as the big file has to be partitioned like an ordinary hard disk before the filesystem can be created.

For the rest of this article we will focus on the second option which is much more appealing as everything is kept inside the big file. I will show you how exactly the big file is turned into a virtual hard disk and how you can access and modify the information stored in the virtual machine's own filesystem.

Getting Partitions And Filesystem Sizes Sorted

Our journey through the big file's internal structure naturally begins with the creation of the big file.


As a second step we use this chunk of 4141875200 bytes to act as a hard disk and try to partition the bigfile as usual:


As expected, the fdisk program throws a number of error messages at us, because we have given a big file instead of a real hard disk to the program. But let's see how the fdisk program recognizes our new hard disk in detail
.
Obviously there is no partition table yet, but the program assumes that the big file represents a hard disk with 255 heads and 63 sectors of 512 bytes data each. Every cylinder of our virtual hard disk is made of 255 x 63 x 512 bytes = 8225280 bytes which represents the units in which we can chop the hard disk space into partitions now. All in all there are 503 cylinders in our virtual hard disk which makes a total of 503 x 8225280 bytes = 4137315840 bytes to spend on partitions.

But wait, didn't we create 4141875200 bytes in the first place? That's 4559360 bytes less than what we had originally. Well, this loss is due to the fact that for the 504th cylinder we'd need 8225280 bytes which we don't have, so this loss is inevitable. But the important consequence of this reduction of space is that we cannot create a filesystem on the whole bunch of data we supplied. At the moment the size of our filesystem is not determined at all. The next step is to create a new primary partition inside our big file using all the space we have:


After having written the partition table to the big file, have you checked for the new device file /dev/loop0p1? Don't worry, it does not exist! Adding p1 to the disk label is fdisk's way to denote partitions, that does not mean that you'll find such a thing in the /dev directory.

Poking Inside The Big File

From the partition table you can see that 4040316 blocks have been allocated for the new partition. With each block storing 1024 bytes we now know our first partition size, it's 4040316 x 1024 bytes = 4137283584 bytes. This is another number we never saw before! After having written off some 4.5 megabytes because we cannot use half a cylinder, we now face another loss of exactly 4137315840 - 4137283584 = 32256 bytes.

Of course these 32256 bytes at the beginning of the big file are there for a purpose, which is to store the partition table. Our first partition begins right after this amount of data, at an offset of 32256 inside the big file. The amount of 32256 bytes results from the fact that one track (63 sectors of 512 bytes for one head) are put away for the partition table. Now it's time to use a second loop device (/dev/loop1) to poke inside the big file at exactly the point where our first partition begins and create a new filesystem there:


It's essential that we supply the number of blocks as a parameter to the mkfs command to ensure, that our new filesystem on the first partition fits exactly in the space we have allocated. Without this parameter our filesystem would become too big, as the 4.5 megabytes after the first partition would be used for the filesystem too, and when the virtual machine is going to use the filesystem its actual size would conflict with the numbers in the partition table. Either the partition table or the filesystem's superblock is lying, which will cause distress for the virtual machine that expects a consistent filesystem to operate.

Writing The Master Boot Record

You can fill up the filesystem with whatever carefully selected quality open source software you can find on the planet, but in the end we need to write the new virtual disk's master boot record to boot the jewel. There is one step of preparation to be done before we can use the grub shell to write the MBR. We have to make a symbolic link named /dev/loop to the device that points to the master boot record, that is to the beginning of the big file, /dev/loop0 in the example above.


Now your spick-and-span virtual hard disk is ready to boot.