Recently in Virtual Private Servers Category

Certainly not, if you store credit card information or passwords in clear text on the servers. Recent data theft disasters have shown, that it is not enough to operate a "secure server" and leave all customer's information unencrypted on this server. Because if you think your secure server is invincible, all your customer's data is at risk, the moment it turns out that the secure server is not as secure as you thought.

What's even worse, your customers have entrusted you with their data believing that operating a secure data center will be sufficient to protect their personal data from falling into the wrong hands. It's time to destroy this false belief.

Almost everything you'll learn to know about why you can trust an online shop or an online service provider boils down to the fact, that they make every possible attempt to secure their servers in the data center with all available bells and whistles of modern technology. But there is very little information - if any - about how they treat your data when it is stored on their secure servers online.

Of course I honestly value every effort to make online servers as secure as possible, but on the other hand I am convinced, that securing your online servers alone is not good enough. Let me explain.

A secure server is not invincible

Today most online services use a scripting language and a database server (think PHP plus MySQL for instance) and sensible information goes into a database. Usually the access to this database is given to every program that knows about the database password and normally runs on the same computer.

   It's a complete miracle to me, why after a default installation of some online shop applications the database password is stored in a file that could be read by any user on the system. It seems to be a commonplace belief that the server running the online shop is invincible, and therefore it poses no problem to store a database password in clear text without any additional protection. As we should know by now, secure servers are not always invincible and storing customer data unencrypted is a very bad idea. Not every online service is that careless about database passwords and it's easy to restrict access further, but today it seems to be the norm to dump the responsibility for the protection of customer data onto the administration personell in the online data center.

I wish to make the point, that we have to create online services in a way that if something goes wrong - despite the care to prevent this - customer data inside the online service is still protected against exploitation by intruders. And I'm convinced that such a protection is not only possible but absolutely essential if customers want to use online services securely.

Security and Convenience - Choose one.

As a consequence I'm sure that we have to correct another common misconception. You cannot have both, security and convenience in the online world. Forget it, you have to dump your notion of an easy, hassle-free, automatic and secure online service. Choose one or the other, you cannot have both.

That does not mean that secure online services have to be a pain in the neck but it certainly means that some procedures that have been introduced to make a customer's online experience "smooth", "seamless" and "easy" have to go if an online service that stores credit card information will ever be secure. Without knowing why, online customers will always think that some (unknown) data center professional is the right one to fight security problems, because after all they are in charge to make sure everything runs well. So it's vital that the ordinary online customer knows why some features of online services cannot remain the way they are today.

In their efforts towards making the online customer's life as uncomplicated as possible some vendors have started to encrypt their customer's sensible information. That's fine. But if you store the encryption key in a file next to the encrypted database, it's a little like locking your door and hiding the key under the door mat. This clever idea was born out of necessity, of course. As online customers demanded automatic payment from their online services there had to be a way that a program on a secure server could access the encrypted data (CC info for instance) without the intervention of a human being. The idea of storing the encryption key in some way on the server had been invented originally to ease the pain of the online customer.

If a customer wants to sign up to a service in the middle of the night (very convenient) the customer's data can either be stored in clear text (a bad idea) or, if it has to be encrypted, the key must be available to the web server program on the secure server (another bad idea). Surely, if you go fighting a bad idea with another bad idea, you'll never end up with a secure online solution.

Before you come to the conclusion that there cannot be a secure online service, let me tell you what I've learned during the last couple of months, while I tried to learn something from the data theft disasters and created a secure online service for small businesses, the secure online bills.

Secure Online Bills

An online service has to protect a customer's data even if there is an intrusion into the secure server, this is the fundamental principle that determined the design of my online service. It's really not easy to follow this principle at all times, because while I coded the system, I tried not to burden the user with avoidable inconveniences. But writing every single line of code myself made it clear to me that there have to be some decisions that will make the user's life a little bit more complicated. And it is inevitable, if the system should be secure.

I know, the last thing a customers wants is complications, "easy" is the marketing word not "complicated". But it became clear to me that there had to be more human intervention in the process if it should ever become more secure. So it is not complication that is inevitable. It's the introduction of the human factor into the process that makes it more reliable and secure.

What we need is more human intervention and less automation. I'm sure that if you begin to see, why it is necessary to rely on the informed decision of a human being instead of an automated web server process, the loss of convenience will become totally irrelevant. Making online services more human is the way to go, that's what I've learned from coding a secure online solution.

For instance, if you sign up for an online service and expect your login details to arrive via email within the next few minutes, your data cannot be secure, because the encryption key must be somewhere under the door mat. This is fine for demonstration purposes, and I use it myself for the secure booking service demo, but it's not good enough for the real thing, where customers rightly expect their information to be protected.

An online service cannot be secure when signing up is a matter of seconds. In fact, the setup of a secure online service requires manual work (of a human) on a server and of course some communication between the interested customer and the responsible person at the online service provider. The reason is simple, if the encryption key cannot be hidden under the door mat, it must be entered into the system from the outside in the moment when the encrypted information is needed. Using the system in a secure way requires preparation that cannot entirely be automated or, let me say, should not be automated at all, if you don't wish to encounter unpleasant surprises along the way.

What about lost passwords ?

Storing passwords in a database with the key under the door mat, no! Once the system is secure the database can only be re-encrypted with a key coming from outside the system. If you think the secure server should be able to reset your account easily, you're wrong. This would only be possible with a second emergency key stored on the server under a second door mat. No, the solution is to contact the service provider and make him use his emergency key which is safely stored outside the system to re-encrypt the database. That means, you'll have to wait until a human being does something useful for you. Is that too much of an inconvenience? I don't think so. Don't expect a server to do it instead. Get rid of this belief.

And if that takes time and costs money, be happy that your problem is being taken care of securely. And stop searching for a dirt-cheap, automatic way to have your negligence corrected without having to talk to a real human being.

Normally, a user is someone listed in the system's database file /etc/passwd. There is no need for a flesh-and-bone user equipped with a password and ready to log in. Many users listed in the system database are simply immaterial daemon users, like mysql, listed for the operating system to be able to assign a name (especially a UID in the form of a number) to a running process.

But at least these daemon users own processes, i.e. the mysql database server, and of course, real objects like files and directories are owned by those users. In a well run environment these users do not have a password, so their account cannot be abused by somebody else, and almost always the shell that would be started if someone could log into this account is the binary /sbin/nologin, which does exactly what's on the tin, denying login.

And then there is another breed of users that seem to be ordinary fellows in the system and still do not exist as a user in the normal sense. These virtual users do have the ability to run certain programs, they all have a mailbox in which their private messages arrive and they all need a password to access their home directory, in which all their assets are being stored, but strangely enough there is no trace of them in the system's user base and password file. One can argue that because the virtual users don't have a valid number (UID) they are no users, and in a strict sense this is certainly true. But how can they own files and use a mailbox, just like other users do?

Hunting down the virtual users

We all know that in order to have a mailbox, there has to be a mailbox file (usually in /var/spool/mail) that is owned by the user respectively. And if someone is attempting to read this mailbox, there has to be some kind of authentication, so the password must be stored somewhere. At this point a real (daemon) user comes into play, the postman Tom, who of course has a valid UID listed in the file /etc/passwd, which is in fact 555 on my system.

Tom works at the heart of the mail delivery process, managing the virtual user's home directories and their mailboxes, and he is the only real user necessary to serve hundreds of virtual users on the system.

Virtual home directories and mailboxes

For every virtual user Tom creates a home directory that is used by IMAP to store the virtual user's inbox and various index and logfiles individually.

All these files have been created by Tom for each of the virtual users. In order to provide this infrastructure, we have to make sure that Tom is able to use the virtual user's password when needed, and most importantly to handle the authentication process for them as well. The following lines of code show how the configuration of DOVECOT has to be changed for Virtual Domains.

All passwords are stored in the file /kx/horde/htpasswd in the usual way required by the apache web server. This is important for two reasons, because the dovecot process can use this file to authenticate virtual users by changing permissions to apache for authentication, and simultaneously this file allows other web-based software to access the virtual user's homes with the same password. We will have a look at this later.

Getting all email in the right direction

By now we have managed to install mailbox access for virtual users via IMAP without the need of registration of all these users in the system's database. What we do not have in place is a mechanism to fill up the virtual user's mailboxes with incoming mail.

As you may suspect another important part of the mail delivery process has to be adjusted to ensure that the virtual users who can have totally different email-addresses will receive their mail without hassle. This time we need to add a few lines to the POSTFIX configuration file and we have to create a mapping between the email addresses and the (real) virtual users, who are supposed to read the mail.

The pivotal point here is the text file "kx/dovecot/virtual_mailbox_map" in which each email address is followed by the virtual user's name. But before postfix can use this database to deliver mail it has to be hashed, i.e converted into a database file "kx/dovecot/virtual_mailbox_map.db" with the postmap command below.

Extending our infrastructure

The most important benefit of this system is not to have virtual users in the system's user database and being able to treat them as ordinary users at the same time. As some users tend to choose a weak password this will help to limit the damage that can be done to the system considerably. But once virtual users are established with an email address and password in the manner I have described, it seems to be an obvious idea to use these credentials for another purpose, to organise a file system-like structure for virtual users, to store their data as well. Fortunately such a file storage can be easily set up using the HORDE framework, and in particular the gollem module, that can be configured to use the file system or a MySQL database to store the virtual user's files while making them available for upload and download via a browser-enabled interface.

Have you ever listened to someone who had been fallen victim to a burglary, someone whose home had been broken into by a criminal, entirely ignoring his privacy, snooping around in his personal belongings and making a total mess of everything coming across his way? It is nearly impossible not to feel the pain yourself this person has been through.

And in the online world things are not that different. After all the effort you've put in to getting a Virtual Private (!) Server going, and after furnishing it with something you think is valuable and worth protecting, you can easily get into a similar rage about people starting to break into your VPS the minute you have finished to make it a venerable target for those careless and often thoughtless digital scumbags on the Internet.

A brief look at the file "/var/log/security" shows how your virtual home is under attack right at the moment.



I sincerely welcome any and all attempts possible to counter the disgusting actions of those, who in most cases are not even remotely aware of or capable to understand, what the software does they are (ab)using.

Statistics of a Break-In Attack

Let's have a look at such an attempt to abuse the ssh service at my VPS. It is recorded in the very first hours this machine became visible on the Internet.

During the best part of an hour the attacker had tried 583 passwords to break into my digital home, nearly every 4 seconds a failed ssh connection was recorded in my logfiles, not unsurprisingly targeting the root account. As it turned out, the IP address was not bound to a registered domain name, so it could even be a innocent user's PC taken-over by the attacker.

Counter-Measures

Some people have suggested trying to play "hide and seek" with the culprits by switching the native port 22 which is used by SSH to something like 51679. Even though that may reduce the rate of attacks, it surely won't deter those who use port scanners to find the port used for SSH.

Others rely on periodically updating their "/etc/hosts.deny" file blocking access from IP-addresses that have proven to be suspicious. This may solve the problem for those attacks that always start at the same hosts, but attackers changing their base frequently will still enjoy an open door, once they are using a freshly taken-over host machine.

I have been taking a different approach, namely to change the response of my firewall to incoming ssh requests. I changed the rules so that only 3 attempts per 2 minute period are being served. This reduced my server's susceptibility greatly, as the automated attacks tend to break off after very few tries and don't return.

The following lines added to my firewall script did the job: