For secure remote access to servers, one of the most secure access control mechanisms is the usage of one time passwords (OTP). With the setup described in this blog post, you can securely log in to your server even from untrusted client machines, like an internet cafe. A potentially installed keylogger would only log a useless, old one-time-password.
To secure SSH remote access to your server with OTPs, you can use the software OPIE. More details on the inner workings of OPIE are described in RFC2289. There are several excellent tutorials available on the internet how to set up OPIE, e.g. one by Heise and one in at ubuntuusers.de (in German), so I'll just list the basic steps here:
When logging in to your server, it will now present you a challenge in the form
otp-md5 21 rs4415 ext
In order to login, you have to enter the correct response, which is a string of six english words, e.g.
MOVE EASY HOSE FEAT CUT NAVY
You could now either install an OPIE client on your notebook or PC, or print a pre-computed list of one-time-passwords and carry it with you. However, it is much more convenient to install a client application on your Android phone to compute the response based on your secret master password. At least in Germany, the app is not available in the Android marketplace, so you'll have to download the APK file manually. Make sure to enable "unknown sources" before trying to install it.