Recover ubuntu box from hack, part 6: How to log out of gnome without a mouse

So my ISP has an Ubuntu repository, so when I set that as my software source, it downloads super fast and doesn’t use quote. Even if we’re capped, its still super fast as the ISP doesn’t cap unmetered data.

So I decided I wanted to do this from the gnome GUI – since it already knows the source, and I don’t have to fight with vim to try find/replace (I use it, but just not that well yet, so GUI should be quicker)

So I plugged a keyboard into the server, and turned on the monitor. Its just running desktop version of Ubuntu so I can just log in like that.

Went into the update manager -> prefences -> server -> other -> find mine in the list. Save and exit. Mandatory reload of data shows that its working.

Now I’m done, so I try to log out.

How do you log out of gnome without a keyboard???

There’s no log out option in any of the main menus – accessible from Alt+F1

Check help for about 10 minutes, reading the keyboard shortcut pages – the help files aren’t written for keyboard only navigation – found nothing

Checked the System -> prefrences -> keyboard shortcut  menu and saw that ‘Log out’ was listed as Ctrl+Alt+Delete
So I try that, but I get a dialog that shows Shutdown, Suspend, Restart or Hibernate.


So I go back to my other computer and do some research. Turns out that the Fast User Switcher applet that is installed by default takes the standard logout and shutdown options from the system menu. This applet is the button in the very top right of the screen, and as far as I know is only accessible via mouse.

I do some more research and find out how to load the logout dialog that has been removed: gnome-session-save –logout-dialog

So I bring up System -> Preferences -> Main Menu and add a new item -> gnome-session-save –logout-dialog
Save that, go System -> Logout and then it shows Logout or Switch User


Problem solved, but 20 minutes to work out how to logout (nicely) without a mouse? I think that’s a bit of a user interface fail.


Recover ubuntu box from hack, part 5: setting up denyhosts

denyhosts is a python program that can be installed from the default ubuntu repositories. It seems to run on a very wide array of OSes, even mac.
What it does is scans the system authorisation log file for repeated failed attempts and other suspicious looking activity, and blocks the offending IP once certain thresholds are passed. This is the kind of thing to stop password guessing for ssh servers – including guessing rsa keys – though that is a bit extreme. In my case – with password access not even allowed – this program is probably only ever going to do anything when I legitimately am trying to access my server but I’ve lost my keys or something.

Anyway in this post I talk my way through installing and setting up this fairly nice piece of software.

  1. sudo aptitude install denyhosts
  2. tried to set it up to email myself – but it seems i need to have a working mail server. Looking into this a bit, it seems a bit more complicated than i want it to be for now, so giving it a miss. Anyway, going to test denyhosts now.
  3. luckily, i had a friend on hand. got them to try sshing a few times to non-existant user: now their IP is listed in /etc/hosts.deny file – excellence.
  4. now to remove the IP from my block list so that next time someone legit uses the IP they’re not blocked – leaving me wondering why:
  5. stop denyhosts – sudo /etc/init.d/denyhosts stop
  6. remove from blacklist – sudo vim /etc/hosts.deny
  7. now to remove the IP from all the files in the WORK_DIR – cd /var/lib/denyhosts
  8. open each file one at a time and remove the relevant entries
  9. hmm friend is gone – so can’t make sure they’re able to log in again
  10. Will try ssh back from uni once i forward the port again.
  11. I think my ssh is secure enough for now. So I will now actually forward the port to the great wide yonder internet so I can log in remotely…
  12. ssh’d into uni, trying to ssh back. doesnt work. check port is forwarded. check dynamic dns is correct with IP. hmmm. Oh that’s right. Uni blocks almost every port in existence. So i need to go back to standard port… hopefully password mode being off and having denyhosts installed will thwart any possible exploit…I guess security through obscurity does make me feel at least a little bit more comfortable after all. Oh well, here goes.
  13. ok so reset the port, restart sshd, modify port forward, try again from uni. and i’m rejected due to rsa key. excellent, as planned.
  14. Now to try to get the ip banned with denyhosts again: try ssh as root – get publickey error. but deny hosts does nothing
  15. try ssh as lol – get publickey error, but then deny hosts does something – It seems that since root login is disabled, sshd handles the deny without reporting anything to /var/log/auth.log
  16. now to remove uni IP from banned list once more. done and tested again. denying due to lack of publickey again – desired behaviour at this point.
  17. adding uni pub key to auth hosts file: winscp into uni, get pubkey, winscp into server, add to auth hosts file, done – just hope my uni account doesn’t get compromised.
  18. tested – successful – Excellence once more.

Now deny hosts is working on top of a pretty darned secure openssh-server – albeit running on default port.
next stop – setting up all the other services I had running – but that’s for another day.

I’m finding that I have a much better idea of what I’m doing this time around. Possibly because I’ve done it before this time, possibly because I have a better understanding of just how insecure I was before, and possibly because I’m actually researching things so I don’t look like an idiot when I put it on a blog. But I’m sure that just keeping track of what I’m doing is helping me remain conscious of what I’m doing.


Recover ubuntu box from hack, part 4: installing and setting up more secure ssh

so time to reboot into the live cd once more, but this time to install

  1. formatted /dev/sda1 to fresh ext4 partition and installed ubuntu 10.04 LTS AMD64 on it.
  2. installed openssh-server
  3. rebooted
  4. checked I could ssh in
  5. Now to configure to use keys instead of passwords…
  6. on my windows computer, using  puttygen – takes a while to generate enough entropy with the mouse. set up passphrase, save keys… done
  7. now winscp in and put the public key in ~/.ssh/authorized_keys file
  8. now try to connect:point putty to private key file and connect – blammo – in. hurrah
  9. now to disable password entry – sudo vim /etc/ssh/sshd_config
  10. find line “#PasswordAuthentication yes”
  11. uncomment (remove the #) and change yes to no
  12. save and exit vim
  13. restart ssh – sudo /etc/init.d/ssh restart
  14. try ssh localhost, failed due to no key set up for localhost (this is good)
  15. try ssh from my putty with the key set up – got in fine
  16. added my other computers public key – tested – works. excellent
  17. changing the port, tested, done

Next step to set up denyhosts – a program to block hosts that try to ssh in too much – and email me if it finds something!

Recover ubuntu box from hack, part 3: verifying I have everything important

So I’m checking that the RAID can be rebuilt from a fresh installation and that I have a copy of everything from the machine.

  1. booted from live cd
  2. using disk utility – comes with ubuntu 10.04 – I can see the RAID is still there, but I can’t mount it yet. need to install mdadm
  3. plugged in network, installed mdadm: does anyone know how to set up the auto-mail-when-drive-failing setting?
  4. can’t remember if I need to create a new array from the old partitions and if that destroys the raid. hmm. I know, copy the mdadm.conf file from old machine!
  5. sudo mkdir /media/oldmachine
  6. sudo mount /dev/sda1 /media/oldmachine
  7. copied /etc/mdadm/mdadm.conf from old partition and ran sudo mdadm –assemble /dev/md0
  8. sudo mkdir /media/raid
  9. sudo mount /dev/md0 /media/raid
  10. raid running from live cd, no worries!
  11. Now to make sure the backup I made is fine (i created a dd copy of /dev/sda1 before I started):
  12. sudo mkdir /media/oldmachineback
  13. sudo mount -o loop /media/raid/public/image /media/oldmachineback

Now to make doubly sure I have the home dirs backed up… whats this??? guest home is still there?

AND THE .bash_history FILE IS STILL THERE!!!

holy crap found the bash historydoesn’t actually look like the intruder managed to actually do anything but make some typos. seems a real live script kiddie at work here. dont think anything is going to be too bad, but i’d better reinstall to be sure. practice is good.

contact me if you want to check it out.

Recover ubuntu box from hack, part 2: the setup

Ok, so, I’m a little bit annoyed. The server was a bit of work to get going. Not least because I didn’t know what I was doing in the first place.

Anyway, lets try to work out what I had going on that server:


  • Ubuntu 10.04 AMD64 alternate
  • AMD Athlon II 250 or something, 4 GB ram
  • 3 x (identical) 2TB HDDs partitioned such that they have a large partition that leaves 50GB left.
  • First drives 50GB partitoned to Ext4 and ubuntu installed to that
  • Second drive has recommended size swap partition
  • using ubuntu and mdadm/some disk management tool, set up raid5 on the large partitions. This is where all the file storage is , and will handle one drive failing at a time. This was tested when I first set it up. Takes most of a day for the raid to be set up again if it needs to be rebuilt
  • remaining space on second and 3rd drive unused
  • I have a username for myself, with a decent strong password, a username for my girlfriend which was never used, and a guest username.
  • only I have sudo access, and root should have no password (as in cannot log in without sudo) as with default ubuntu setup.
  • guest password was guest – this account had no privileges beyond standard ubuntu user so I didn’t think this mattered. Just had that user so that my housemates could/would use the server at all
  • no real securing of anything except ubuntu’s default security
  • /home was just on the same partition as / and noone had anything moved from there.
  • on the raid drive (mounted as /media/share) was a public folder as well as a private folder for myself and another for my girlfriend. I used my private folder for personal stuff, not sure if my girlfriend used hers. Have to check. Public folder was full of all my valuable media.


  • ssh server – set up on default port 22 and port forwarded through router. standard password auth active and root access turned off (but log in + sudo works if you have sudo)
  • transmission – complete with web interface and set up  – for completely legal torrenting – from a central location to the house. Took some work setting up because there are about 3 different config files and I’m still not sure exactly which one it actually uses. ssh+tunneling allowed it to be done from anywhere
  • apache –  from repos. with cgi in some fashion, but not much of anything running on it. just for farting around trying to learn web stuff
  • rails with rvm. this was a bit of effort to set up but found out at the end that if I just read the readme I could have had it running in about 2 minutes. This was just to do some basic stuff to learn rails. I had 1 tutorial project done and 1 ongoing personal project. Used rails’ built in server as i was only doing dev. work anyway. similarly, only used sqlite.
  • squid – set up to allow tunnelling for about 10 minutes until I realised ssh server provided dynamic tunnelling which is all i needed anyway.
  • svn – set up to provide version tracking for my projects. all my repo’s were on the raid, but I’m not sure which projects were actively using svn.
  • trac – set up with its own built in web server. Had to set up some init.d scripts to get it going though. Had 2 projects on trac.
  • minecraft server – of course. this was on the raid anyway. still have port forwarded to play from Internet.
  • ddclient  – to update IP address with dyndns service
  • samba – to share the share directory of the raid setup

What to do now

  1. Well, I need to make sure I have all important data. I think what I’ll do is boot it up one more time, without Internet cable, and copy everything to a directory on the raid. I’ll just have a quick check to make sure the home directories are fine. Then I’ll copy the minecraft server world file, so that my house mate can get on with that.
  2. Once thats done, I’ll shutdown and hook up the dvd drive. Then I’ll boot Ubuntu from a live CD and hook up the Internet again. I’ll install mdadm to the live boot and make sure I can mount the raid.
  3. Then I’ll wipe the partition Ubuntu is installed to, and reinstall over it.
  4. Set up the raid again and make sure that works without the live CD.

I’ll report back here when that’s done; wish me luck!

Recover ubuntu box from hack, part 1: the story


As some of you may be aware, I’ve recently found out my ubuntu server box has been hacked.

This and the following posts in the series will follow my recovery from this – as a way of documenting and organising my thoughts and processes, as well as provide a medium for people to tell me where I’m being stupid.

The hack:

I have a publically accessible ubuntu box – ssh server on deafult port 22 which is forwarded through the router. I also have a dyndns account set up to track our IP.

I have a regular account on the server, guest, with an easily guessable password – guest.

The ssh server doesnt require keys, only password.

So I noticed the guest password had changed a couple of times over the last couple of weeks. Can’t be sure it didn’t happen longer ago than that, as I don’t use it often.

I checked it out the second time i noticed, and checked the .bash_history file for the guest account. Sure enough, someone has wgetted some image from russia somewhere, untared the image and executed the resulting root privilege escalation hack. Can’t be sure they were successful or not as I’m not experienced, and in my rush to sort it out, I rebooted the server and deleted the guest account. Haven’t noticed anything out of the ordinary since, using netstat or top, everything seems fine, but I can never be sure if its going to have access to the internet at all.

Tried for some time to try to undelete the files but I can’t really work out what im doing and the program I’ve used before (testdisk – its really good!)  is segfaulting (yeah, heaps good).


Anyway the authentication failure log has alot of attempts – about every 10 seconds – to log in as root, from various remote IPs. Seems that they were still trying to get in up till I rebooted, but again, can’t really be sure they weren’t successful. They even left the .bash_history, which is fairly nooby, but again can’t tell.


So that’s that. Next post is working out what I had on the server so I know what to set up again.

New blog

Hello all,

Thought I might try to document setting my server up this time. I don’t know what I’m doing really, so I’m looking for feedback, and I’ve been looking for an excuse to make a blog for a while I suppose – just never had anything worth writing a sizeable amount about – that anyone might be interested in – until now (I hope someone’s interested anyway).

Updates will come before anyone reads this anyway – so no point saying:

Watch this space


%d bloggers like this: