To be or not to be - Hacked

After christmas I learned that we had a suspected break-in in one of our production sites. The incident occured just before christmas and I was dumb found to find out about it after the holidays and even more so when the incident had been closed with a non conclusive result. I don't know if it just me; because in my book the suspicion alone is a "stop the world operations event". You are not content with and you just don't leave it in an inconclusive state...

The reason we suspected it was that /var and /root and /sbin was gone from one of the machines. Yes - there are more non obvious ways to hide you're doings but this did remove any potential traces and did cause some havoc. Althoug I must confess that I would expect more from any one capable of penetrating us; either in the capacity of destruction or the sutelty of their presence.
After Christmas I learned that we had a suspected break-in in one of our production sites. The incident occurred just before Christmas and I was dumb found to find out about it after the holidays and even more so when the incident had been closed with a non-conclusive result. I don't know if it just me; because in my book the suspicion alone is a "stop the world operations event". You are not content with and you just don't leave it in an inconclusive state...

The reason we suspected it was that /var and /root and /sbin was gone from one of the machines. Yes - there are more non-obvious ways to hide you're doings but this did remove any potential traces and did cause some havoc. Although I must confess that I would expect more from any one capable of penetrating us, either in the capacity of destruction or the subtlety of their presence. 

Fortunately we do have all kinds of fancy stuff to pull the server back in a state it had just moments before the time we identified. Well we would have had that if there hadn't been a Christmas in between the incident and some one thought it worthwhile to mention (yes I was fuming with anger). Regardless after some digging we did find a prime suspect cause. Since it had been such a long time since the incident and that the logs where gone we couldn't prove it to a 100% without actually testing it. Revert the server to an even earlier state and testing the suspected cause did result in exactly the same result.

The lesson for me was that - I do not know what is usually done when a server is owned and I have no experience in back-tracking the bad guy’s steps.

So hence - I will setup my own honeypot and explore what is done to it when someone owns it. I’ll also to try to learn how to detect and how to perform some kind of basic forensics on the machine. Sounds like fun... and what can possibly go wrong :-)

Honeypot Controller and Honeypot
The setup will consist of a firewall, a honeypot and a controller. The machines will run Ubuntu 13.04.


First splash

I'm still working on the setup and how to allow someone to break in without putting the rest of the devices at risk. There are also some work to be left in terms of monitoring the honeypot and gathering of data before the break-in. However with a fairly modern ssh-server and a complex password I felt it safe to open up for ssh access. I wanted to see how long it took before I got visitors.

19:54 This is the time when I closed the connection and open up for ssh and not more than a few hours later I have someone knocking at the door for the first time. After 24 hours two different attack methods had been deployed. One was brute forcing root and the second was trying some standard users/passwords.

Brute forcing root.

1st attack (Turkey)
4th attack (Turkey)
Jan 11 22:45:20 192.168.0.101 sshd[7410]: Did not receive identification string from X.Y.Z.W
Jan 11 22:45:21 192.168.0.101 sshd[7411]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=server-176.53.62.227.as42926.ne
t  user=root
Jan 11 22:45:23 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:24 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:27 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:29 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:31 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:33 192.168.0.101 sshd[7411]: Failed password for root from X.Y.Z.W port 28128 ssh2
Jan 11 22:45:33 192.168.0.101 sshd[7411]: Disconnecting: Too many authentication failures for root [preauth]
Jan 11 22:45:33 192.168.0.101 sshd[7411]: PAM 5 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=server-X.Y.Z.W.as42926.net  user=root
Jan 11 22:45:33 192.168.0.101 sshd[7411]: PAM service(sshd) ignoring max retries; 6 > 3

User/Password

2nd attack (Netherlands) 
3rd attack (Spain)

 Jan 12 02:39:16 192.168.0.101 sshd[9535]: Address X.Y.Z.W maps to server.anonymous-hosting-service.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Jan 12 02:39:16 192.168.0.101 sshd[9535]: Invalid user auto from X.Y.Z.W
Jan 12 02:39:16 192.168.0.101 sshd[9535]: input_userauth_request: invalid user auto [preauth]
Jan 12 02:39:16 192.168.0.101 sshd[9535]: pam_unix(sshd:auth): check pass; user unknown
Jan 12 02:39:16 192.168.0.101 sshd[9535]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=X.Y.Z.W
Jan 12 02:39:18 192.168.0.101 sshd[9535]: Failed password for invalid user auto from X.Y.Z.W port 43813 ssh2
Jan 12 02:39:18 192.168.0.101 sshd[9535]: Received disconnect from X.Y.Z.W: 11: Bye Bye [preauth]
Jan 12 02:39:19 192.168.0.101 sshd[9537]: Address X.Y.Z.W maps to server.anonymous-hosting-service.com, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!

 Users tried
Must say that I was surprised that it was so few and maybe not the first ones I would try. Get a feeling that there is more behind this than I realize at this point.

 adm
akai
alfonso
apache
araceli
archivo
aron
arun
aseanmt-khm
auto
auto
beaulaptic
boot
card
chanon
cliente
compras
copie
copy
danger
dangerous
daniel
david
ddo
desarrollo
diego
direccion
edilson
email
eric
eugene
export
fabio
financiero
free
ftp
ftpuser
gast
gast
gerencia
gosc
gosc
guest
hacker
henry
home3
host
hosting
hosts
html
http
ident
info
joe
john
johny
kevin
klog
kubota
kubota
laboratorio
landscape
langson24h
libb
linux
liudongfeng
liyiduo
lumi
maggie
mailnull
manager
master
mensajes
mihai
monitor
mp3
mtapia
mysql
namep
nathan
netdump
network
newstaller
nologin
ntp
office
operations
operator
oprah
oracle
oracle1
orange
paradise
pgsql
plesk
postgres
primaveras
prueba
puxiaolong
quark
reservas
saito
saito
shit
shit
style
test
usuario1234
vyatta

Will be interesting to see how this develops with a few more days under the belt.

Day Country Kind Attempts
1 Turkey Brute root X
1 Netherlands User/Pwd 2
1 Spain User/Pwd 46
1 Turkey Brute root X
2 Brasil Brute root X
2 Bulgaria User/Pwd 24
2 India User/Pwd 1
2 USA User/Pwd 160
2 USA BruteRoot+User/Pwd 71+6

Comments

Post a Comment

Popular posts from this blog

Possible SYN flooding on port 3306 (MySQL)

The system setup is such that the MySQL servers that put the "Possible SYN flooding on port 3306" in the log files only are exposed to system internal backend services. These in turn aren't exposed to the wild wild web. Fronting the system we have the servers publishing services to internet. Thus I was kind of stunned when the log messages started to appear and even though we had done a release of the system I found it far fetched that we should start to DOS our self. So why did the messages appear? Two different error messages could be identified in the log files and they seem to be related, especially since the Java servers with link failure do communicate with the MySQL servers. Possible SYN flooding on port 3306 @ MySQL server Communication link failure @ Java backend servers After some tcpdumping, head scratching and googling I think I have it down to the root cause, hidden in how TCP works in general, and the OS config in combi...
Read more

Part 1 - Disaster Recovery with SRM and vSphere Replication

Image
Abstract  The company I work for has needs for disaster recovery but there is also external demands for it. The company not being very big has so far not been able to implement any automated disaster recovery solution although we do have a documented disaster recovery plan. The plan regardless of how good or bad it is isn't tested in a long time and the RTO (Recovery Time Objective) is wished by management to be days but I can't see that it is anything else than weeks. Since we virtualized our production system I've been looking at VMware vCenter Site Recovery Manager as a driver but the cost array based replication has stopped any attempts dead in their tracks. VMware vSphere® Replication has been around for a while now and we hope that it's hardened enough for a slightly bigger production critical implementation. In this article I will try to document and explain how we experiment, design and implement disaster recovery and even more important, in my opinion...
Read more

Part 2 - Disaster Recovery with SRM and vSphere Replication

Image
In the previous article we went through the installation and configuration of the SRM and vSphere infrastructure. The time has now come to actually doing some tests and failover some VM's.  In the simple scenario I expect everything to go smoothly but there are a few things that I'm concerned about at this point since the protected environment I'm ultimately is failing over isn't so simple. It has multiple dvSwitches, vLans, load balancer, firewall, ldap, a set of test drivers to verify the integrity of the system and access to the system in a sensible way for administrators so there are a few things that needs to be resolved. Protected Setup Test Failover The small test In this scenario I do a test failover of a single machine and verifies that it starts and that I can log into it. The steps are: Setup the machine running RHEL 5.5.  Install VMware tools  Kick of the vSphere Replication Create a Protection Group and ... no that didn...
Read more