Hacked
 

I have a 10.3.4 OS X server which was hacked in Sydney Australia;


systemlog:

Sep 28 20:23:41 macintoshhd sshd[5560]: reverse mapping checking getaddrinfo for s14.newhope.galati.rdsnet.ro failed - POSSIBLE BREAKIN ATTEMPT!

Sep 28 20:22:48 macintoshhd xinetd[390]: START: ssh pid=5560 from=82.79.116.77
Sep 28 20:23:57 macintoshhd sshd[5560]: Accepted password for root from 82.79.116.77 port 1631 ssh2

systemlog
Sep 28 20:27:42 macintoshhd mach_init[2]: Server 0 in bootstrap d03 uid 0: "/System/Library/CoreServices/UserNotificationCenter.app/Contents/MacOS/UserNotificationCenter": exited with non-zero status 1
[pid 5967]
Sep 28 20:27:43 macintoshhd sshd[4137]: error: BSM audit: solaris_audit_record failed to write "sshd logout " record: Operation not supported
Sep 28 20:28:47 macintoshhd kernel: hfs: WARNING - blocks on volume DailyBak not allocated!
Sep 28 20:28:47 macintoshhd kernel: hfs: WARNING - blocks on volume DailyBak not allocated!


.bash_history
Image7:/Volumes/Data//var/root hma$ tail .bash_history
exit
w
cd /tmp
ls -all
cd /
ls
rm -rf *
ls -all
rm -rf *
kill -9 -1


After this command all my data was gone apart from:
Desktop DB Network automount cores etc mach.sym opt sbin usr
Desktop DF Volumes bin dev mach mach_kernel private tmp var

I did have SSH port 22 open and my root password was alphanumeric like "spoon78". I tracked down the culprit from Romania:


How would this person got the password - I have checked previous sys logs which have no log of the IP/domain address - what other logs can I check to see what else is recorded.?

Is there a way to see what time the commands where done in .bash_history?

I have sent am email to the admin of the domain.

Sorry, but that is a MAD easy password to crack.

Here is a Public Service Announcement on how to craft a good password. Please take it's advice to heart.


It's unlikely that the IP address this came from is the actual computer this person is sitting at. That would just be plain dumb. More likely it's another hacked box that was used a conduit to get to your box. How did he get your password? Could be a variety of ways. Maybe sniffed from another compromised box on your network? Maybe an unhappy employee? Maybe he got the nidump and cracked it?

Suggestions so this doesn't happen again?
USE BETTER/STRONGER PASSWORDS! This is your last line of defense. And for god's sake, change all your current passwords.
Disable root. You can use sudo, so there's no need for root.
Learn to use ipfw from the command line. If the firewall control in Server is anything like OS X client, it's COMPLETELY USELESS. Learn to use ipfw so you can only allow ranges of IPs to connect via ssh. Not running a firewall? Run a firewall! At least then you can log all traffic to ssh.
Use "AllowUsers" in /etc/sshd_config to only allow certain user into ssh. root, if you have to have it enabled, should not be one of them. Read the man page for more details.

If your Mac was "hacked" and the root password known, then you can't trust anything on that machine. The OS might have been modified to do nefarious things and to hide further traces.
You should grab what data you need (but note that the data may also have been modified so don't trust it either) and then erase the disk and reinstall from a CD.

Ways a cracker can get your root password--don't do any of these things:

1. If you've used the root password with insecure services like ftp, telnet, e-mail (any type unless SSL encrypted) it can be easily sniffed.
2. If you've used the root password for another of your accounts, especially when said account is on a remote computer. For example, if your e-mail password is the same as your root password, you may as well just post the root password on alt.crackerz
3. If you've given the root password to someone else, say the repair dude working on your hardware at Joe's Computer Repair and Bait Shop.
4. Social Engineering--calling you up and pretending to be someone legitimate who needs the root password.
5. Brute force--a password cracking utility.
6. Shoulder surfing--difficult from Romania, but you have no actual evidence that the cracker is from there-just that the IP directly before accessing your computer is in Romania.
7. If you sent the root password in an non-encrypted e-mail, it could have been intercepted.

Trevor

It's appalling how often this works.

Hoops,

Thank you for posting this. In addition to the fine suggestions of Yellow, I would also point out that you can disable root login through SSH by adding this line to your /etc/sshd_config file:

PermitRootLogin no

This is useful if you ever have to enable root for whatever reason and forget to set it back.

I also use the AllowUsers directive in my config, but cannot set up a good IP range since I am on the move too much. If Yellow would like to explain more about his feelings on the uselessness of the built-in firewall, I am all ears.

I also would like to ask if SSH protocol 1 is still used on the default Mac setup? Would folks suggest modifying this? And if so, how?

Thanks!

P.S.Here is a link Yellow started to get some good password tools/discussions.

Sorry, I might not have been clear. I have no problems with ipfw (the built-in Mac OS X firewall). I like it quite a bit, for a firewall vulnerable to DNS spoofing. What I have problems with is how Apple offered to control it (in the client, I've never used OS X Server) with their pathetic GUI from the Sharing Prefpane. The CLI can be really scary to people, and BrickHouse/SunShield can be really confusing, to a firewall-n00b.

My problems with Apple's GUI control for ipfw:

1) Not enough fine grained control.

2) No logging. Without logging, what good is a firewall? I'm just supposed to assume that it's working and assume that no one is touching the "special areas" of my box? (no jokes please :rolleyes: )

3) The worst offender of them all: ports are fully open or fully closed, you cannot specify IP address or ranges to allow or disallow. What is the difference between a Mac with ssh turned on, and no firewall, and a Mac with ssh turned on, and the Apple's GUI-controlled-version of the firewall? Nothing. They look the exact same to anyone in the world who wants to scan my box. So what is the point?

I hope this clears up my vagueness from above.

Back to the original topic... the cracker seems to not be very um, well, elite. He uses ls -all, a syntax indicating that he has only a trivial understanding of the ls command. He doesn't cover his tracks. And most important, he jumps immediately to erasing the hard drive rather than using the computer as a mule.

Yet there is no record of failed password attempts (at least in what you've shown us). This indicates that the cracker already knew the root password before attempting to break in.

So who would fit these two attributes? A disgruntled former employee? Or a shoulder-surfing disgruntled current employee?

Trevor

And what does the outgoing port being 1631 mean? When I ssh into other computers, my outgoing port is in the 50,000 range. Does anyone know the significance of port 1631?

Trevor

.

For anybody wanting a GUI, Little Snitch:
http://www.obdev.at/products/littlesnitch/index.html

Good thing is that a script with 'killall -m LittleSnitch' will not be enough, it will still relaunch.

And about OSX build-in firewall, well...
'defaults write /Library/Preferences/com.apple.sharing.firewall state no &'


Trevor is right, childish and unsophisticated break-in...

.

It would be useful to look back in the logs (just how did these not get wiped anyway?) and see if you could find any trace of previous doorknob-rattling. This is a good example of the importance of keeping your logs, and having a system whereby your logs are transfered to another host/medium for archival purposes would be good.

I'm surprised that "rm -rf *" left what it did. Did someone catch it before it could remove everything? Will it not remove potential system files automagically (seems unlikely)?

The rm command hit /bin/rm and subsequently stopped, I believe.

Could be wrong.

visitview 1631/tcp Visit view
visitview 1631/udp Visit view

Network address translation (NAT) - RFC 1631?


.

Yeah, that's what I have in /etc/services too. But what does it mean? Why would that be an outgoing port used by ssh?

For example, does it mean the cracker was not using a *nix box, but was using Putty or something similar on Windows?

Trevor

The 'origin' of a lot of SSH scanning seems to be in Romania and China. It seems like an automated malware circulating the internet trying sshd logins using very 'simple' username-password combinations.

I can agree with you Sao, for my own experiences recently. 90% of the IPs I've blocked because of attempted connections via ssh have been from China.

Have you had any luck narrowing down the culprit? As others have said, which i completely agree with, it looks like he knew the password, which would rule out it being just a random act (maybe he spent that minute and nine seconds making sure he was typing it exactly correct? :p)

It just seems like a malicious, possibly personally motivated, attack. Once he got in, and after some sloppy Unix, he went straight for deleting your files, he wasn't looking for anything. Anyone come to mind?

I hope you find out who it was and take appropriate action.

OK first up many thanks for the replies.

The box is behind a Cisco PIX managed firewall with NAT and SSH, SMTP and Timbuktu ports open.

FYI I have reinstalled OS, changed all passwords, restored data from tape and currently disabled SSH. Also I have enabled password policies thus disabling accounts on 3 failed attempts.

Yes it was an easy password, but I don't think it was cracked as the log states one connection from the intruder and a successful password straightaway.

Sep 28 20:22:48 macintoshhd xinetd[390]: START: ssh pid=5560 from=82.79.116.77
Sep 28 20:23:57 macintoshhd sshd[5560]: Accepted password for root from 82.79.116.77 port 1631 ssh2

That evening/ night I had a load of unsuccessful attempts from a Korean IP address approximately 12 a minute from 6pm until the system was wiped at 8.30pm (hard to believe the OS allows this many attempts - this is why I have enabled password policies) - these also continued after the intruder got in.
Below is an example of the unsuccessful attempts:


I am glad the rm -rf* didn't wipe the base bsd files as I have the logs for investigation.

The most alarming thing for me is the fact that it could be a person in the company, in my area no one here has any UNIX knowledge , only graphics people - however there is an IT dept in another floor who have a bunch of people capable, I have had a couple of minor tussles with them - but who do I trust? (None are Romanian :).

I have sent emails to both the Romanian and Korean admins of the IP addresses but have heard nothing.

Unfortunately, you probably won't. :(

I would suspect the in-house angle as your first line of suspicion.
Hope everything works out for you.

how easy is it to disguise your IP address so to say that the intruder wasn't really at 82.79.116.77 or would it be more a case of 82.79.116.77 already being hacked and using that box as a mask to attack mine?

Yeah, don't be hung up on the culprit being physically in Romania.

For future security, you should examine your true business needs and come up with a comprehensive security policy. This is a big part of my job--preventing just this sort of thing.

One of the basic suggestions I can offer is to consider using a VPN and disallowing all remote access that is not authenticated/tunneled by the VPN. You can have greater control with shared keys, or at least extra-tough passwords for the VPN users (different from their system passwords). It can be done pretty economically.

IMO, it's more likely the latter then the former. I also wouldn't expect that box to be the first link back to the original machine either.

Off the top of my head, I cannot thing of a way to spoof an IP when connecting directly to a service like ssh. However, I'm neither a hacker/cracker, nor a trained computer forensic tech.

Here are some of the ways...

1. Buy an account on a server in that country. Server space is cheap in most countries. Security is lax.
2. Use an anonymizing or tunneling service. Hacker groups have them for use by their members.
3. Gain access to a zombie or network of zombies. It's not that hard.
4. Crack another account, and use that to bounce your attacks from.

I was ops manager for an ISP in the past, and now do a lot of security work. You might be shocked at how easy it is for someone with ill intent to launch a semi-anonymous attack. The new legions of zombied Windows XP workstations are going to be a huge problem for everyone soon. Right now they're mostly used to send phishing spam, but I expect they will be used for much more soon.

I see DDoSes in Microsoft's future. :(

On X.3.5, setting these two config options causes ssh to fail. What version of ssh and/or sshd are you using?

Trevor eventhough the command execution is blunt and crude which could point to a person in a hurry or suffers from poor unix commands the fact that they contolled it via a potential hacked box in Romania (I am not hung up on the Romanian thing - but it did come from there) shows that they have some nouse on cracking.

Well, you would think that would be true, and I really wish it were. Unfortunately, due to Microsoft's incompetent "secure" programming, Windows boxes are easily susceptible to viruses and worms that open large gaping back doors that any script kiddie who can download a couple of programs can exploit. Basically, there are literally millions of pre-compromised boxes sitting on the Internet available for just anyone to take over and do anything they want with.

So, if a disgruntled employee at your workplace shoulder surfs you to get the root password, and wants to get revenge for some past tussle with you, he could easily just find some compromised Windows box to bounce off of before hitting your XServe. This would make him think nobody can trace him.

Trevor

aubreyapple, I have PermitRootLogin no set in my /etc/sshd_config file in OS X 10.3.5 without any failure of ssh or sshd. I am using the versions of ssh and sshd that ship with OS X. I don't use the AllowUsers directive, though, perhaps that is the problem in 10.3.5?

Trevor

OpenSSH_3.6.1p1+CAN-2004-0175, SSH protocols 1.5/2.0, OpenSSL 0x0090704f

OpenSSH_3.6.1p1+CAN-2004-0175, SSH protocols 1.5/2.0, OpenSSL 0x0090704f is my installation. Wierd.

I am two security updates behind, so maybe it is in there. I do not have 2004-09-07 or the next one. Too many people reporting the latest ones
break things, I guess.

You might find out why if you put sshd into debug mode via the -d option.
Note that you will have to restart sshd after each connection when it is in this mode.

Ok, first of all, the version of ssh I had was:
OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, OpenSSL 0x0090702f

Updated the security updates and now my version agrees.

However, ssh out still fails with PermitRootLogin no. I do not see that
sshd is involved with this failure.

Oddly enough, ssh inbound is ok with this line in the file. It is ssh outboud which fails and sshd does not run when ssh is working outbound.

And is the PermitRootLogin no set in your ssh_config or your sshd_config?

Actually, perhaps this needs to be in a new thread...

Trevor

I am blind... aha sshd_config not ssh_config. That works and has the desired effect.

Nah, it doesn't sound like your IT guys to me. I'd go over there with a box of donuts sometime and see what they think of your situation. IT guys like this kind of thing. And they like donuts.

;)

Anyway, I am concerned about your new security policy.
So what this means is if they stumble across one of your account names, you have allowed them to create a DoS attack on that account. You may end up with more phone calls than you want from any user with a guess-able name. You can easily tell what usernames they are using by grepping them out of the logs. If you think your account names are safe, fine. Just be aware of this type of risk.

And as MervTormel once said... :)

.

God I miss Merv!

Anyway, I still don't think it's an inside job. Are the IT guys good enough to fake their IP? Granted I have only been on Unix since OS X came out, but I am far from that good. I doubt someone that competent would wipe your hard drive with such poor grammar. They'd screw with you a bit first. If I this was personal and I knew a whole lot more than someone I didn't like, I'd let 'em know. :cool:

Unless you're busy pissing off more than just the IT people at your office. :rolleyes:

I like to think I'm pissing no one off - but I have had a couple of bad things happen recently;

got the svchost.exe worm on a Win2000 box Domino web enabled,
then a root kit installed on a RHEL WS 3 which aslo had SSH open, couldn'tget the logs on that one,

the 3 incidents happnened on separate LANS but connected by a national WAN.

all this in one month and nothing prior in 5 years.

Yeah, CAlverez made note of larger scale incidents. What's a RHEL WS 3?

RHEL WS 3

Red Hat Enterprise Linux Workstation

(used as a accucobol based financial transaction server)

No offense, but I hope your passwords on the Red Hat weren't as easy as the one on the Mac. Personally, I am seriously thinking of retiring my current ones. I have used them far too long.

re: changing passwords
I agree. I recently moved my main user files (which was an admin) to a new user account without admin privelages and put a massive password (well, 11 characters) on my admin account. The admin password is never used to log in remotely (I have my router's firewall blocking ssh from outside anyway) so anyone finding my user name/pwd will only be able to muck with my files, not the system (and not with other users on the machine).

also, I have root disabled (and will never enable it) so an attacker would have to guess my only admin user name....

oh and one more thing - the default admin user is the first account set up on OSX, and the short user name is built from their full name (Joe Bloggs = jbloggs). The 'Computer Name' as set in the sharing pref pane is by default "Joe Blogg's Computer". I reccomend changing that, cos otherwise people will be able to work out the short user name of your admin account on any OSX box just by browsing through the network.

My Red Hat password was actually a lot more substantial like:

a%foreu&she

Although what hoops posted shows 'no record' of failed password attempts, I would tentatively agree with you. An attacker can use 'traffic-analysis' techniques to interactive ssh connections to gather enough information about someone's terminal sessions to locate typed passwords in the session data stream and reduce by half the time needed to guess the password (the techniques involved are not specific to ssh but apply to most interactive remote-terminal protocols) and these sorts of 'information leaks' are very difficult to prevent.

.

There is something going on here guys. I find it really strange that two unix boxes with presumably good passwords were cracked. SSH may not have been the vector in, but it was used once the password was known. The first attack on the Mac seems to indicate the sort of blind password-guessing done by an automated process. Once a way in was found, the human at the other end took over a minute to notice and get in, right? Unless this was some sort of decoy to cover the real attack, then this password was just 'guessed'? Pretty damned lucky, huh? Of course, by starting with the Windoze box, which I think we all would agree makes more sense as the first vector in, the cracker would then be in a better position to capture network traffic and attack your passwords in some other-than-brute manner.

Too bad there is no forensic from the Red Hat attack. It makes me wonder how to set up a system for moving logs to a read-only media. I would love to have them on a CD, burning a session a night or something. Anyone else have some good ideas/experience with protecting logs?

Direct them to another source via /etc/syslog.conf. You can then harden that source as much as you like. Disallow all outside connections other then sysloging.


*.debug @foo.domain.org

So let's say I root your box. First thing I am going to do is read all the logs and check cron tasks. I will find this connection to your outside source. What's to stop me from following it and blowing away the files there? Once I am root, I mean. I still like the idea of read-only media...

There's nothing stopping you. Once a box is owned, well, it's owned.

But this isn't controlled by cron and it wouldn't appear in the local logs. Every log entry at level debug or higher automatically is sent and logged on the remote source in "real time" (limited by network latency). The only way it would be found was if you looked at syslog.conf, or checked outbound connections with netstat, or open files with lsof -i. By the time you did all that and put a stop to it, I'd still have all the logs up to that point on a "secure" remote source. Naturally your network speed is a possible limitation here where syslog entries can get munged because they can appear on the remote server incomplete, but it's more data then you would have had locally.

Centralized logging, in addition to local logging, is good UNIX admin practice.

Ultimately, IMO, this is more effective then (assuming it was possible) streaming to a read-only source. Burning would have to be automated and would be caught if someone looked at the crontab. The cache used to create the burn would probably be found as well and removed. Etc, etc. You'd be in the same boat as what you detailed above. The only advantage I can see in streaming to read-only media is that it would be unexpected, so it wouldn't necessarily be caught right away.

streaming to a read-only source.... how about something as simple as an email to a web-based gmail account? no passwords to the email account would be stored on the box itself, so the messages would be irretrievable to the root-ed hacker.

the log files might have to be zipped up and encrypted prior to transmission, of course...

I think the centralized logging idea is a good one, and if I had more than two computers, I would probably head that way. :-)

Do you think breaking into your centralized logger is going to be more difficult for someone capable of what we have seen? I don't pretend to be that good yet, but I know that Buddha himself isn't going to be able to remove the logs I manage to save to a CD.

I am looking at the options that hdiutil gives for automating this idea. Looks promising. The more I think about it, the more I like the idea of incrementally writing to a read-only media in general. I think I could come up with all kinds of uses for this.

As far as an Email, that's a good idea too. I don't know how to include attachments from the command line yet. :-)

It is if there's it's firewalls to disallow any external connections with the exception of syslog traffic from specified machines. At most, a hacker could fill my logs with useless crap from a r00ted box.

Are you a key target? What sort of business are you in? Those are more problems than I'd expect any average small business to run into.

Really, you should consider the VPN idea.

I don't recall seeing any mention in this thread of not allowing passwords at all....

I do not allow ssh logins with passwords, I can only login with a key and a pass phrase.

I also limit my ssh connections from IP addresses I know.

Stetner,

Is this kind of like what you are talking about? I followed the instructions up to the part about the "ssh-agent". I don't mind typing in a password to access my key.

If you have any more good resources, could you post them for us? Thanks!

While stetner is on call...
When you said "I also limit my ssh connections from IP addresses I know." did you mean using the 'only_from' option in '/etc/xinetd.conf' ?

If so, did you have to set up some things before using it? Some difficulty with using the 'only_from' option in /etc/xinetd.conf is mentioned in this thread. According to that thread, there is a bug that prevents this from working correctly without making another change in /etc/xinetd.d/ssh . I'm not sure it is a bug, but perhaps the protocol IPv6 must be set up.

Yes, that is what I was talking about. That seems like a pretty good resource for setting up keys, don't know of any that are better than that one....

Occasionally when I am doing stuff that will require me to connect a number of times, like working on some source that is stored in SVN which I access over ssh, I will fire up ssh-agent and use it, but I tend not to use ssh-agent all the time as I don't mind typing in a pass phrase either.

Yes, I am using only_from in xinetd. I have not had a problem, but I have disabled IPv6 on my system and my xinet.d/ssh file looks like this:so in my case I am allowing access from ranges of IPs due to the way proxies work, I am not guaranteed to get a specific IP, but I know it will be in a certain range. Less secure than a single IP, but more secure than open slather.