We'll discuss the unix logfiles, how the administrator of the system can secure the system enough to have log files containing trustful and reliable information. We'll also discuss the possibilities for the hacker to trick the system logfiles and delete the traces of his activities. As always, I'm trying to discuss the things from 'both sides of chessboard'. Hackers and administrators are all the same, most of good and security-minded admins did some annoyance in past by hacking. In any case, if someone wants to secure his system well, its really important to understand things from both sides.
Adminstration of system logfiles guarantees the unix systems syslogd daemon. this software has been around for a while, in fact, it hasn't been changed really in principle within last 4-5 years. because of that, syslogd is almost identical at all commercial and noncommercial flavors of unix. all of them grew from old BSD sources. But the difference is in the way its defaultly configured at particular distribution. At fresh installation of AIX or Solaris, syslogd logs almost nothing by default. Its because commercial UNIX stations are still used for the purposes inside the company's intranet, rather than as Internet servers. But many times, when they're decided to serve as Intranet servers, administrators often do not configure syslogd for this kind of purpose. Distributions of Linux and FreeBSD are usually quite-well pre-configured. However 'quite-well' is often not enough anyway. Here comes a few possibilities how to improve security of logging:
Logging to remote server
If you start syslogd with '-r' switch, it enables you to forward the log entries to another machine. You need to add into /etc/syslogd.conf a line
and all of the logs will be stored at the machine loghost as well. Thus, in case anyone compromises your system, and deletes the log files, you can check the logs at 'loghost' machine (ofcourse, supposing it wasn't hacked as well). If there is more unix machines in your locality, as it is usually at the ISP's, its a good idea to dedicate a separate machine just to store system log files of all the other servers. This may really be just some old unused i386 computer, and it should be totally secured -- no running daemons, no remote access, no mail, nothing, except the console access. Such machine is not only perfectly secured, but its also useful to have all server's logs at one place, if the adminstrator needs to dig some information about the servers and network activity.
Logging to console
Any script kiddie is capable of deleting file. For this you can print the log entries to the terminal. Also, it's kinda useful to have such report by hand visible directly at the terminal in realtime. You can do that in linux with :
Sure, its not so hard to trick that, its enough for hacker after he compromises the machine, to edit logs, delete the traces, followed by something like 'tail -100 /var/log/messages >/dev/ttyp12', and root will have faked information at the terminal. But honestly, nowadays, when the internet is just full of wannabe hackers who do not bother to peek into /etc/syslog.conf, it's highly probably that under-average wannabies won't notice that logs are logging to console (translator's note: until they read this article!). Regular script kiddie, who can just run exploit code from shell, can be tricked very easy by setting up syslogd to log to some unusual directory. In general we could say, if you configure your server daemons differently from default system settings, it may confuse hacker enough, that he won't be able to delete his traces. And for the hackers, there's a good rule to read carefully important /etc files, and to know how this system configured and where the important files are. To rely on default settings, may be for hacker a deadly mistake.
linux has the possibility to set up a kernel in such way, that some of the files wouldn't be deletable or modifiable, just allowing append, as described in one of phrack zines in the form of kernel patch.
Syslogd with encryption?
I have been thinking for a long time about syslogd version, that would encrypt the stored log files. I think it wouldn't be that hard to do, just to use some of regular DES libraries for example.. I doubt I'm the only one with this idea, but I haven't seen such syslgod version anywhere yet.
Binary log files
Another type of logs are binary logs: utmp, wtmp, lastlog, and acc. utmp (mostly /var/run/utmp) including the information about momentary logged users. wtmp (usually /var/log/wmp) holds information about who was logged from where, when and how long in the past. lastlog (/var/log/lastlog) holds the information about last log in of every user. acc (/var/log/acc) has info about all running programs. these files have binary structure, where you can study their formats in documentation available for each of them (for ex. # man utmp). However, there are trace wipers available for almost every platform. Still, many administrators trust the information stored in these binary logs. many of them have even no idea that they're editable, thus, if you wipe your traces from these files, you can trick a lot of administrators.
Regarding the aspect of psychology with adminstrators, in my opinion, its a better idea to create false traces, rather than just delete all the traces after the attack. For this purpose the ingenious bebet's wtkill, which allows to add records to wtmp. Think of this example: you hack a system and change index WWW page of some website there. Shocked root runs to the server and starts to dig in the logs. If you wiped all traces after you, it won't make him sleep well, and he'll just won't stop searching and searching and searching, until he really finds something that could be used to trace you. So by a psychological point of view, it is best to create some false log entries into /var/log/wtmp, messages, .bash_history, etc.. all information must of course fit other perfectly.
By this hack you can usually let your tools and helping scripts run on his server, he will find the false log entries, and being satisfied enough, he'll start to trace the attacker the way you align him. A good idea is to create false traces leading somewhere to another part of the world, to some university lab PC, where there is public access for students. This way, all the adminstrator can do is to exchange a couple of helpless e-mails with innoncent admins of that mentioned lab PC. Or, just create the traces leading to some gigantic
After cramming you with various hackers' methods, let's go back to some boring technical things :) In terms of security it's good for an administrator to set the users login at least 500 lines of shell history. This history may be often be used by admin as a source of information in the case of an attack. Not every hacker knows how to turn logging of history off. Most elegant way is to change shell environment variables immediately after login, to not log the history.
For example in bash:
eventually by changing the values of HISTSIZE or HISTFILESIZE, for more info check '# man bash' or '# man tcsh'
A really funny way of logging is snooping - monitoring remote shell access to your server in real time. In linux, there is for ex. telnetsnoopd. For this purpose, by which you need to replace your telnetd, this program is a part of standard slackware distribution. Sure there are kernel modules that allow snooping, just dig the net.
For a hacker the snooping can be a source of many funny stories. many of regular users, even admins, don't know that such snooping tools exist. That's why, if you jump to someone's terminal session and write directly to his shell, he is going to deem you as ultra-genius hacker, and his jaw will fall down by surprise! There are numerous funny stories related to snooping. my favorite is when I once jumped into a telnet session of a friend of mine who was arguing with his former girlfriend via IRC. I was watching their quarrel for a while. After some time, he wrote to his gf that he'd be gone for a while (away of computer). Right after I entered his telnet session continuing conversation with his girlfriend (she's thinking that its my friend still), and I managed to get 'myself' (persuading I'm my friend) back together with the former girlfriend that way, that when he (having no idea about the conversation) came to her to dorm room, she was waiting for him... horny, wet and open :) Sure, it was a big surprise for my innocent and none suspecting friend :)
Damn, yes I know, I promised pure technical information. Ok, ok. Except the syslogd and binary logs, there are numeours other programs which guarantee monitoring of a unix system. I personally like cZert sniffer 'czsniff', which sniffs traffic on the ethernet. You can set it up to log all the connections to the machine and encrypt data into the file. its a useful logger. DES encrypted and hidden files usually won't be modified or edited by a hacker who wants to wipe his traces.
As for all the other programs, I'll mention only real secure 4.2 from Internet Security Systems. Even though it is commercial and pretty expensive software, it has many interesting features. It's a complex sniffer that analyses traffic on the network, and which should be placed right behind the router (firewall). Real secure is able to recognize usual regular traffic from the hacker's attack, like for ex. someone starts a regular telnet session - it's fine, but if it detects 50 connections per minute, it immediately blocks the access from particular IPs etc. It can recognize by the packet pattern many remote root exploit scanners, scanning for a password using public services, dozens of denial of service attacks, and many types of flooding etc. It reports found attempts to compromise the network to console and writes reports to the files. As it should be installed on a separate machine, that is not running any remote services, it should be unhackable and invisible. It's easy to configure and it can only detect the attack, but alarm the admin by SMS/pager/email, and eventually react with a counter attack by access restrictions, etc.
It allows having a security management workstation where it will forward all logs and sniffer reports from all the servers. I have a vision and see the possibility to build business just on specialized teams of professionals, who's job is just to monitor, analyse incoming reports, and react in order to protect the customers' networks.
Access time file attributes
Security of internet servers progresses with mile steps. nowadays there is a lot of institutions who employ security professionals. Those are not so easily tricked, just by erasing traces from /var/log/wtmp and /var/log/messages.
When you're erasing the traces, you need to think even about the smallest of details. The example: access time file attributes. After the attack, it's essential to set (see '# man touch') the access times of all modified files - ones you've read, or copied. Especially if you're attacking the machine in order to gather particular information from the server. The administrator can easily tell if you've accessed sensitive documents, by checking access time attributes of those files (ls -lau). By setting these attributes you can hide the fact you've accessed something you were not allowed to.
On the other side, for the administrator, access time attributes are a rich source of information after the attack. If you, as admin, are suspicious that there was a hacked account of some particular user, but there is no trace in wtmp or syslogd logs, try to check access timestamps of the files in that user's home directory (for ex. .cshrc, .bash_login etc.). If wtmp says that the user logged last time a week ago, but his .cshrc was readed yesterday's night, it's clear that something is wrong.
If you're a beginner hax0r, note that cleaning /var/log/wtmp and /var/log/messages is in no way perfect hiding against traces. Realize that on the internet there has already been for a long time anarchy of the wild wild west, where you can mess without a penalty. If you've already hacked some machine, it's highly probably that it uses some of the mentioned logging methods described above, and that the admins will find out and --f*ck you up badly--. If you're an admin, the aim of this article was to not blindly believe information in log files, and that if you want to have a trusted and stable source of information about activity on your server, the default configured syslogd and binary logs are just not enough.