Thursday, February 3, 2011

What are you missing? AIX

Happy February Readers,
I didn't want to miss last week's posting, but I also didn't have the time to make a quality post before leaving on a trip. So quality over quantity will hopefully gain favor with you. I'm taking a break in the What was wiped series to give myself some more time to gather what I need and instead I am continuing the What are you missing series in this post.
Doing forensics on specialized servers, which I will define as anything non wintel and whose file systems have no parsers supported in forensic tools, is an interesting challenge. You have to:
1. Research where the system log files exist
2. Determine what format the logs are in
3. Capture the metadata of the file system
4. Determine if the file system can be parsed by anything but the running OS
5. Determine if it's feasible to image the server via DD
6. Determine if here is any hardware specific evidence that exists
A good example of this would be an older AIX system as detailed below


1. Research where the system log files exist:
AIX does not have syslog on by default, rather it uses 'errdemon'.
Those logs are stored in /var/adm/ras/errlog by default, it's also quite limited to 1mb by default but this can also be adjusted. Keep this in mind as it may limit how far back your log evidence goes. Also this file may overwrite itself after the buffer is filled, though I have found some references to a .old log being created so I'm not sure if this is a variance in versions of AIX.

Remember that many people do still enable syslog and when they do the /etc/syslong.conf will still tell you where those logs reside.

2. Determine what format the logs are in:
The log file /var/adm/ras/errlog is a binary file so you cannot immediately access it. IBM provides a tool called 'errrpt' that can generate the logs you need to capture. Your ability to do this after the fact depends on your ability to have another running AIX system parse the file as I know of no ported version or open source parser for the format.

Once you have them generated from errrpt you have something like this as a summary:

IDENTIFIER TIMESTAMP T C RESOURCE_NAME DESCRIPTION
8650BE3F 0820122810 I H ent2 ETHERCHANNEL RECOVERY
F3846E13 0820122510 P H ent2 ETHERCHANNEL FAILOVER
8650BE3F 0820104410 I H ent2 ETHERCHANNEL RECOVERY
F3846E13 0820093810 P H ent2 ETHERCHANNEL FAILOVER
8650BE3F 0820090910 I H ent2 ETHERCHANNEL RECOVERY
CB4A951F 0819114610 I S SRC SOFTWARE PROGRAM ERROR
CB4A951F 0819114510 I S SRC SOFTWARE PROGRAM ERROR
DE3B8540 0817101710 P H hdisk0 PATH HAS FAILED

Taken from: http://answers.splunk.com/questions/6248/aix-errpt-timestamp as showing client data in blog posts would be regrettable.

The timestamp is in the following format:
0820122810 (08 = month, 20 = day, 12 = hour, 28 = min, 10 = year)
And something like this as a full log per error:
     LABEL:            JFS_FS_FRAGMENTED
     IDENTIFIER:       5DFED6F1
     Date/Time:        Tue Feb 20 05:43:35
     Sequence Number:  146643
     Machine Id:       00018294A400
     Node Id:          rescue
     Class:            O
     Type:             INFO
     Resource Name:    SYSPFS
     Description
     UNABLE TO ALLOCATE SPACE IN FILE SYSTEM
     Probable Causes
     FILE SYSTEM FREE SPACE FRAGMENTED
     Recommended Actions
           CONSOLIDATE FREE SPACE USING DEFRAGFS UTILITY
     Detail Data
     MAJOR/MINOR DEVICE NUMBER
     000A 0006
     FILE SYSTEM DEVICE AND MOUNT POINT
     /dev/hd9var, /var
If you wanted to parse that data back to a csv for your review or to load into something like log2timeline (with some tweaks since it's not officially supported) you can use the following perl code:
#!/usr/bin/perl -w

use File::Find;

$path = shift;

$next = 0;

$final = 0;

open (OUTFILE, ">ibmerrors.csv");

find(\&wanted, $path);

sub wanted

{

$file = $_;

if ($file !~ /^errpt.+txt$/)

{

next;

}

open (INFILE, "<$file");

while ()

{

$line = $_;

chomp($line);

if ($line =~ /^---/)

{

$final =0;

print OUTFILE "\n";

next;

}

elsif ($line =~ /^LABEL:\s+(.+)/)

{

print "Date: $1\n";

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^IDENTIFIER:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^DATE.TIME:\s+(.+)$/i)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Node Id:\s+(.+)$/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Class:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Type:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Resource Name:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Resource Class:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Resource Type:\s+(.+)/)

{

print OUTFILE "$1,";

next;

}

elsif ($line =~ /^Description/)

{

$next=1;

next;

}

elsif ($line =~ /^Failure Causes/)

{

$next=1;

next;

}

elsif ($line =~ /^Recommended Actions/)

{

$next=1;

next;

}

elsif ($line =~ /^Detail Data/)

{

$final=1;

next;

}

elsif ($line =~ /^PROBLEM DATA/)

{

$final=0;

next;

}

elsif ($line =~ /^FILE NAME/)

{

$final=0;

next;

}

elsif ($line =~ /^Event Information/)

{

$final=0;

next;

}

if ($next ==1)

{

print OUTFILE "$line,";

$next=0;

next;

}

if ($final ==1)

{

print OUTFILE "$line ";

next;

}

}

}

which will go through a directory of generated error reports if they are named like errpt*anything here*txt and generate a csv of all their contents named ibmerrors.csv. I can provide this as an executable if you need it.
3. Capture the metadata of the file system:
If you can't parse the file system after the fact, as in JFS v1 which AIX runs by default, you have to capture the metadata first before you change anything. I do this by doing a recursive ls in the following order to capture it:
1. ls -a -F -i -m -R -l -u / > *systemname*-accesstimes.csv
2. ls -a -F -i -m -R -l -e / > *systemname*-modtimes.csv
The order is important as we want to capture the access times first as they are more volatile than the modification times.
4. Determine if the file system can be parsed by anything but the running OS:
In most AIX systems the file system is JFS v1. While JFS v2 is open source and you can later mount and parse it using free and commercial tools there exists no such option for JFS v1 as far as I know (if you know of something that does PLEASE let me know). So because of this we will not be able to recreate the file system if we capture the disk to an image unless we restore it to similar hardware and then boot it back up.
In cases where you have multiple drives in your AIX system, which is most of the time, you further have to deal with volume groups. In older AIX systems there was no hardware RAID instead volumes where spanned across disks in order to shield against risk (as I was told). The only way I know how to recreate these volumes after the fact is to restore the images to a new set of disks, or if you VERY confident in your abilities the original disks, and boot off of AIX installation media.
From there you can go into single user mode, probe to find and then reassemble the volume groups and mount them read only to do your analysis. While this isn't a great method its one that works, and the only one that works for post mortem analysis of AIX systems that I know of.
It can get more interesting if one of the volume groups is using a file system that is application specific, such as a database. You will have to get the database application running, or at least the libraries that support the utilities, to then mount the volume and access the underlying data. I found this to common on many financial institutions AIX systems.
5. Determine if it's feasible to image the server via DD :
Will you benefit from post mortem analysis?
Does the client intend to pursue legal action afterword's?
Is the size of volumes going to permit you to actually capture an image to the storage you have?
If the answer to these questions is yes then you need to attach a network storage location to the system as they have no USB/Esata/etc... hot swap ports. They also do not support SMB out of the box so NFS will be your first choice. There are still NFS utilities for windows and of course Linux that will allow your local external storage to become your image repository. If you are running windows and have access to MSDN follow these instructions to get the Microsoft NFS client running http://support.microsoft.com/kb/324055.
As far as how to capture the disk image, the only utility that will work out of the box is dd. If you intend to capture an image and intend to use another tool that has already been cross compiled for AIX, cc/gcc do not come standard typically, then that works. Otherwise I would not recommend writing a tool to the local disk before imaging it.
If you are going to do this remember to perform this step first!
6. Determine if here is any hardware specific evidence that exists:
When dealing with non wintel hardware it's important to research and find out what is implemented. In the case of AIX servers there is a NVRAM storage dedicated to error logging called FFDC which stands for First-Failure Data Capture. In the event of a system error that would otherwise prevent an error from making it out of the write cache or output buffer and committed to disk before the loss of power/function the FFDC log will capture it in NVRAM.
These logs can prove to be very valuable depending on your investigation.
In order to get these logs you have to run the 'fcstkrpt' command, syntax can be found here:
Update: I should state that even though you can't parse the file system, an image of the disk would still allow you to carve out known file formats such as log files from the image.