AIX Health Check documentation

Index
  1. Obtaining Support
  2. Using AIX Health Check
  3. Reports generated by AIX Health Check
  4. Installing AIX Health Check
  5. Determining the version of AIX Health Check
  6. Running checks individually
  7. Using the checkall.ksh master script
  8. Using options with the checkall.ksh master script
  9. Return codes
  10. Running categories of scripts
  11. Automating the download of AIX Health Check
  12. Enabling password-less access through SSH on PowerHA/HACMP nodes
  13. Resolving email issues
  14. Resolving issues found by AIX Health Check

1. Obtaining Support
Getting support for AIX Health Check is easy: Send us an email at support@aixhealthcheck.com or complete our online Contact form. When you ask for support, please include the following information:
  • Your contact information.
  • A description of the issue you've encountered.
  • The output file generated by AIX Health Check that includes the issue.

2. Using AIX Health Check
We provide AIX Health Check services to scan entire AIX systems. Just like going to a doctor for a full check-up. You'll get a report of all features of your system, and how the system is doing. If the system is properly installed, configured and maintained, an all green report with a score of 100% will be given. If for some reason some things are off, we'll let you know in the report what is wrong, and how you can fix it.

Your goal should be to get the AIX system(s) on an AIX Health Check score of 95% or higher. There is no need to aim for a 100% score; there are always reasons why a certain system is configured differently than how the best practices would recommend to configure a system.

AIX Health Check can be beneficial in several stages of an AIX system lifecycle. Most of our customers simply run the entire health check daily, to provide them with a report of their server status. But doing an AIX Health Check is also very useful at certain important points in the life cycle of an AIX system:
  • Before putting a new system into production - to make sure the system was installed and configured correctly.
  • Before and after doing a change on a server - to make sure no issues exist before applying patches, and that no adverse situations were introduced as part of a change.
  • During audits - to make sure a system is compliant with company and federal regulations.
  • During yearly performance reviews - to determine if your system administrators have taken good care of your AIX servers.
  • Before or during any Bare Metal Restore or Disaster Recovery exercises - to validate that an AIX server has been recovered correctly.
  • When taking over AIX systems from third parties or other accounts to perform an initial quality scan.
  • ... and of course:
  • As part of the daily routine to check and monitor systems, because not daily reviewing AIX systems can have costly consequences.

3. Reports generated by Health Check
Reports are generated by AIX Health Check in different formats. By default, it will generate a report in TEXT format. You can also select to have a report generated in web page (HTML) format, or comma separated (CSV) format, or in Extended Markup Language (XML) format. Use one of these options to select a specific format:

OptionExtensionDescription
-l.logTEXT format (default)
-h.htmlWeb page format (HTML)
-c.csvComma separated format (CSV)
-x.xmlExtended Markup Language format (XML)

If you combine any of the options above, the order of the options used, determines which format is used to generate the report. Always the option that is mentioned last, will determine the format. For example, if you selection options "-l", "-x" and "-h", the format of the report is HTML, becuase the "-h" option (for HTML output) was mentioned last.

A report generated by AIX Health Check consists of multiple parts:

  1. A copyright message.

    This copyright message states that the use of AIX Health Check is subject to license agreements, along with displaying information on how to make changes based on issues discovered by AIX Health Check, and also information about how to get additional support. You can suppress displaying this information by using the -b option.

  2. An overview of the options selected when running AIX Health Check.

    This section will provide display the options selected when checkall.ksh of AIX Health Check was run. You can suppress this output by using the -b option.

  3. A system configuration section.

    This is a short overview of many configuration items of the server, which includes items like the serial number of the server, the amount of memory, the CPU clock rate, and the OS level installed. The system configuration section is not displayed when the -b option is used.

  4. The output of individual check scripts.

    In this part of the report you'll see the output of individual check scripts that were run on the system. For each check script, it will show the name check script that was run, a description of the check script (if the -d option was used with the checkall.ksh script; the -d option displays descriptions of check scripts), the output of the script, and if applicable, any errors that were generated when the check script was run. At the end you'll also see the returncode of the script (See Return codes), along with the amount of time in seconds that it took to run the check script. If the -g option (lowercase G) was used with the checkall.ksh script, then only the output of individual check scripts is shown in this section when any of the check scripts returns a non-zero returncode. If the -G option (capital G) was used with the checkall.ksh script, then only output of individual check scripts is shown in this section when any of the check scripts return an ERROR (returncode 1). Any check scripts that return a WARNING (returncode 2) are ignored with the -G optionk scripts that return a WARNING (returncode 2) are ignored with the -G option..

  5. A results summary.

    This section displays the total run time for all check scripts, the number of check scripts that were run, the number of checks that returned an OK (returncode 0), WARNING (returncode 1) and ERROR (returncode 2), and also a total score for the system. It will finally point to the location of the output file that was generated by AIX Health Check.

4. Installing AIX Health Check
The AIX Health Check software will be provided to you once a purchase order has been received, or when you've acquired AIX Health Check online. The software will be made available to you through download from our website. As soon as we've received a purchase order, or your online payment, we'll send you a login and password, so you can download the software.

Transfer the AIX Health Check tar image (called "ahc_latest.tar") onto the AIX system you wish to run it on, and un-tar the file. It is recommended to un-tar the file in a separate directory. Any directory will do; you can transfer the tar image into the directory of your preference and un-tar it there. The example below assumes that you will be installing the software in folder /ahc. We recommend that you create a separate file system of at least 64 MB in an external volume group, and use that file system for storing the AIX Health Check and any reports generated by it. We also recommend that you run AIX Health Check initially on any of your test servers, before running it on any production server.

Note: All the commands shown in this documentation are to be run as user root; AIX Health Check requires root, and no support is available if AIX Health Check is run using any other account.

AIX Health Check requires 64 MB to be free in the /var and /tmp file systems. If that space is not available, either try clearing out these file systems by deleting old files, or you may try extending the file systems:
# chfs -a size=+64M /var
# chfs -a size=+64M /tmp
AIX Health Check requires that the ncargs setting on the AIX system is set to 256 or higher. This setting determines the maximum number of command line arguments that can be used on the system, and due to the large number of check scripts available in AIX Health Check, this value should be 256 or higher. The default value for ncargs in AIX 6 and AIX 7 is 256. On older AIX versions you may encounter that it is still set to 6. Also, on some older AIX versions, ncargs can't be configured higher than 128.

To check the current ncargs setting, run:
# lsattr -El sys0 -a ncargs
To determine the maximum value ncargs can be configured to, run:
# lsattr -R -l sys0 -a ncargs
Set ncargs to either 256, or if 256 is not supported, set it to the highest available value, such as 128, by running:
# chdev -l sys0 -a ncargs=256
Create the installation directory:
# mkdir /ahc
# cd /ahc
Move the AIX Health Check tar image into the directory (assuming you have placed the tar image in /tmp on the system that you wish to run AIX Health Check on):
# mv /tmp/ahc_latest.tar /ahc
# ls *tar
ahc_latest.tar
Next, you may verify the check sum of the downloaded tar file. When you download the tar file from the AIX Health Check website, you can find both the file size and the checksum. For example:


Note: This is an example. Please verify the actual file size and check sum while downloading AIX Health Check.

The check sum of ahc_latest.tar can be easily determined by running:
# cd /ahc
# sum ahc_latest.tar
12311  2300 ahc_latest.tar
Also, please verify the file size matches with the filesize provided when downloading AIX Health Check, to ensure that you've correctly downloaded the AIX Health Check tar file:
# ls -als ahc_latest.tar
2304 -rw-r--r-- 1 root system 2355200 Jun 11 13:53 ahc_latest.tar
Then, un-tar the image:
# tar -xvf ahc_latest.tar
x checkactivatedrpcservices.ksh, 1625 bytes, 4 media blocks.
x checkadaptersdefined.ksh, 297 bytes, 1 media blocks.
x checkadapters.ksh, 319 bytes, 1 media blocks.
x checkaiooa.ksh, 334 bytes, 1 media blocks.
x checkaiostatus.ksh, 1765 bytes, 4 media blocks.
x checkall.ksh, 26286 bytes, 52 media blocks.
x checkaudit.ksh, 235 bytes, 1 media blocks.
...
...
[lines removed]
...
...
x checkxntpd.ksh, 2557 bytes, 5 media blocks.
x checkzombies.ksh, 407 bytes, 1 media blocks.
x COPYRIGHT, 930 bytes, 2 media blocks.
x DESCRIPTIONS, 124444 bytes, 244 media blocks.
Once the tar command completes un-tarring the image file, you will find a large number of check*ksh scripts in the same directory, along with the master script, checkall.ksh, and some other files like CHANGES, COPYRIGHT, COUNT, DOCUMENTATION, FAQ, SUPPORT, VERSION and DESCRIPTIONS.

File checkall.ksh is the script that can be used to run more than one check script, and is referred to as the master script. You will generally use it to run all the scripts, for example once every day.

Files check*ksh (except for checkall.ksh) are the actual checks that are performed by AIX Health Check. You should find several hundreds of those, at least 1000. You can run each script individually if you like, but it is common to run all or part of them through the checkall.ksh script. No check script will ever change anything on your system. It is safe to run them at any time, without affecting the system in any way. However, due to the nature of the commands run by some check scripts, it may be possible that some harmless errors may occur in the error report (for example when unavailable disks are attempted to be accessed by a check script). Please do not be alarmed by this; it is normal behavior.

File APARS is a file containing all high severity APARs available for AIX, as released by IBM. The source of this file is http://www-304.ibm.com/webapp/set2/flrt/doc?page=aparCSV and is used by AIX Health Check to check for the correct APARs installed on the AIX system.

File CHANGES is an overview of the most recent changes to AIX Health Check.

File COPYRIGHT is the copyright message that is part of AIX Health Check.

File COUNT shows the number of checks that is available in the full version of AIX Health Check.

File DEMO (only present in demo version) provides information about the demo version of AIX Health Check.

File DESCRIPTIONS contains a description for each check script available in the AIX Health Check package. It will provide you information of what each script does. Also, each check script has a description in the header of the file. By simply viewing a check script in an editor such as VI, you will be able to see what the check script is actually checking. AIX Health Check will include the description for each check in the report, if you add the "-d" option; we'll discuss that later on in this documentation.

File DOCUMENTATION provides a link to this web page, so you will never have to search for the latest documentation for AIX Health Check.

File FAQ provides a link to our frequently asked questions, also on the AIX Health Check website.

File SUPPORT provides a link to our support page on www.aixhealthcheck.com.

File VERSION contains the version of AIX Health Check.

When the ahc_latest.tar image has been un-tarred, you can safely remove the tar image:
# rm ahc_latest.tar
The mode for all the check scripts should already be set to 755. Please verify that this is the case. If not, please run:
# chmod 755 check*ksh
Please repeat the installation of ahc_latest.tar on every AIX system, that you wish to check with AIX Health Check.

5. Determining the version of AIX Health Check
To determine the version of AIX Health Check that you are using, run the checkall.ksh script using the "-u" option. The "-u" option can be used to show all the different options for AIX Health Check, along with the version number:
# ./checkall.ksh -u

Copyright (c) 2004-2015 AIX Health Check - All Rights Reserved

www.aixhealthcheck.com

This is confidential and unpublished work of authorship subject to
limited use license agreements and is a trade secret, which is the
property of AIX Health Check (www.aixhealthcheck.com). All use,
disclosure and/or reproduction not specifically authorized in writing
by AIX Health Check is strictly prohibited.

Any expressed or implied warranties are disclaimed. In no event shall
AIX Health Check be liable for any direct, indirect, incidental,
special, exemplary, or consequential damages (including, but not
limited to, loss of use, data, profits, or business interruption)
however caused and on any theory of liability, whether in contract,
strict liability, or tort (including negligence or otherwise) arising
in any way out of the use of these scripts, even if advised of the
possibility of such damage.

Version: 15.01.12

Usage : checkall.ksh [ -b ] [ -c ] [ -C ] [ -d ] [ -D ] [ -e ]
                     [ -E script1,script2,... ] [ -f file ] [ -g ] [ -G ]
                     [ -h ] [ -i ] [ -l ] [ -m emailaddress1,emailaddress2,... ]
                     [ -n ] [ -s script1,script2,... ] [ -u ] [ -v ]
                     [ -w # ] [ -x ]

 -b : Basics: no system configuration, end results and copyright message.
 -c : Write comma separated output (CSV).
 -C : Comma separated list of categories of scripts to run.
 -d : Add descriptions of checks to output.
 -D : Enable the debug option - for AIX Health Check support team use only.
 -e : Exclude ALL check scripts (will only run system configuration).
 -E : Comma-separated list of check scripts to exclude from the run.
 -f : Specify location of output.
 -g : Suppress showing output of check scripts that complete successfully.
 -G : Display check scripts that complete with ERROR; no WARNINGs.
 -h : Write web page output (HTML).
 -i : Inventory. Deprecated option. Use "-C inventory" instead.
 -l : Write text output to log file (TEXT) (default).
 -m : Comma-separated list of email addresses to send output to.
 -n : Only send email when result is less than 100% (errors only).
 -s : Comma-separated list of check scripts to run.
 -u : Display usage information.
 -v : Verbose output to screen.
 -w : Specify the width of the output.
 -x : Write Extended Markup Language output (XML).
The example above shows the version number after the copyright message. Another way to determine the version of AIX Health Check is to cat the VERSION file:
# cat VERSION
The version of AIX Health Check is a reversed date. Version 13.08.19 is the release of AIX Health check on August 19, 2013.

Be sure to check regularly for updated versions of AIX Health Check, as they are released often. The latest version is always available on our Support page (Click the "download the latest version of AIX Health Check" link, and log in with the user credentials provided to you upon purchase of AIX Health Check).

There's also a possibility to automate the download of AIX Health Check directly to your AIX systems. For more information see: Automating the download of AIX Health Check.

6. Running checks individually
Once the tar image has been unpacked, you will notice a lot of files within the chosen directory. Most of the files are individual check scripts that start with "check". Each individual script (except for script checkall.ksh) will check a certain function or configuration. You can run each script individually if you like. For example, to check if wget is installed, run checkwget.ksh. To determine the model name of the AIX server, run checkmodelname.ksh.
# checkwget.ksh
wget-1.9.1-1
# checkmodelname.ksh
9117-MMB
Please note that AIX Health Check is designed to run as user root only. Many scripts will still run using a different user account, but AIX Health Check is only supported by running via the root account. Root access is required, because AIX Health Check runs several root-level commands. Also please note, that AIX Health Check does never change anything on the AIX system; it only reports. AIX Health Check is not designed to automatically resolve any issues found, because the configuration can depend on your environment or infrastructure. From the output of the check script(s), you can determine what issue was found (if an issue is found), and what possible action should be taken, to remediate the issue.

7. Using the checkall.ksh master script
As described earlier, most users will usually use the checkall.ksh master script to run either a selection or all of the check scripts to generate a report. Running checks individually can be tedious task with over 1000 different check scripts. You can use the checkall.ksh master script to run a combination of different checks. For example, if you wish to run check scripts checkwget.ksh and checkmodelname.ksh on your server, you can use the "-s" (for "Script") option:
# ./checkall.ksh -s checkwget.ksh,checkmodelname.ksh
Multiple scripts can be specified using the "-s" option. Scripts need to be separated using the comma (",") only.

Please note that when you run the above command, that no output will be shown on the screen. When the checkall.ksh script completes running all or the selected scripts, it will simply return the command prompt. AIX Health Check will log it's output by default to a log file. After running the above command you will be able to find a log file within the directory in which checkall.ksh is located:
# ls *log
checkall_hostname.log
You can open the log file with an editor and review the output. If you rather have AIX Health Check both log its output to a log file and also display the output on the screen while it is running, then add the "-v" (for "Verbose output") option:
# ./checkall.ksh -s checkwget.ksh,checkmodelname.ksh -v
You can determine the width of the output using the -w (for "Width") option. For example, if you wish AIX Health Check to generate a report that is only 70 characters wide, run:
# ./checkall.ksh -s checkwget.ksh -v -w 70
######################################################################

2015-01-12 22:12:34: AIX HEALTH CHECK

Copyright (c) 2004-2015 AIX Health Check - All Rights Reserved

www.aixhealthcheck.com

This is confidential and unpublished work of authorship subject to
limited use license agreements and is a trade secret, which is the
property of AIX Health Check (www.aixhealthcheck.com). All use,
disclosure and/or reproduction not specifically authorized in writing
by AIX Health Check is strictly prohibited.

Any expressed or implied warranties are disclaimed. In no event shall
AIX Health Check be liable for any direct, indirect, incidental,
special, exemplary, or consequential damages (including, but not
limited to, loss of use, data, profits, or business interruption)
however caused and on any theory of liability, whether in contract,
strict liability, or tort (including negligence or otherwise) arising
in any way out of the use of these scripts, even if advised of the
possibility of such damage.

This report is generated by AIX Health Check. It is an overview of
check scripts run on an AIX system, and depending on the options
selected when the checkall.ksh script was run, it will list each
check script, the returncode of the check script, the output of the
check script and a description. At the end of this report is an
overview of all scripts run and a score report.

Any individual implementing changes should completely understand the
change and deem each change appropriate before it is applied to the
system. As a standard rule, please take into consideration the impact
on other components before implementing the change. Also, we
encourage all to conduct a peer review of all changes before
implementation. Most importantly, if the effect of a change is not
fully understood, do not implement that change until a satisfactory
explanation can be given as to what the effects of the change are. We
recommend implementation of one change at a time. The application of
many changes all at once will increase the likelihood of confusion,
if issues arise.

For more information, check website http://www.aixhealthcheck.com.
For support, email to support@aixhealthcheck.com.

######################################################################

2015-01-12 22:12:34: OPTIONS SELECTED
-------------------------------------

2015-01-12 22:12:34: Version:         15.01.12
2015-01-12 22:12:34: Start at:        01/12/2015 22:12:34 CST
2015-01-12 22:12:34: Options:         -s checkwget.ksh -v -w 70
2015-01-12 22:12:34: Output file:     checkall_hostname.log
2015-01-12 22:12:34: Width:           70
2015-01-12 22:12:34: Display:         All checks
2015-01-12 22:12:34: Descriptions:    No
2015-01-12 22:12:34: Output type:     TEXT
2015-01-12 22:12:34: # Checks:        1
2015-01-12 22:12:34: Script:          checkwget.ksh

######################################################################

2015-01-12 22:12:35: SYSTEM CONFIGURATION
-----------------------------------------

2015-01-12 22:12:35: Hostname:        hostname
2015-01-12 22:12:35: IP Address:      192.168.0.11
2015-01-12 22:12:35: IP Assignment:   Static
2015-01-12 22:12:35: Subnet Mask:     255.255.255.0
2015-01-12 22:12:35: Default Gateway: 192.168.0.1
2015-01-12 22:12:35: Name Server(s):  192.168.0.30 192.168.0.31
2015-01-12 22:12:35: LPAR / VM:       6 hostname
2015-01-12 22:12:35: OS Level:        AIX 6.1.8.15 6100-08
2015-01-12 22:12:35: Model:           IBM,9119-FHB Power 795
2015-01-12 22:12:35: Serial Number:   02CG2ZR
2015-01-12 22:12:35: Firmware Level:  AH780_059
2015-01-12 22:12:35: Kernel:          64 bit
2015-01-12 22:12:35: Hardware:        64 bit
2015-01-12 22:12:35: Processor Type:  PowerPC_POWER7
2015-01-12 22:12:35: CPU Clock Rate:  4004 MHz
2015-01-12 22:12:35: rPerf:           23.27 rPerf estimated based on
2.00 Virtual CPU cores
2015-01-12 22:12:35: CPUs:            2
2015-01-12 22:12:35: Logical CPUs:    8
2015-01-12 22:12:35: Capacity:        Min: 0.10 Entitled: 0.20 Max:
0.40 Increment: 0.01
2015-01-12 22:12:35: Physical CPUs:   10
2015-01-12 22:12:35: System type:     mode=Uncapped type=Shared-SMT-4
weight=128 smt=4
2015-01-12 22:12:35: Virtual CPUs:    Desired: 2 Min: 1 Online: 2
Max: 2
2015-01-12 22:12:35: Memory:          2048MB
2015-01-12 22:12:35: Memory Settings: min: 1024MB, desired: 2048MB,
max: 4096MB, online: 2048MB
2015-01-12 22:12:35: Paging Space:    2048MB (1% in use)
2015-01-12 22:12:35: Uptime:          10:12PM up 12 days, 6:22, 1
user, load average: 0.18, 0.06, 0.03

######################################################################

2015-01-12 22:12:36: CHECK SCRIPTS
----------------------------------

----------------------------------------------------------------------
2015-01-12 22:12:35: Running check script 1 of 1: checkwget.ksh

wget-1.9.1-1

2015-01-12 22:12:36: Check checkwget.ksh completed successfully:
returncode 0
2015-01-12 22:12:36: Runtime: 1 second(s)
2015-01-12 22:12:36: 100% complete - 0 checks to go.
----------------------------------------------------------------------

######################################################################

2015-01-12 22:12:36: RESULTS
----------------------------

2015-01-12 22:12:36: Run time for all checks              : 2 seconds
2015-01-12 22:12:36: Total number of checks               : 1
2015-01-12 22:12:36: # Checks with result OK              : 1
2015-01-12 22:12:36: # Checks with result WARNING         : 0
2015-01-12 22:12:36: # Checks with result ERROR           : 0
2015-01-12 22:12:36: Score [Percentage OK/WARNING]        : 100.00 %

2015-01-12 22:12:36: For details see logfile              :
/ahc/checkall_hostname.log

######################################################################
Note that the width setting should be between 70 or 1024 characters. By default, AIX Health Check will produce output up to 130 characters wide.

To simply run all the check scripts, run:
# ./checkall.ksh
By not specifying the -s option, AIX Health Check's master script, checkall.ksh, will run all the scripts, and generate an output file. Please note that when run without the -s option, the checkall.ksh script will determine which scripts should be run on your system, and may not run all scripts. For example, if your system is not a VIOS, or is not part of a PowerHA/HACMP cluster, checkall.ksh will skip any checks related to VIOS and/or PowerHA/HACMP systems, reducing the number of scripts that it has to run, and shortening the time needed to run all the scripts. This is why you may see that checkall.ksh runs less checks than what is included in the full AIX Health Check software.

8. Using options with the checkall.ksh master script
Shown below are all the options you can use with the AIX Health Check master script checkall.ksh:

OptionDescription
-bShows only the basic output. It will not show any of the end results, the configuraton section will be skipped and also no copyright message will be shown.
-cThis option will write a CSV file output instead of the default log file (-l) output. CSV is comma separated output, which can be useful if you wish to use the information generated by AIX Health Check in Microsoft Windows Excel or in a database.

Note: The order of the options determine which type of output is written. If you specify both the -l and -c option, the last option provided will determine the type of output written. This also applies to the -h (for HTML output) and -x (for XML output) options.

By default, AIX Health Check will write CSV output to file checkall_hostname.csv in the same folder where AIX Health Check is run from.
-CThis option allows running scripts of a certain category. Specify a single category, or a comma separated list of categories of scripts to be run. For example to run all security scripts, use the -C option as follows:
-C security
Or to run both memory and storage related scripts, use the -C option as follows:
-C memory,storage
You can choose from the following categories:

backupRun all scripts that check the backup of the system, including TSM, EMC Avamar/Networker, OSSV Snapvault, Commvault Simpana, and Veritas Netbackup.
bootingRun all check scripts that check items of the boot sequence, including inittab, rc.d files, autorestart, and start scripts.
capacityRun all check scripts that check the capacity of the system, including storage, CPU, network and memory.
clusterRun all scripts that check cluster configuration and status, including PowerHA/HACMP and Veritas.
daemonsRun all scripts that check daemons and services on the system, including inetd.
datetimeRun all scripts that check date and time settings on the server, including ntp configuration.
debuggingRun all scripts that check dubug items, such as debug filesets, core dump settings, error logging and system dumps.
hardwareRun all scripts that check the physical hardware of the system, incluing adapters, devices, power supplies, firmware, and lpar configuration.
inventoryRun all scripts that generate inventory information and only display output of commands for inventory purposes.
loggingRun all check scripts for checking log files, alog and syslog configuration.
mailRun all script that check mail related items, such as the mail configuration and aliases.
memoryRun all memory scripts that check memory usage, vmo options, and memory cache.
monitoringRun all check scripts that check the monitoring of the system, including BMC Patrol, Ganglia, Marimba and Nimsoft.
networkRun all scripts that check network settings, DHCP, DNS, and NFS.
performanceRun all scripts that check the performance of the system, including those that check Active Memory Expansion.
powervmRun all scripts that check PowerVM configuration and settings, including dlpar, VIOS, and WPARs.
printingRun all scripts that check printing and printer queues, including Cups.
redundancyRun all scripts that check redundancy features and configuration on the server (does not include cluster items).
schedulingRun all check scripts that check the scheduling of jobs on the system, including at and cron.
securityRun all scripts that check permissions/owner of files, the ssh configuration, old files, user acounts and user limits.
softwareRun all scripts that check the software installed on the system, such as filesets (lslpp) and rpm packages, and additional software installed on the server.
storageRun all scripts that check the storage of the system, including SDD, SDDPCM, disks, and EMC Powerpath.

Please note that scripts may belong to multiple categories. For example, a script that checks if xmdaily is disabled in the root crontab, checkxmdaily.ksh, belongs to both the category "scheduling" and the category "monitoring", as the xmdaily entry in the root crontab controls running xmwlm/topasrec processes for monitoring purposes on the system.
-dThis option will add descriptions of each individual check script to the output. This is very useful if you wish to know how to resolve certain issues. The descriptions for check scripts can also be found in the DESCRIPTIONS file.
-DThis option will enable the debugging feature of the checkall.ksh master check script. Use this option only when instructed to do so by the AIX Health Check support team, for the purpose of gathering information regarding an issue with AIX Health Check.
-eThis option will exclude running all scripts. In normal operations, this will cause checkall.ksh to only generate the system configuration section.

The -e option is overruled by the -s option. If you specify one or more check scripts to run using the -s option, these check scripts will still be run, even if you specified the -e option.

The -e option however, overrules the -C and -E options. If you choose to run a certain category of scripts using the -C option, or if you exclude specific scripts using the -E option, then including the -e option will result in still all the scripts being excluded.
-EThis option can be used to exclude specific scripts from being run. This option is useful if one or more check scripts generate a warning or error, and you have made sure that the warning or error does not apply to your environment. In that case you can use the -E option to exclude specific scripts. Provide a comma-separated list of scripts that you would like to have excluded. For example:
# ./checkall.ksh -v -E checkbit.ksh,checkuptime.ksh
The -e option (lowercase) overrules the -E option (uppercase). If both are specified, the -e option will exclude all check scripts.
-fYou can use this option to specify a different location to write the output to. By default, the output will be written into the same folder from which the checkall.ksh script is run. You need to supply the filename, or the full path to the filename for this option, for example:
# ./checkall.ksh -f filename
# ./checkall.ksh -f /full/path/to/filename
-gThis option suppresses the output of any checks that complete successfully. This is specifically very useful, when running all scripts through the master check script checkall.ksh. Many of the check scripts will probably return a zero error code, and thus are usually not worth reviewing, because they complete successfully and no issue is identified. Using the -g option will make sure the checks that complete successfully will not end up in the output. By default, the output of all scripts is shown.
-GThis option suppresses the output of any checks that complete successfully - and check scripts that return a WARNING (returncode 2). Only check scripts that return an ERROR (returncode 1) will be displayed. The -G option supersedes the -g option. This option is useful if you only wish to review the most critical items identified on the system, and when you do not care about WARNING messages.
-hThis option will write an HTML file output instead of the default log file (-l) output. By generating HTML output, it is easier to see which check scripts return an error or warning, and which check scripts complete successfully. This is accomplished by using different colors for each script. A script that completes successfully (or returncode 0) will be listed as green. Check scripts that end with a WARNING (or return code 2) will be listed as yellow. Check scripts that end with an ERROR (or return code 1) will be listed as red.

Note: The order of the options determine which type of output is written. If you specify both the -l and -h option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -x (for XML output) options.

By default, AIX Health Check will write HTML output to file checkall_[hostname].html in the same folder where AIX Health Check is run from.

A sample of HTML output can be seen here.

To view the HTML reports that AIX Health Check generates, one of the following browsers is required:
  • Microsoft Internet Explorer 8 or higher
  • Mozilla Firefox
  • Google Chrome
-iThis is an old option, and its use is no longer supported. It was in use to generate an inventory section. If you wish to run any scripts of the inventory category, use option "-C inventory" instead now.
-lThis option will write a text log file output. This is the default output type if neither of the options -l, -c or -h is provided. By default, AIX Health Check will write a text log file to file name checkall_[hostname].log in the same folder wheere AIX Health Check is run from.

Note: The order of the options determine which type of output is written. If you specify both the -h and -l option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -x (for XML output) options.
-mThis option can be used to provide a comma-separated list of email addresses to send output to, for example:
# checkall.ksh -m email@domain.com,info@aixhealthcheck.com
Mail is sent through sendmail from AIX. Before you attempt to use this option, please make sure that it is possible to send email by running:
# cat /etc/motd | sendmail -v email@domain.com
Email will be sent in the type of output that you've specified. If you didn't specify any of the -c, -h or -l options, a text file output (by default, the -l option) will be sent through email. If you specified -c, a CSV style output will be sent through email. If you specified -h, an HTML type report will be sent through email. If you specified -x, a XML type report will be sent through email.

To view the HTML reports by email, only Microsoft Outlook 2003/2010 and Windows Live Mail are supported. Other email clients may or may not be able to view the HTML report generated by AIX Health Check properly.

Multiple email addresses may only be separated by a single comma.
-n The -n option will ensure that only output is sent by email if there are any check scripts that didn't complete successfully (returncode 0).

This is every useful to use in combination with the -m option. For example, if you have a number of crontab entries set up to monitor your systems, you can use the -n option to only alert you by email if there is indeed a check that generates an ERROR message (returncode 1). No email is sent if all of the check scripts complete successfully (returncode 0) or if only check scripts completed with a WARNING message (returncode 2).

For example, you could add the following to the root user's crontab, in order to alert you if there's an issue on the AIX system:
# Check every hour:  Free space in file systems, 
# paging space usage, memory utilization, 
# high CPU using processes, printer queues, 
# error report, zombie processes
0 * * * *   /ahc/checkall.ksh -gdhbnm email@address.com 
	    -s checkfreespaceinfs.ksh,checkpgspusage.ksh,
            checkmemoryutilization.ksh,checkhighcpu.ksh
5 * * * *   /ahc/checkall.ksh -gdhbnm email@address.com 
            -s checkenq.ksh,checkerrpt.ksh,checkzombies.ksh
# Check once a day: TSM backups
9 9 * * *   /ahc/checkall.ksh -gdhbnm email@address.com 
	    -s checktsmsched.ksh
# Check once a week: mksysb backups, logical volume/file 
# system structure
0 8 * * 1,5 /ahc/checkall.ksh -gdhbnm email@address.com 
            -s checkmksysb.ksh,checklvfscreate.ksh
A useful combination for the -n option, is to also use the -b option. This way, also any end results and copyright messages are eliminated, thus further reducing the emails sent out.
-sYou can use the -s option to provide a comma-separated list of check scripts to run. The default action of checkall.ksh is (without supplying the -s option) to run all the check scripts. With the -s option, you can specify which check scripts to run. For example:
# ./checkall.ksh -s checkbit.ksh,checkpgspusage.ksh
Multiple check scripts may only be separated by a single comma. Do not specify any folder names when specifying any check scripts to run. Only the name of a check script including the .ksh extension is allowed. For example: "-s /tmp/checkbit.ksh" will not work, but "-s checkbit.ksh" will work. AIX Health Check assumes to find all the check scripts in the same folder where checkall.ksh is run from. AIX Health Check will skip running any scripts it can't find in the same folder where checkall.ksh is run from.

The -s option overrules the -e option. Scripts specified with the -s option will be run, even if the same scripts were excluded using the -e option. The -s option also overrules the -E option. If specific scripts were excluded using -E option, and still included using the -s option, then the scripts will be run. Finally, the -s option also overrules the -C option. If certain scripts were specified using the -s option, then the -C option (for specifying categories of scripts) will be ignored.
-uThe -u option displays the usage information, and then exits checkall.ksh. Using the question mark (-?) instead of the -u option has the same effect.
-vThe -v option displays verbose output information on screen, while checkall.ksh is running.
-wWith the -w option you can specify the width of the output. By default, AIX Health Check will generate output up to 130 characters wide. You can specify any value with -w from 70 to 1024 characters:
# ./checkall.ksh -w 512
Specifying the width of the output has no effect if HTML output (-h) is selected.
-xThis will write a XML file output instead of the default log file (-l) output. By generating XML output, output of AIX Health Check can be loaded into various other systems/applications or can be viewed in tree-like form with the use of a XML reader.

Note: The order of the options determine which type of output is written. If you specify both the -l and -x option, the last option provided will determine the type of output written. This also applies to the -c (for CSV output) and -h (for HTML output) options.

By default, AIX Health Check will write HTML output to file checkall_[hostname].xml in the same folder where AIX Health Check is run from.

To view the XML reports that AIX Health Check produces, a XML reader is required, for example the free XML editor, FOXE, from First Object.

9. Return codes
Each check script ends with one of the following three exit/return codes:

0
The script completed successfully.
1
The script found an ERROR.
2
The script ended with a WARNING.

Errors are usually something that requires some form of remediation, and should be resolved, if possible. Warnings are just that, warnings. They may be ignored.

You can see the return code of a single check script by checking the $? value. For example:
# ./checkpgspsize.ksh
# echo $?
0
This particular script ended with return code 0, and thus ended successfully.

The following is an example of a check script that ends with an ERROR status:
# ./checkbosadtdebug.ksh
Fileset bos.adt.debug not installed.
# echo $?
1
As you can see, a script that ends with an ERROR (or a WARNING) status, will also output a message about the error found.

10. Running categories of scripts
AIX Health Check has a feature to run certain categories of scripts. Often you probably don't want to run all the check scripts that are included in AIX Health Check, but like to focus only on those features that are relevent for you at that time. For example, you may want to check all the security items of an AIX system, and not be bothered by anything else. Or sometimes, you may wish to generate only an inventory, by running only those scripts that provide inventory type information.

You can use the "-C" option available in AIX Health Check, to run a category of scripts. For example, to run all the security type scripts, run:
# ./checkall.ksh -C security
Likewise, if you wish to generate the inventory information of an AIX system, run:
# ./checkall.ksh -C inventory
You can also combine several categories of scripts by providing a comma-separated list of categories to be run. For example if you wish to run all the storage and security type scripts, you can run:
# ./checkall.ksh -C storage,security
You can choose from the following available categories:

backupRun all scripts that check the backup of the system, including TSM, EMC Avamar/Networker, OSSV Snapvault, Commvault Simpana, and Veritas Netbackup.
bootingRun all check scripts that check items of the boot sequence, including inittab, rc.d files, autorestart, and start scripts.
capacityRun all check scripts that check the capacity of the system, including storage, CPU, network and memory.
clusterRun all scripts that check cluster configuration and status, including PowerHA/HACMP and Veritas.
daemonsRun all scripts that check daemons and services on the system, including inetd.
datetimeRun all scripts that check date and time settings on the server, including ntp configuration.
debuggingRun all scripts that check dubug items, such as debug filesets, core dump settings, error logging and system dumps.
hardwareRun all scripts that check the physical hardware of the system, incluing adapters, devices, power supplies, firmware, and lpar configuration.
inventoryRun all scripts that generate inventory information and only display output of commands for inventory purposes.
loggingRun all check scripts for checking log files, alog and syslog configuration.
mailRun all script that check mail related items, such as the mail configuration and aliases.
memoryRun all memory scripts that check memory usage, vmo options, and memory cache.
monitoringRun all check scripts that check the monitoring of the system, including BMC Patrol, Ganglia, Marimba and Nimsoft.
networkRun all scripts that check network settings, DHCP, DNS, and NFS.
performanceRun all scripts that check the performance of the system, including those that check Active Memory Expansion.
powervmRun all scripts that check PowerVM configuration and settings, including dlpar, VIOS, and WPARs.
printingRun all scripts that check printing and printer queues, including Cups.
redundancyRun all scripts that check redundancy features and configuration on the server (does not include cluster items).
schedulingRun all check scripts that check the scheduling of jobs on the system, including at and cron.
securityRun all scripts that check permissions/owner of files, the ssh configuration, old files, user acounts and user limits.
softwareRun all scripts that check the software installed on the system, such as filesets (lslpp) and rpm packages, and additional software installed on the server.
storageRun all scripts that check the storage of the system, including SDD, SDDPCM, disks, and EMC Powerpath.

Please note that scripts may belong to multiple categories. For example, a script that checks if xmdaily is disabled in the root crontab, checkxmdaily.ksh, belongs to both the category "scheduling" and the category "monitoring", as the xmdaily entry in the root crontab controls running xmwlm/topasrec processes for monitoring purposes on the system.

Also keep in mind that by selecting one or more categories, that this determines how many scripts will be run. For example, the security category is a large category of scripts, that includes over 300 scripts to be run, while the datetime category is a much smaller category, that includes only a few scripts.

11. Automating the download of AIX Health Check
It is fairly easy to automate the download of the latest version of AIX Health Check to your AIX system. Doing so, removes the need for downloading AIX Health Check manually, and also removes the need to transfer the downloaded file onto your AIX systems manually.

10a. What do you need?
  • Install either curl or wget on your AIX system. These tools are not part of a regular AIX installation. You can download the RPM for these tools here:
    AIX Toolbox for Linux Applications
  • Your AIX system needs Internet access, and needs to be able to resolve "aixhealthcheck.com" through DNS.

    For example, test with a simple ping:
    # ping aixhealthcheck.com
  • You need a valid license for AIX Health Check, which provides you a login/email address and a password to access the AIX Health Check download page.
10b. Examples of using curl and wget to download AIX Health Check

Here's an example of how you can use curl to download the latest version of AIX Health Check (assuming your login/email address is "user@email.com" and your password is "mypassword"):
curl -d "emailaddress=user@email.com&password=mypassword" \
   --referer http://www.aixhealthcheck.com \
   http://www.aixhealthcheck.com/downloadauto.php \
   -o ahc_latest.tar
This command will download file ahc_latest.tar in your current folder. Please note that providing the email address, password and referer is required. Without it, the download will fail.

The same download can be accomplished with wget:
wget --post-data 'emailaddress=user@email.com&password=mypassword' \
   --referer=http://www.aixhealthcheck.com \
   http://www.aixhealthcheck.com/downloadauto.php \
   -O ahc_latest.tar
Once you have downloaded the ahc_latest.tar file, you will be able to un-tar the file and use AIX Health Check.

10c. Automating download with a script

You can automate downloading AIX Health Check easily with the use of a Korn Shell script. The script that is shown below downloads AIX Health Check with wget, will untar the downloaded file, run AIX Health Check, send an HTML style report through email, and clean up afterwards.
#!/usr/bin/ksh
 
# VARIABLES
emailaddress="user@email.com"
password="mypassword"
folder="/ahc"
 
# Test if folder already exists
if [ -d ${folder} ] ; then
        echo "Folder ${folder} already exists. Aborting."
        exit
fi
 
# Test if wget is executable
if [ ! -x /usr/bin/wget ] ; then
        echo "Wget does not exist or is not executable. Aborting."
        exit
fi
 
# Create a folder to store AIX Health Check
echo "Creating folder ${folder}..."
mkdir -p ${folder}
 
# Download the file
echo "Downloading AIX Health Check..."
wget --post-data "emailaddress=$emailaddress&password=$password" \
   --referer=http://www.aixhealthcheck.com \
   http://www.aixhealthcheck.com/downloadauto.php \
   -O ${folder}/ahc_latest.tar >/dev/null 2>&1
 
# Un-tar the downloaded file
if [ -s ${folder}/ahc_latest.tar ] ; then
        cd /ahc
        echo "Un-tar the downloaded file..."
        tar -xvf ${folder}/ahc_latest.tar >/dev/null 2>&1
        cd - >/dev/null 2>&1
        echo "Removing downloaded file..."
        rm ${folder}/ahc_latest.tar
 
        # Run AIX Health Check
        if [ -x ${folder}/checkall.ksh ] ; then
                echo "Running AIX Health Check..."
		${folder}/checkall.ksh -hdm user@email.com 
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
        else
                echo "Encountered an error with checkall.ksh"
                echo "Removing folder ${folder}..."
                rm -rf ${folder}
                exit
        fi
else
        echo "Error downloading AIX Health Check."
        echo "Removing folder ${folder}..."
        rm -rf ${folder}
        exit
fi
The script above can be used to download AIX Health Check to your AIX system in folder /ahc. Please make sure to provide the correct login/email address and password in the beginning of the script for variables emailaddress and password, and if you wish to specify a different location for storing AIX Health Check, you can change this in the script as well, by updating the folder variable.

The output of the script above, will look like this (assuming you have named the script "run-ahc.ksh"):
# ./run-ahc.ksh
Creating folder /ahc...
Downloading AIX Health Check...
Un-tar the downloaded file...
Removing downloaded file...
Running AIX Health Check...
Removing folder /ahc...

12. Enabling password-less access through SSH on PowerHA/HACMP nodes
When running AIX Health Check on PowerHA/HACMP nodes, it is important that password-less access for user root is available from one cluster node to another. If this is not set up correctly, some scripts may fail. Here are the steps to perform on each cluster node: First, make sure OpenSSH and OpenSSL is installed on the two clustered UNIX servers, serverA and serverB.

On each server, as user root, type:
# ssh-keygen -t dsa
This will prompt you for a secret passphrase. If this is your primary identity key, use an empty passphrase (which is not secure, but the easiest to work with). You will get two files called id_dsa and id_dsa.pub in your .ssh directory in the root user's home directory.

Copy the id_dsa.pub file to the other host's .ssh dir with the name authorized_keys. For example, on serverA:
# scp ~/.ssh/id_dsa.pub serverB:.ssh/authorized_keys2
Now serverB is ready to accept your SSH key from serverA.

For a test, type:
# ssh serverB
This should let you in without typing a password or passphrase.

13. Resolving email issues
When sending email from AIX Health Check's master script (checkall.ksh) using the -m option, you may run into issues with sending email.

Before you attempt to use the email feature of AIX Health Check, please make sure to verify that you can send email from your AIX system, by running:
# cat /etc/motd | sendmail -v email@domain.com
By using the -v option with the sendmail command, you will see verbose output of the email you're sending. If mail does not arrive at the designated mailbox, check the mail queue on your system to determine if mail has been sent or not:
# mailq
If mail is not being sent, this is usually do to two common issues:
  • DNS not properly configured
    For the mail function of your AIX system, it is important that your system is capable to resolve the fully qualified domain name of your system. Make sure that the short hostname, the fully qualified domain name, e.g. host.domain.com, and the IP address of your server can be resolved, by testing with the nslookup command.

    For example:
    # nslookup myserver
    Name:   myserver.domain.com
    Address: 10.43.242.82
    
    # nslookup myserver.domain.com
    Name:   myserver.domain.com
    Address: 10.43.242.82
    
    nslookup 10.43.242.82
    82.242.43.10.in-addr.arpa name = myserver.domain.com.
    
    If this does not work in the example above, please check with your network administrator to correct the DNS settings for your server.
  • Missing SMTP server entry in /etc/sendmail.cf file
    Mail can also not be sent properly, if the name of the SMTP server is missing in /etc/sendmail.cf file. Look for an entry in this file that starts with "DS", for example:
    # "Smart" relay host (may be null)
    DSrelay.domain.com
    
    If no entry is present, discuss with your email administrator which SMTP relay host can be used to relay email from the AIX system to your desired mailbox.

14. Resolving issues found by AIX Health Check
When reviewing reports generated by AIX Health Check, you most likely will see issues that need to be remediated.

Please take note of the warning in the report:

"Any individual implementing changes should completely understand the change and deem each change appropriate before it is applied to the system. As a standard rule, please take into consideration the impact on other components before implementing the change. Also, we encourage all to conduct a peer review of all changes before implementation. Most importantly, if the effect of a change is not fully understood, do not implement that change until a satisfactory explanation can be given as to what the effects of the change are. We recommend implementation of one change at a time. The application of many changes all at once will increase the likelihood of confusion, if issues arise."

It is important to follow change procedures properly while resolving any issues found by AIX Health Check.