Topics: AIX, Monitoring, Networking, Red Hat, Security, System Administration

Determining type of system remotely

If you run into a system that you can't access, but is available on the network, and have no idea what type of system that is, then there are few tricks you can use to determine the type of system remotely.

The first one, is by looking at the TTL (Time To Live), when doing a ping to the system's IP address. For example, a ping to an AIX system may look like this:

# ping 10.11.12.82
PING 10.11.12.82 (10.11.12.82) 56(84) bytes of data.
64 bytes from 10.11.12.82 (10.11.12.82): icmp_seq=1 ttl=253 time=0.394 ms
...
TTL (Time To Live) is a timer value included in packets sent over networks that tells the recipient how long to hold or use the packet before discarding and expiring the data (packet). TTL values are different for different Operating Systems. So, you can determine the OS based on the TTL value. A detailed list of operating systems and their TTL values can be found here. Basically, a UNIX/Linux system has a TTL of 64. Windows uses 128, and AIX/Solaris uses 254.

Now, in the example above, you can see "ttl=253". It's still an AIX system, but there's most likely a router in between, decreasing the TTL with one.

Another good method is by using nmap. The nmap utility has a -O option that allows for OS detection:
# nmap -O -v 10.11.12.82 | grep OS
Initiating OS detection (try #1) against 10.11.12.82 (10.11.12.82)
OS details: IBM AIX 5.3
OS detection performed.
Okay, so it isn't a perfect method either. We ran the nmap command above against an AIX 7.1 system, and it came back as AIX 5.3 instead. And sometimes, you'll have to run nmap a couple of times, before it successfully discovers the OS type. But still, we now know it's an AIX system behind that IP.

Another option you may use, is to query SNMP information. If the device is SNMP enabled (it is running a SNMP daemon and it allows you to query SNMP information), then you may be able to run a command like this:
# snmpinfo -h 10.11.12.82 -m get -v sysDescr.0
sysDescr.0 = "IBM PowerPC CHRP Computer
Machine Type: 0x0800004c Processor id: 0000962CG400
Base Operating System Runtime AIX version: 06.01.0008.0015
TCP/IP Client Support  version: 06.01.0008.0015"
By the way, the example for SNMP above is exactly why AIX Health Check generally recommends to disable SNMP, or at least to dis-allow providing such system information trough SNMP by updating the /etc/snmpdv3.conf file appropriately, because this information can be really useful to hackers. On the other hand, your organization may use monitoring that relies of SNMP, in which case it needs to be enabled. But then you stil have the opportunity of changing the SNMP community name to something else (the default is "public"), which also limits the remote information gathering possibilities.

Topics: AIX, Networking, System Administration

Using tcpdump to discover network information

As an AIX admin, you may not always know what switches a certain server is connected to. If you have Cisco switches, here's an interesting method to identify the switch your server is connected to.

First, run ifconfig to look up the interfaces that are in use:

# ifconfig -a | grep en | grep UP | cut -f1 -d:
en0
en4
en8
Okay, so on this system, you have interfaces en0, en4 and en8 active. So, if you want to determine the switch en4 is connected to, run this command:
#  tcpdump -nn -v -i en4 -s 1500 -c 1 'ether[20:2] == 0x2000'
tcpdump: listening on en4, link-type 1, capture size 1500 bytes
After a while, it will display the following information:
11:40:14.176810 CDP v2, ttl: 180s, checksum: 692 (unverified)
   Device-ID (0x01), length: 22 bytes: 'switch1.host.com'
   Version String (0x05), length: 263 bytes:
   Cisco IOS Software, Catalyst 4500 L3 Switch Software 
      (cat4500e-IPBASEK9-M), Version 12.2(52)XO, RELEASE SOFTWARE
   Technical Support: http://www.cisco.com/techsupport
   Copyright (c) 1986-2009 by Cisco Systems, Inc.
   Compiled Sun 17-May-09 18:51 by prod_rel_team
   Platform (0x06), length: 16 bytes: 'cisco WS-C4506-E'
   Address (0x02), length: 13 bytes: IPv4 (1) 111.22.33.44
   Port-ID (0x03), length: 18 bytes: 'GigabitEthernet2/7'
   Capability (0x04), length: 4 bytes: (0x00000029): 
      Router, L2 Switch, IGMP snooping
   VTP Management Domain (0x09), length: 2 bytes: ''''
   Native VLAN ID (0x0a), length: 2 bytes: 970
   Duplex (0x0b), length: 1 byte: full
   Management Addresses (0x16), length: 13 bytes: IPv4 (1) 
      111.22.33.44
   unknown field type (0x1a), length: 12 bytes:
      0x0000:  0000 0001 0000 0000 ffff ffff
47 packets received by filter
0 packets dropped by kernel
This will help you determine, that en4 is connected to a network switch called 'switch1.host.com', with IP address '111.22.33.44', and that it is connected to port 'GigabitEthernet2/7' (most likely port 7 on blade 2 of this switch).

If you're running the same command on an Etherchannelled interface, keep in mind that it will only display the information of the active interface in the Etherchannel configuration. You may have to fail over the Etherchannel to a backup adapter, to determine the switch information for the backup adapter in the Etherchannel configuration.

If your LPAR has virtual Ethernet adapters, this will not work (the command will just hang). Instead, run the command on the VIOS instead.

Also note that you may need to run the command a couple of times, for tcpdump to discover the necessary information.

Another interesting way to use tcpdump is to discover what VLAN an network interface is connected to. For example, if you have 2 interfaces on an AIX system, and you would want to configure them in an Etherchannel, or you would want to use one of them as a production interface, and another as a standby interface. In that case, it is important to know that both interfaces are within the same VLAN. Obviously, you can ask your network team to validate, but it is also good to be able to validate on the host side. Also, you can just configure an IP address on it, and see if it will work. But for production systems, that may not always be possible.

The trick basically is, to run tcpdump on an interface, and check what network traffic can be discovered. For example, if you have 2 network interfaces, like these:
# netstat -ni | grep en[0,1]
en0 1500 link#2    0.21.5e.c0.d0.12 1426632806  0 86513680  0  0
en0 1500 10.27.18  10.27.18.64      1426632806  0 86513680  0  0
en1 1500 link#3    0.21.5e.c0.d0.13   20198022  0  7426576  0  0
en1 1500 10.27.130 10.27.130.10       20198022  0  7426576  0  0
In this case, interface en0 uses IP address 10.27.18.64, and is within the 10.27.18.x subnet. Interface en1 uses IP address 10.27.130.10, and is within the 10.27.130.x subnet (assuming both interfaces use a subnet mask of 255.255.255.0).

Now, if en0 is a production interface, and you would like to confirm that en1, the standby interface, can be used to fail over the production interface to, then you need to know that both of the interfaces are within the same VLAN. To determine that, for en1, run tcpdump, and check if any network traffic in the 10.27.18 subnet (used by en0) can be seen (press CTRL-C after seeing any such network traffic, to cancel the tcpdump command):
# tcpdump -i en1 -qn net 10.27.18
tcpdump: verbose output suppressed, 
use -v or -vv for full protocol decode
listening on en1, link-type 1, capture size 96 bytes
07:27:25.842887 ARP, Request who-has 10.27.18.136
   (ff:ff:ff:ff:ff:ff) tell 10.27.18.2, length 46
07:27:25.846134 ARP, Request who-has 10.27.18.135 
   (ff:ff:ff:ff:ff:ff) tell 10.27.18.2, length 46
07:27:25.917068 IP 10.27.18.2.1985 > 224.0.0.2.1985: UDP, length 20
07:27:25.931376 IP 10.27.18.3.1985 > 224.0.0.2.1985: UDP, length 20
^C
24 packets received by filter
0 packets dropped by kernel
After seeing this, you know for sure that on interface en1, even though it has an IP address in subnet 10.27.130.x, network traffic for 10.27.18.x subnet can be seen, and thus that failing over the production interface IP address from en0 to en1 should work just fine.

Topics: AIX, Networking, System Administration

Using iptrace

The iptrace command can be very useful to find out what network traffic flows to and from an AIX system.

You can use any combination of these options, but you do not need to use them all:

  • -a   Do NOT print out ARP packets.
  • -s [source IP]   Limit trace to source/client IP address, if known.
  • -d [destination IP]   Limit trace to destination IP, if known.
  • -b   Capture bidirectional network traffic (send and receive packets).
  • -p [port]   Specify the port to be traced.
  • -i [interface]   Only trace for network traffic on a specific interface.
Example:

Run iptrace on AIX interface en1 to capture port 80 traffic to file trace.out from a single client IP to a server IP:
# iptrace -a -i en1 -s clientip -b -d serverip -p 80 trace.out
This trace will capture both directions of the port 80 traffic on interface en1 between the clientip and serverip and sends this to the raw file of trace.out.

To stop the trace:
# ps -ef|grep iptrace
# kill 
The ipreport command can be used to transform the trace file generated by iptrace to human readable format:
# ipreport trace.out > trace.report

Topics: AIX, Networking, System Administration

IP alias

To configure IP aliases on AIX:

Use the ifconfig command to create an IP alias. To have the alias created when the system starts, add the ifconfig command to the /etc/rc.net script.

The following example creates an alias on the en1 network interface. The alias must be defined on the same subnet as the network interface.

# ifconfig en1 alias 9.37.207.29 netmask 255.255.255.0 up
The following example deletes the alias:
# ifconfig en1 delete 9.37.207.29

Topics: Networking, Red Hat

Howto Red Hat Enterprise Linux 5 configuring the network

Red Hat LogoRed hat Linux provides following tools to make changes to Network configuration such as add new card, assign IP address, change DNS server, etcetera:

  • GUI tool (X windows required) - system-config-network
  • Command line text based GUI tool (No X windows required) - system-config-network-tui
  • Edit configuration files directly, stored in /etc/sysconfig/network-scripts directory
The following instructions are compatible with CentOS, Fedora Core and Red Hat Enterprise Linux 3, 4 and 5.

Editing the configuration files stored in /etc/sysconfig/network-scripts:

First change directory to /etc/sysconfig/network-scripts/:
# cd /etc/sysconfig/network-scripts/
You need to edit / create files as follows:
  • /etc/sysconfig/network-scripts/ifcfg-eth0 : First Ethernet card configuration file
  • /etc/sysconfig/network-scripts/ifcfg-eth1 : Second Ethernet card configuration file
To edit/create the first NIC file, type the following command:
# vi ifcfg-eth0
Append/modify as follows:
# Intel Corporation 82573E Gigabit Ethernet Controller (Copper)
DEVICE=eth0
BOOTPROTO=static
DHCPCLASS=
HWADDR=00:30:48:56:A6:2E
IPADDR=10.251.17.204
NETMASK=255.255.255.0
ONBOOT=yes
Save and close the file. Define the default gateway (router IP) and hostname in /etc/sysconfig/network file:
# vi /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=host.domain.com
GATEWAY=10.251.17.1
Save and close the file. Restart networking:
# /etc/init.d/network restart
Make sure you have correct DNS server defined in /etc/resolv.conf file. Try to ping the gateway, and other hosts on your network. Also check if you can resolv host names:
# nslookup host.domain.com
And verify if the NTP servers are correct in /etc/ntp.conf, and if you can connect to the time server, by running the ntpdate command against one of the NTP servers:
# ntpdate 10.20.30.40
This should synchronize system time with time server 10.20.30.40.

Topics: AIX, Networking, System Administration

Map a socket to a process

Let's say you want to know what process is tying up port 25000:

# netstat -aAn | grep 25000
f100060020cf1398  tcp4  0  0  *.25000  *.*  LISTEN
f10006000d490c08  stream  0  0  f1df487f8  0  0  0  /tmp/.sapicm25000
So, now let's see what the process is:
# rmsock f100060020cf1398 tcpcb
The socket 0x20cf1008 is being held by proccess 1806748 (icman).
If you have lsof installed, you can get the same result with the lsof command:
# lsof -i :[PORT]
Example:
# lsof -i :5710
COMMAND     PID   USER   FD   TYPE     DEVICE  SIZE/OFF NODE NAME
oracle  2638066 oracle   18u  IPv4 0xf1b3f398 0t1716253  TCP host:5710

Topics: Hardware, Networking

Integrated Virtual Ethernet adapter

The "Integrated Virtual Ethernet" or IVE adapter is an adapter directly on the GX+ bus, and thus up to 3 times faster dan a regular PCI card. You can order Power6 frames with different kinds of IVE adapters, up to 10GB ports.

The IVE adapter acts as a layer-2 switch. You can create port groups. In each port group up to 16 logical ports can be defined. Every port group requires at least 1 physical port (but 2 is also possible). Each logical port can have a MAC address assigned. These MAC addresses are located in the VPD chip of the IVE. When you replace an IVE adapters, LPARS will get new new MAC addresses.

Each LPAR can only use 1 logical port per physical port. Different LPARs that use logical ports from the same port group can communicate without any external hardware needed, and thus communicate very fast.

The IVE is not hot-swappable. It can and may only be replaced by certified IBM service personnel.

First you need to configure an HAE adapter; not in promiscues mode, because that is meant to be used if you wish to assign a physical port dedicated to an LPAR. After that, you need to assign a LHAE (logical host ethernet adapter) to an LPAR. The HAE needs to be configured, and the frame needs to be restarted, in order to function correctly (because of the setting of multi-core scaling on the HAE itself).

So, to conclude: You can assign physical ports of the IVE adapter to separate LPARS (promiscues mode). If you have an IVE with two ports, up to two LPARS can use these ports. But you can also configure it as an HAE and have up to 16 LPARS per physical port in a port group using the same interface (10Gb ports are recommended). There are different kinds of IVE adapters; some allow to create more port groups and thus more network connectivity. The IVE is a method of virtualizing ethernet without the need for VIOS.

Topics: AIX, Networking, System Administration

SCP Stalls

When you encounter an issue where ssh through a firewall works perfectly, but when doing scp of large files (for example mksysb images) the scp connection stalls, then there's a solution to this problem: Add "-l 8192" to the scp command.

The reason for scp to stall, is because scp greedily grabs as much bandwith of the network as possible when it transfers files, any delay caused by the network switch of the firewall can easily make the TCP connection stalled.

Adding the option "-l 8192" limits the scp session bandwith up to 8192 Kbit/second, which seems to work safe and fast enough (up to 1 MB/second):

# scp -l 8192 SOURCE DESTINATION

Topics: AIX, Networking, PowerHA / HACMP

Using an alternative MAC address

HACMP is capable of using an alternative MAC address in combination with its service address. So, how do you set this MAC address without HACMP, just using the command line? (Could come in handy, in case you wish to configure the service address on a system, without having to start HACMP).

# ifconfig enX down
# ifconfig enX detach
# chdev -l entX -a use_alt_addr=yes
# chdev -l entX -a alt_addr=0x00xxxxxxxxxx
# ifconfig enX xxx.xxx.xxx.xxx
# ifconfig enX up
And if you wish to remove it again:
# ifconfig enX down
# ifconfig enX detach
# chdev -l entX -a use_alt_addr=no
# chdev -l entX -a alt_addr=0x00000000000

Topics: AIX, Networking, ODM

Delete multiple default gateways

First, obtain how many gateways there are:

# odmget -q "attribute=route" CuAt

CuAt:
        name = "inet0"
        attribute = "route"
        value = "net,-hopcount,0,,0,192.168.0.1"
        type = "R"
        generic = "DU"
        rep = "s"
        nls_index = 0

CuAt:
        name = "inet0"
        attribute = "route"
        value = "net,-hopcount,0,,0,192.168.0.2"
        type = "R"
        generic = "DU"
        rep = "s"
        nls_index = 0
If there are more than one, you need to remove the excess route:
# chdev -l inet0 -a delroute="net,-hopcount,0,,0,192.168.0.2"
Method error (/usr/lib/methods/chginet):
        0514-068 Cause not known.
0821-279 writing to routing socket: The process does not exist.
route: not in table or multiple matches
0821-207 chginet: Cannot add route record to CuAt.
Then verify again:
# odmget -q "attribute=route" CuAt

CuAt:
        name = "inet0"
        attribute = "route"
        value = "net,-hopcount,0,,0,192.168.0.1"
        type = "R"
        generic = "DU"
        rep = "s"
        nls_index = 0

Number of results found for topic Networking: 17.
Displaying results: 1 - 10.