Secrets of an AIX Administrator, Part 5: What to Do After the First Server Reboot

If you’ve been an AIX systems administrator for a while, you know that when you've finished building a server and installing an OS, you haven't really finished building the server. Developers or application owners might think that because a system is up and pingable they can go ahead and load all their stuff onto it, but there’s still a lot of other work that must be done to make the system usable. In this fifth and final article in my "Secrets of an AIX Administrator" series, we’ll walk through the steps admins should take immediately after building an AIX server from scratch for the first time—the tasks you need to perform to complete the server-building process.

Step 1: Lock Down Root

The very first step you should take after your server reboots following the installation of AIX is to change the root file system password. This is when the server is most vulnerable to problems, and the key to the kingdom must be secured.

Step 2: Mirror the OS

If you’re using internal drives, I recommend that at this time you mirror the root volume group (rootvg) because there will be next to no I/O going on with the server. Although mirroring should typically fall later on the priority list, it’s worth your while to take 20 minutes now to have the system mirror the rootvg, instead of later on, when you'll be contending against users logging in and doing their initial configuration.

Step 3: Patch the Server

Similar to mirroring the OS, if you patch your server with the proper technology level (TL) and service pack (SP) at this point in the process, you’ll also avoid the hassle later on of trying to find a window of time in which to reboot the server. Patching the server now provides the additional benefit of letting you establish an expectation for the OS deployment, which will prevent users from complaining about future patching interfering with their installations and software.

Step 4: Set the Date and Time Zone

Although your server will likely have the correct time by this point, check the TZ variable in the /etc/environment file. Make sure that the system's time zone is in the correct locale, or set the time zone if your project requires doing so. You want to avoid the painful future scenario of discovering that a database or web page has been logging time in GMT when the system should have been in PST, requiring records to be backlogged or skipped forward by hours to correct the discrepancy. Also, by this point in the process, you shouldn’t have to reboot your server from now on for any further initial OS work.

Step 5: Disable Unnecessary Network Services; Install and Use SSH

The /etc/inetd.conf file is chock-full of various network services, from telnet and ftp to rsh and fingerd. Pretty much every service running either poses a security risk in today's world or is unnecessary. Thus, you should comment out the entire file and refresh the inetd daemon and instead use more secure communication protocols such as Secure Shell (SSH) for your communications to and from the server.

Step 6: Set Up IPs

Now that your system is secure from both a login and network perspective, this is the time to set up the IP addresses that you’ll need for communicating with the world.

Step 7: Grow and Develop rootvg File Systems

Depending on physical partition sizes, when an AIX server is built from scratch, the process will typically create the file systems on the server with the minimum amount of necessary space possible. This space limitation will become a hazard as soon as the server is put into use, so as a starting point, I recommend growing the file systems to at least the sizes listed in Table 1. 

I’ll also do two other things at this time. First, I’ll break out /var/tmp into its own file system, sized at about 1GB. I’ll do this because many applications will use this space as a temporary area for files, and separating /var/tmp in this way will offset some of the risk of a user maxing out /var by doing something like using vi on a mammoth-sized file. The second thing I'll do is grow the paging space and dump devices to sizes that will work for application owners and enable the capture the data from any crashes that might happen.

Step 8: Define User IDs and Groups

Usually, I would recommend hooking users up with access (including sudo) as one of the last things to do on a server, because the users could prematurely hop onto the box and start doing work before everything is completed. But there’s one good reason I put this action here in the list of things to do: Defining user IDs makes it easier to perform the next step.


Step 9: Establish External Volume Groups

Rather than coming back around after disk space has been appropriated, logical volumes are defined, and file systems have been mounted to set ownership and permissions on all applicable directories and files, I typically prefer that user IDs (or at least the local application-specific user IDs) already exist, to make establishing external volume groups a job that can be done in one fell swoop.

Step 10: Create a mksysb image

If you haven't had a chance to do so, create a mksysb image of the server and store it away. There’s nothing more frustrating than having to redo all your work, and a mksysb image will save you time and energy in case you have to go to backups for a restore. And speaking of mksysb images…

Step 11: Create a "Golden" Image Using NIM

Ideally, you’ll only ever have to set up a server from scratch once. After you do this, the mksysb image that you created can be used as a "gold standard" from which all future AIX builds can be done, facilitated by Network Installation Manager (NIM). This golden image can be rapidly distributed to new servers, which reduces your time investment in creating AIX builds. Plus, if there are any components such as tunables, additional user IDs, or system configurations that you need to mass-distribute in the future, it's much easier to do so using a single administrative source rather than having to go out to each and every server individually.

Shorten Your AIX Learning Curve

Becoming a skilled AIX systems admin is no easy task. It will take years to develop and hone your skills and to become familiar with your servers and customers. Most professional admins I know will agree that it typically takes at least six months to become proficient with an individual component of new technology, such as when LPARs and Virtual I/O Servers came on the scene, and at least nine months of understanding your employer's business before your learning curve takes you from being dangerous to knowing the basics of how their business works.

In this five-article series, I’ve shared a number of tips and tricks I've learned along the way—sometimes painfully—that have helped me whenever I go into new environments and lay hands on keys for the first time in different industries. Integrating these ideas and concepts into your own skill set will better your chances of avoiding mistakes, improve your work, and help you develop as an AIX systems admin.

Courtesy Christian Pruett &  POWER IT Pro.

Secrets of an AIX Administrator, Part 4: Networking

In AIX, I’ve found that networking is probably one of the most complicated subjects around. Some components of networking are addressed by making changes to flat files, while others are handled through the Object Data Manager (ODM). There are typically at least three commands or ways of doing something, from adding a route to turning off an interface. And what seems like a subtle or innocuous action can wind up causing headaches on the server in due time—for example, a reboot that brings out duplicate default routes.

I have extensive experience in both setting up and messing up networking on AIX servers worldwide. Along the way, I’ve taken copious notes on how to manage networking, so that others will hopefully avoid the same plights I encountered. The following is a condensed version of the knowledge I've gleaned about AIX networking through hard-won experience.

Understand Physical and Logical Relationships

Although most forms of UNIX and Linux will have only one device under the /dev directory to describe each networking interface, AIX is different in that it will create both a device for the physical interface (e.g., ent0) and a device for the logical interface (e.g., en0). Each device serves a unique purpose; for example, media speeds are set on the physical interface, whereas IP addresses are defined on the logical interface.

Knowing this about the physical and logical devices, your one key take-away is that if you need to do something to the physical interface, you’ll need to shut down, disable, or remove the logical interface before AIX will accept any commands to manipulate the physical interface.

Establish EtherChannel Devices First

Before you define any IPs on your servers, to create redundancy and get higher performance, I recommend setting up devices that use Cisco EtherChannel technology wherever possible. Much like bonding in Linux or IP network multipathing (IPMP) in Solaris, EtherChannel lets you gather multiple physical interfaces into one connection. The interfaces can be aggregated so that traffic goes down both pipes simultaneously, or they can work in an active-passive mode conditional upon failover. The "smitty etherchannel" fastpath is the easiest way I’ve found to define EtherChannel devices.

One thing worth noting is that the EtherChannel device, although logical in nature, still has a physical and a logical component. For example, if you choose to use ent0 and ent1 to create an EtherChannel adapter, the physical device will be created as ent2 with a logical device of en2. The EtherChannel adapter behaves almost identically to any other standard network adapter as far as setting network tunable parameters or establishing IP aliases, but the physical interface will show some additional information when viewed using the lsdev or lsattr command.

Use SMIT for Your Initial IPs

I’ve always subscribed to the philosophy that the proficiency of an AIX administrator can generally be determined by how often or seldom the admin relies upon the System Management Interface Tool (SMIT) to do his or her work. But there are a few tasks where, rather than memorize long commands with dozens of flags, I will turn to SMIT to accomplish my work. Setting up initial IP addresses is one of those tasks.

The "smitty tcpip" fastpath is the quickest route to navigating to the menus for setting up everything network related. From the SMIT Minimum Configuration & Startup screen, define your first IP address and default gateway through the menu. Then, as you need to add IPs to other interfaces, use the "smitty chinet" fastpath for those other devices. Using the Minimum Configuration & Startup screen to define additional interfaces could adversely affect the routing on the server.

Use the ODM Wisely

Most networking information in AIX is stored within the ODM. Only a few flat files that pertain to networking can be modified manually, such as /etc/hosts or the /etc/rc.tcpip and /etc/rc.nfs files. This can make it difficult to manage, change, or remove items like network routing from the server quickly and easily.

Aside from the initial plumbing-up of IPs on interfaces, the one other thing I’ll do through the ODM is to assign any IP aliases onto devices. I do so using the following command:

chdev -l enX -a alias4=$IP,$NETMASK

This way, following a reboot, all IPs on that specific adapter will come up automatically. However, beyond this point, I stick with modifying other flat files.

Case in point: I’ve found the task of adding additional routes to a server by means of the ODM to be a beast. I've had to hunt through obscure corners of the CuAt database when trying to remove or adjust a single route. Instead of trying to jump through hoops in adding or removing routes through the ODM, I typically modify the /etc/rc.net file under "Part II - Traditional Configuration" and add all my routes to this area using the route command. This way, I can delete active routes off the server using a route delete command and strip them from the file without having to do a reboot or worrying about the ODM.

Smoother Networking

The tips I've provided here should help make your AIX networking routine go a little more smoothly. In my next article, I'll pull together the information I've provided in this article and the other "Secrets of an AIX Administrator" articles (see the Learning Path for an articles list and links), and I’ll conclude this series by giving you some tips on what to do following your first reboot after building a new server.

Courtesy Christian Pruett &  POWER IT Pro.