A Few Linux Commands to Keep you Sane

Some simple terminal commands you can run on Linux to help keep you sane.

Every program we run on the Linux or Unix command line automatically has three data streams connected to it.

  • STDIN (0) - Standard input (data fed into the program)
  • STDOUT (1) - Standard output (data printed by the program, defaults to the terminal)
  • STDERR (2) - Standard error (for error messages, also defaults to the terminal)

Here are a few ways to manipulate the three streams to make commands more functional or get better-formatted data.

Output Ansible to the Screen and to a Log File.
$> unbuffer ansible-playbook playbook.yml --become | tee ansibleplaybook.log

When you use a | (pipe) in a Linux command to redirect the STDOUT of the command from on the left side of the | to the STDIN of the command on the right side of the pipe, the system will start to buffer the output from the command on the left into memory. This will make the second program wait until the first has completed before it receives any information. Normally, this is not a bad thing as most commands complete relatively quickly. In most cases you want the second command to act on the full output of the first command.

What if you have a long-running command with a lot of output? What if you want to see if that command is running correctly? What if you don’t want the second command to have to wait for the first command to finish?

This is where the unbuffer command comes into play. Unbuffer connects to the ansible-playbook command via a pseudo-terminal. This makes the system treat both commands as an interactive process, even with a |, and allows the output from ansible-playbook to be written to the STDOUT without the buffering.

The tee command takes input from STDIN and writes it to STDOUT and to a file. Since STDOUT defaults to the terminal screen, you will see the log data as it is generated.

I use this for long-running commands that print out a lot of data to STOUT. This allows me to verify that the command is still running. This also allows me to keep a log of all data that is generated by the first command in a file for later review.

Find the Size of Sub-directories and Files

$> du -k | sort -n | perl -ne 'if ( /^(\d+)\s+(.*$)/){$l=log($1+.1);$m=int($l/log(1024)); printf ("%6.1f\t%s\t%25s %s\n",($1/(2**(10*$m))),(("K","M","G","T","P")[$m]),"*"x (1.5*$l),$2);}'
 2.6 M *********** ./.git/modules/roles/submodules
 3.3 M ************ ./.git/modules/roles
 4.7 M ************ ./.git/modules
 5.7 M ************ ./.git
 8.3 M ************* .

While the above command looks gnarly, it is merely a du command whose output will get sorted and fed to a Perl script to make it look pretty.

The du command is used to estimate the disk usage of directories from your current location. I add the -k option to scale the output size to 1.05 MB (the default) chunks.

The data from du is passed to STDOUT piped (|) into STDIN of sort. The sort command sorts lines of text. Since the first part of each line, we are sending to sort has the size of the directory I use the -n option to sort the data numerically.

The Perl script takes the data from sort and formats each line to print out the size of the directory, a number of stars to give a visual representation of the size and the directory name.

A simpler command is as follows but lacks the nice format of the first.

$> du -a | sort -n | tail -5
80480 ./venv/lib/python2.7/site-packages
81252 ./venv/lib/python2.7
81284 ./venv/lib
81632 ./venv
136360 .

The -a is for all and will include files in with your directories.

Listing Open Ports (TCP or UDP)

Another big task is listing active ports on the servers. A great tool for this is netstat.

$> netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN 2609/smbd 
tcp 0 0 0.0.0.0:9101 0.0.0.0:* LISTEN 1880/bareos-dir 
tcp 0 0 0.0.0.0:9102 0.0.0.0:* LISTEN 1657/bareos-fd 
tcp 0 0 0.0.0.0:9103 0.0.0.0:* LISTEN 1658/bareos-sd 
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 1505/dnsmasq 
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 22772/cupsd 
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1625/postgres 
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 2551/master 
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN 2609/smbd 
tcp6 0 0 :::139 :::* LISTEN 2609/smbd 
tcp6 0 0 ::1:53 :::* LISTEN 1505/dnsmasq 
tcp6 0 0 ::1:631 :::* LISTEN 22772/cupsd

Netstat will allow you to input the option flags individually or as a group.

netstat -plnt = netstat -p -l -n -t

Using the -p will show you the PID and the name of the program on the output.

The -l will display only sockets that are listing on your server.

The -n shows the numerical address of IP the server is listening for instead of host names.

You can tell netstat to only look for TCP ports with the use of the -t flag. If you want UDP ports I substitute -u for the -t. If you like to get both add the -u to the command. Many times admins like to get creative and try to arrange the flag options to look like words so they are easier to remember.

$> netstat -plunt
$> netstat -peanut

Help on Commands

If an internet connection is not available to you, the command man, short for manual, is an invaluable tool. Use this command to get instructions on how to use a command and what options that command has in your Linux or Unix distribution.

$> man ls

If you are really feeling adventurous do a man on man.

$> man man