The work of operations is often helped by using a combination of tools. The tools you choose to use will depend heavily on your environment and work style. Over time you may even change the tools you use.
This is a list of commonly used tools, we recommend trying as many of them as you have time for. Finding the right tool is often like finding a good pair of gloves - they have to fit just right!
Rather than stick with the default terminal, many sysadmins on OS X install iTerm2. iTerm2’s greater flexibility and larger feature set have made it a de facto standard among technical Mac users. Like most modern terminals, it supports vertical and horizontal splits, tabs, profiles, etc. In addition, it stores the rendered buffer for a period of time which serves as a way to view previous things that might have been lost due to editor crashes or the like.
Many advanced Linux sysadmins prefer tiling window managers such as i3, Awesome, xmonad, herbstluftwm, and others. However, these window managers can be daunting to a new user when compared to a more Windows-like desktop environment like Unity, KDE, or XFCE. A popular compromise is to use a tiling terminal emulator, such as Terminator, within a regular window manager.
Terminator has some features which are convenient for performing similar commands in several places at once, such as grouping terminals together and then broadcasting your keystrokes to all terminals in the group.
If your operating system has a good terminal emulator available (e.g., Linux, BSD, OSX), then the ssh command line utility is by far the best tool to use for SSH interactions.
If you use Windows as your primary work space, PuTTY is possibly one of the best SSH clients available.
Connectbot is a free app available in the Google Play store, and offers most of the functionality you would need from an SSH client.
A universal app available on the App Store, iSSH supports most–if not all–of the features a demanding power user needs.
Far more than just a secure way to access a shell on a remote machine, ssh has evolved to support a large collection of ways to encapsulate traffic over the encrypted channel. Each of these is useful in different situations. This section will present some common problems, and will present how to solve the problems with ssh. It is important to note that all port forwarding and proxying occurs over the secure ssh channel. Besides working around firewall rules, the tunnels also provided a layer of security where there may not otherwise be one.
For this usecase, consider a loadbalancer–lb-foo-1–with a Web management interface listening on port 9090. This interface is only routable to a LAN private to the datacenter–10.10.10.0/24. Its IP address is 10.10.10.20. There is a machine on the 10.10.10.0/24 network with ssh access–jumphost-foo-1, accessible via DNS with the LAN IP 10.10.10.19. Therefore, one way to access lb-foo-1 is via bouncing through jumphost-foo-1. OpenSSH supplies the -L option to bind a local port to a port opened on the other side of the tunnel. On the remote end, ssh opens a connection to the host and port specified to the -L option. An example for this usecase:
ssh -L 9090:lb-foo-1:9090 jumphost-foo-1
This will open a connection from jumphost-foo-1 to lb-foo-1 on port 9090, and will bind the local port 9090 to a tunnel through the connection to that port. All traffic sent to the local port 9090 will reach lb-foo-1:9090. It is important to note that the host given to -L uses the perspective of the machine ssh connects to directly. This is important to remember for environments using hosts files or private / split-horizon DNS. In this usecase, an invocation like the following would work as well:
ssh -L 9090:10.10.10.20:9090 jumphost-foo-1
In this case, jumphost-foo-1 would try to connect to 10.10.10.20 directly, skipping the DNS lookup.
For this usecase, consider the rest of the machines in the 10.10.10.0/24 network. There are many ways to access them, including VPNs of various sorts. In addition to creating tunnels to specific ports, OpenSSH can also create a SOCKS proxy. OpenSSH provides the -D option for this. As an example, to bounce from jumphost-1 to the rest of the network:
ssh -D 9999 jumphost-foo-1
Once the SOCKS proxy is in place, the operating system or applications needing access to the proxy need to be configured to use it. This will vary per-application. Some operating systems (OS X) provide system-wide proxy configuration.
In some situations it may be necessary to forward ports from the remote machine to the local one. While not as common as other port forwarding techniques, it is often the only option available in the situations where it sees use. For this feature, OpenSSH listens to a port on the remote machine and sends the traffic to the local host and port. The syntax for the -R option is the same as for the -L option described above. As an example, to have OpenSSH tunnel the remote port 9090 to the local port 8000 on host workstation:
ssh -R 9090:workstation:8000 jumphost-foo-1
Then, programs on jumphost-foo-1–or that can access its port 9090–will be able to access the local port 8000 on workstation.
This method makes using a jumphost transparent. The idea is to tunnel not a single port, but the entire connection over the ssh channel. This usage is typically a matter of configuration in the ssh_config file, taking advantage of its wildcard-capable Host directive:
Host *.lan
ProxyCommand ssh -W %h:22 jumphost-foo-1
This configuration uses the ProxyCommand feature to cooperate with the -W flag. Hostnames are resolved from the perspective of the jumphost. In this example, the machines use an internal pseudo-top level domain of .lan. To reach, e.g., www-1.lan:
ssh www-1.lan
Before doing DNS resolution, OpenSSH will look in its ssh_config file for Host entries. Therefore, internal DNS on the foreign end is sufficient.
Operations work regularly involves connecting to remote servers (there will be times when you do nothing but work on remote servers - parts of this curriculum were even typed on remote servers rather than contributors desktops and laptops!).
There are however two limitations to working this way:
Multiplexers are a good solution to this. They allow you to run multiple “virtual” windows inside a single windows. For example:
Bob works on 10 remote servers, all of which run Linux. Bob’s internet connection at work is questionable. To work around this, Bob connects to server1 which is at his data centre. It is a reliable server which is close to the other servers Bob works on. On server1, Bob starts a multiplexer. The multiplexer gives Bob a regular looking command prompt, and Bob continues his work.
If Bob’s internet connection drops, he can reconnect to server1, and then re-attach to the multiplexer he started previously. His session is in the same state he left it before being disconnected, and he can continue his work.
The multiplexer also lets Bob open more than one command prompt and switch between them as he needs to. Bob can now connect to many servers and see them all in one window.
screen is one of the longest lived multiplexers. Almost everyone who has used a multiplexer has used screen, and you can’t go far wrong with it.
screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells). It is useful for creating sessions that can be disconnected from and reconnected to later. This is useful for running tasks that can take a long time that you do not want to have an ssh session timeout on, such as a large database import. In these cases cron is also a very good way to run one off long running tasks.
screen is also very useful for creating sessions that users can share.
Debian and descendants (Ubuntu, Mint, Suse, etc):
apt-get install screen
On RedHat-style distributions install with the command:
yum install screen
Create a session:
screen -S session1
To detach from a session - in the session type Ctrl+a+d
List available screen sessions:
screen -ls
[gary@mc9 ~]# screen -ls
There are screens on:
21707.session2 (Detached)
21692.session1 (Detached)
21936.session3 (Attached)
3 Sockets in /var/run/screen/S-gary.
[gary@mc9 ~]#
Here we can see 3 screen sessions are running, 2 detached and 1 attached.
Reattach to a session:
screen -r session1
Share a session:
User Alice starts session:
screen -S session1
User Bob can then attach to the same session (both Alice and Bob can send commands to the session):
sudo screen -x alice/session1
Non root users, must use sudo to attach to another user’s session.
Create a session with a log:
screen -L -S session1
screen will output the session log to the user’s home directory with the file ~/screenlog.0 (0 being the session id). PuTTY is also as a very useful and featureful ssh client that can be used for logging ssh sessions locally (Windows and Linux). screen can be used within a PuTTY session.
Create a session with a log and 20000 lines of scrollback in the terminal:
screen -h 20000 -L -S session1
screen has a fairly extensive set of configuration options, when screen is invoked, it executes initialization commands from the files /etc/screenrc and .screenrc in the user’s home directory.
tmux [1] is relatively new compared to screen. It covers the same basic feature set and has added a few more advanced features. It is recommended you get comfortable with screen first before attempting to use tmux.
In this chapter you will learn to start a tmux session, get to know a few first keyboard shortcuts and detach from and re-attach to the session.
tmux is available on Debian and its descendants like Ubuntu or Mint with the command:
apt-get install tmux
On RedHat-style distributions you will have to use the EPEL repository to get a pre-built package, and install with the command:
yum install tmux
On MacOS you can use Homebrew to install via:
brew install tmux
tmux is usually started with the command tmux in a terminal window. Depending of your version of tmux you will see either a line at the bottom of the screen or nothing at all. tmux is controlled with keyboard shortcuts, the default shortcut usually is ctrl-b. If you press ctrl-b and then a t in the newly started tmux window you should see the local time displayed as a large digital clock. If you hit ctrl-b and c you should see a new empty window with an empty input prompt.
If you want to detach from the session you have to hit ctrl-b and d. The tmux window will disappear and you will see a message [detached] in your terminal window. All the shells and processes you started onside the tmux session continue to run, you can see this with a simple
ps -ef | grep tmux
You should see something like the following:
cdrexler 13751 1 0 Nov30 ? 00:00:41 tmux
You will notice that the tmux process has a parent process id of 1 which means that it is not a child process of the shell you started it in anymore. Accordingly you can leave your working shell, start a new one and attach to the running tmux process again which is very handy if your connectivity is flaky or you have to work from different locations. If you check the process table for the process id of the tmux process
ps -ef | grep 13751
you will find that is the parent process of the two shells you created in the beginning of the chapter:
cdrexler 4525 13751 0 17:54 pts/2 00:00:00 -zsh
cdrexler 4533 13751 0 17:54 pts/5 00:00:00 -zsh
If you want to get an overview of the running tmux processes on your system you can use the command
tmux ls
It will list all available tmux sessions on your system [2]. If there is only one you can attach to it with the command:
tmux att
If there is more than one session the output of tmux ls will look like this:
0: 3 windows (created Fri Nov 30 18:32:37 2012) [80x38]
4: 1 windows (created Sun Dec 2 17:44:15 2012) [150x39] (attached)
You will then have to select the right session with the -t command line switch:
tmux att -t 4
tmux runs as a server process that can handle several sessions so you should only see one tmux process per user per system.
You should see the original session with the two shells again after running this command.
tmux is configured via a config file which is usually called .tmux.conf that should live in your $HOME directory.
A typical .tmux.conf looks like this:
#set keyboard shortcut to ctrl-g
unbind C-b
set -g prefix C-g
bind C-g send-prefix
bind g send-prefix
#end of keybord shortcut setting
# Highlight active window
set-window-option -g window-status-current-bg red
# Set window notifications
setw -g monitor-activity on
set -g visual-activity on
#automatically rename windows according to the running program
setw -g automatic-rename
#set scroll back buffer
set -g history-limit 10000
set -g default-terminal "xterm-256color"
set -g base-index 1
set -g status-left '#[fg=green]#H
This illustrates a method to change the default keybinding and some useful settings.
Please note that you can force tmux to use another configfile with the -f command line switch like so:
tmux -f mytmuxconf.conf
There is a nifty cheat sheet [3] for the most important screen and tmux keybindings or even a whole book about tmux [4].
Byobu is a wrapper around one of screen or tmux. It provides profile support, F-keybindings, configuration utilities and a system status notification bar for most Linux, BSD or Mac operating systems.
Byobu is available in major distros as a packaged binary. Launching byobu will run whichever the package maintainer included as a dependency, if you have both installed, you can select explicitly with byobu-{screen,tmux}. Basic configuration is launched with F9 or byobu-config.
Scrollback starts with F7, mark the start of your buffer by hitting the spacebar, then use the spacebar again to mark the end of your selection (which will be copied into byobu’s clipboard automatically). Press Byobu-Escape (typically Ctrl-A) + [ to paste.
| [1] | http://tmux.sourceforge.net/ |
| [2] | Please note that tmux ls will only list tmux sessions that belong to your userid! |
| [3] | http://www.dayid.org/os/notes/tm.html |
| [4] | http://pragprog.com/book/bhtmux/tmux |
| [5] | https://launchpad.net/byobu |
| [Byobu] | http://byobu.co/ |
As you read in Shells, your shell is your primary tool during the work day. It’s also incredibly customisable to suit your needs. Let’s look at some changes you can make.
Your shell’s configuration is stored in its rc file. For bash, this file is ~/.bashrc. Each time you edit this, you can reload the configuration by typing:
source ~/.bashrc
Your default prompt probably looks something like this:
bash-3.2$
That’s pretty plain and doesn’t tell you much. In fact, all it does tell you is that you’re using Bash version 3.2, and that you are not the root user (the $ at the end signifies a regular user, whereas if you were root, you would see a # instead).
Let’s change this up a little. Edit your ~/.bashrc file, and add this line to the end:
PS1="\u@\h \w> "
Save, quit, and then reload your .bashrc file. Your prompt should change to something like this:
avleen@laptop ~>
Much better! Now your know your username, the name of the machine you’re on (in this case “laptop”), and the directory you’re in (“~” is your home directory).
The PS1 variable has a lot of different options you can use to customise it further.
Mosh (MObile SHell) is an alternative to remote shell commands, such as ssh or rsh. The beauty of the Mosh protocol is that it supports intermittent connectivity without losing the remote session.
You can start a mosh session just like you would with ssh on one side of town with one IP address, shut your laptop and go home, then open your laptop and connect back to your Mosh session like it was never interrupted. Also, if your wifi is spotty or internet connection is intermittent, mosh doesn’t break the session when the connection drops out, unlike ssh. Mosh does not wait for the remote server to confirm each keystroke before displaying it to a terminal. Instead, it displays the typed characters locally and confirms entry on the remote end. There are packages available for GNU/Linux, FreeBSD, Solaris, Mac OS X, Chrome and even Android apps.
Mosh must be installed on both the client and remote server. When a session is started it spins up a mosh-server and a local mosh-client process. It can be installed in a home directory without privileged access. Mosh respects your current ~/.ssh/config so migrating from ssh to mosh is relatively seamless.
SSH connections work by sending a stream of data back and forth between the client and server. This stream can be broken in various ways, such as the connection timing out due to inactivity, or the client machine suspending state and shutting down network devices. The Mosh protocol is based on UDP packets compared to the SSH protocol that uses TCP packets. Mosh is the first application to use the Stateless Syncronization Protocol. Instead of a stream of data between the server and client that makes up a session over SSH, Mosh works by keeping a mirrored copy of the session on the client and server and syncronizing the changes between them.
While ssh has been around long enough to have time tested security, mosh is relatively new and has not been through the same extensive testing. It is the first application to use the SSP protocol. Mosh does tunnel traffic encrypted with AES-128 in OES mode, however Mosh hasn’t been under the security spotlight as long as SSH has.
Mosh works just like ssh:
mosh username@remoteserver.org
You can also have mosh utilize ssh style commands such as: This specifies the private key to use to authenticate with the remove server.
mosh username@remoteserver.org --ssh="ssh -i ~/.ssh/identity_file"
This tells mosh to connect via the ssh port 1234 on the remote server, where ssh normally runs on port 22.
mosh username@remoteserver.org --ssh="ssh -p 1234"``