Monday, 22 February 2021

Linux Admin - Firewall Setup

 firewalld is the default front-end controller for iptables on CentOS. The firewalld front-end has two main advantages over raw iptables −

  • Uses easy-to-configure and implement zones abstracting chains and rules.

  • Rulesets are dynamic, meaning stateful connections are uninterrupted when the settings are changed and/or modified.

Remember, firewalld is the wrapper for iptables - not a replacement. While custom iptables commands can be used with firewalld, it is recommended to use firewalld as to not break the firewall functionality.

First, let's make sure firewalld is both started and enabled.

[root@CentOS rdc]# systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon 
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) 
Active: active (running) since Thu 2017-01-26 21:42:05 MST; 3h 46min ago 
 Docs: man:firewalld(1) 
Main PID: 712 (firewalld) 
  Memory: 34.7M 
 CGroup: /system.slice/firewalld.service 
       └─712 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

We can see, firewalld is both active (to start on boot) and currently running. If inactive or not started we can use −

systemctl start firewalld && systemctl enable firewalld

Now that we have our firewalld service configured, let's assure it is operational.

[root@CentOS]# firewall-cmd --state 
running 
[root@CentOS]#

We can see, the firewalld service is fully functional.

Firewalld works on the concept of zones. A zone is applied to network interfaces through the Network Manager. We will discuss this in configuring networking. But for now, by default, changing the default zone will change any network adapters left in the default state of "Default Zone".

Let's take a quick look at each zone that comes out-of-the-box with firewalld.

Sr.No.Zone & Description
1

drop

Low trust level. All incoming connections and packetsare dropped and only outgoing connections are possible via statefullness

2

block

Incoming connections are replied with an icmp message letting the initiator know the request is prohibited

3

public

All networks are restricted. However, selected incoming connections can be explicitly allowed

4

external

Configures firewalld for NAT. Internal network remains private but reachable

5

dmz

Only certain incoming connections are allowed. Used for systems in DMZ isolation

6

work

By default, trust more computers on the network assuming the system is in a secured work environment

7

hone

By default, more services are unfiltered. Assuming a system is on a home network where services such as NFS, SAMBA and SSDP will be used

8

trusted

All machines on the network are trusted. Most incoming connections are allowed unfettered. This is not meant for interfaces exposed to the Internet

The most common zones to use are:public, drop, work, and home.

Some scenarios where each common zone would be used are −

  • public − It is the most common zone used by an administrator. It will let you apply the custom settings and abide by RFC specifications for operations on a LAN.

  • drop − A good example of when to use drop is at a security conference, on public WiFi, or on an interface connected directly to the Internet. drop assumes all unsolicited requests are malicious including ICMP probes. So any request out of state will not receive a reply. The downside of drop is that it can break the functionality of applications in certain situations requiring strict RFC compliance.

  • work − You are on a semi-secure corporate LAN. Where all traffic can be assumed moderately safe. This means it is not WiFi and we possibly have IDS, IPS, and physical security or 802.1x in place. We also should be familiar with the people using the LAN.

  • home − You are on a home LAN. You are personally accountable for every system and the user on the LAN. You know every machine on the LAN and that none have been compromised. Often new services are brought up for media sharing amongst trusted individuals and you don't need to take extra time for the sake of security.

Zones and network interfaces work on a one to many level. One network interface can only have a single zone applied to it at a time. While, a zone can be applied to many interfaces simultaneously.

Let's see what zones are available and what are the currently applied zone.

[root@CentOS]# firewall-cmd --get-zones 
 work drop internal external trusted home dmz public block

[root@CentOS]# firewall-cmd --get-default-zone 
public
[root@CentOS]#

Ready to add some customized rules in firewalld?

First, let's see what our box looks like, to a portscanner from outside.

bash-3.2# nmap -sS -p 1-1024 -T 5  10.211.55.1
 
Starting Nmap 7.30 ( https://nmap.org ) at 2017-01-27 23:36 MST 
Nmap scan report for centos.shared (10.211.55.1) 
Host is up (0.00046s latency). 
Not shown: 1023 filtered ports 
PORT   STATE SERVICE 
22/tcp open  ssh


Nmap done: 1 IP address (1 host up) scanned in 3.71 seconds 
bash-3.2#

Let's allow the incoming requests to port 80.

First, check to see what zone is applied as default.

[root@CentOs]# firewall-cmd --get-default-zone 
public
[root@CentOS]#

Then, set the rule allowing port 80 to the current default zone.

[root@CentOS]# firewall-cmd --zone=public --add-port = 80/tcp 
success
[root@CentOS]#

Now, let's check our box after allowing port 80 connections.

bash-3.2# nmap -sS -p 1-1024 -T 5  10.211.55.1

Starting Nmap 7.30 ( https://nmap.org ) at 2017-01-27 23:42 MST 
Nmap scan report for centos.shared (10.211.55.1) 
Host is up (0.00053s latency). 
Not shown: 1022 filtered ports 
PORT   STATE  SERVICE 
22/tcp open   ssh 
80/tcp closed http

Nmap done: 1 IP address (1 host up) scanned in 3.67 seconds 
bash-3.2#

It now allows unsolicited traffic to 80.

Let's put the default zone to drop and see what happens to port scan.

[root@CentOS]# firewall-cmd --set-default-zone=drop 
success

[root@CentOS]# firewall-cmd --get-default-zone 
drop

[root@CentOs]#

Now let's scan the host with the network interface in a more secure zone.

bash-3.2# nmap -sS -p 1-1024 -T 5  10.211.55.1 
Starting Nmap 7.30 ( https://nmap.org ) at 2017-01-27 23:50 MST 
Nmap scan report for centos.shared (10.211.55.1) 
Host is up (0.00094s latency). 
All 1024 scanned ports on centos.shared (10.211.55.1) are filtered

Nmap done: 1 IP address (1 host up) scanned in 12.61 seconds 
bash-3.2#

Now, everything is filtered from outside.

As demonstrated below, the host will not even respond to ICMP ping requests when in drop.

bash-3.2# ping 10.211.55.1 
PING 10.211.55.1 (10.211.55.1): 56 data bytes 
Request timeout for icmp_seq 0 
Request timeout for icmp_seq 1 
Request timeout for icmp_seq 2

Let's set the default zone to public again.

[root@CentOs]# firewall-cmd --set-default-zone=public 
success

[root@CentOS]# firewall-cmd --get-default-zone 
public

[root@CentOS]#

Now let's check our current filtering ruleset in public.

[root@CentOS]# firewall-cmd --zone=public --list-all 
public (active) 
target: default 
icmp-block-inversion: no 
interfaces: enp0s5 
sources:  
services: dhcpv6-client ssh 
ports: 80/tcp 
protocols:  
masquerade: no 
forward-ports:  
sourceports:  
icmp-blocks:  
rich rules:

[root@CentOS rdc]#

As configured, our port 80 filter rule is only within the context of the running configuration. This means once the system is rebooted or the firewalld service is restarted, our rule will be discarded.

We will be configuring an httpd daemon soon, so let's make our changes persistent −

[root@CentOS]# firewall-cmd --zone=public --add-port=80/tcp --permanent 
success

[root@CentOS]# systemctl restart firewalld

[root@CentOS]#

Now our port 80 rule in the public zone is persistent across reboots and service restarts.

Following are the common firewalld commands applied with firewall-cmd.

CommandAction
firewall-cmd --get-zonesLists all zones that can be applied to an interface
firewall-cmd —statusReturns the currents status of the firewalld service
firewall-cmd --get-default-zoneGets the current default zone
firewall-cmd --set-default-zone=<zone>Sets the default zone into the current context
firewall-cmd --get-active-zoneGets the current zones in context as applied to an interface
firewall-cmd --zone=<zone> --list-allLists the configuration of supplied zone
firewall-cmd --zone=<zone> --addport=<port/transport protocol>Applies a port rule to the zone filter
--permanentMakes changes to the zone persistent. Flag is used inline with modification commands

These are the basic concepts of administrating and configuring firewalld.

Configuring host-based firewall services in CentOS can be a complex task in more sophisticated networking scenarios. Advanced usage and configuration of firewalld and iptables in CentOS can take an entire tutorial. However, we have presented the basics that should be enough to complete a majority of daily tasks.

Linux Admin - Process Management

 Following are the common commands used with Process Management–bg, fg, nohup, ps, pstree, top, kill, killall, free, uptime, nice.

Work with Processes

Quick Note − Process PID in Linux

In Linux every running process is given a PID or Process ID Number. This PID is how CentOS identifies a particular process. As we have discussed, systemd is the first process started and given a PID of 1 in CentOS.

Pgrep is used to get Linux PID for a given process name.

[root@CentOS]# pgrep systemd 
1 
[root@CentOS]#

As seen, the pgrep command returns the current PID of systemd.

Basic CentOS Process and Job Management in CentOS

When working with processes in Linux it is important to know how basic foregrounding and backgrounding processes is performed at the command line.

  • fg − Bringsthe process to the foreground

  • bg − Movesthe process to the background

  • jobs − List of the current processes attached to the shell

  • ctrl+z − Control + z key combination to sleep the current process

  • & − Startsthe process in the background

Let's start using the shell command sleepsleep will simply do as it is named, sleep for a defined period of time − sleep.

[root@CentOS ~]$ jobs

[root@CentOS ~]$ sleep 10 & 
[1] 12454 

[root@CentOS ~]$ sleep 20 & 
[2] 12479

[root@CentOS ~]$ jobs 
[1]-  Running                 sleep 10 & 
[2]+  Running                 sleep 20 &

[cnetos@CentOS ~]$

Now, let's bring the first job to the foreground −

[root@CentOS ~]$ fg 1 
sleep 10

If you are following along, you'll notice the foreground job is stuck in your shell. Now, let's put the process to sleep, then re-enable it in the background.

  • Hit control+z
  • Type: bg 1, sending the first job into the background and starting it.
[root@CentOS ~]$ fg 1 
sleep 20 
^Z 
[1]+  Stopped                 sleep 20

[root@CentOS ~]$ bg 1 
[1]+ sleep 20 &

[root@CentOS ~]$

nohup

When working from a shell or terminal, it is worth noting that by default all the processes and jobs attached to the shell will terminate when the shell is closed or the user logs out. When using nohup the process will continue to run if the user logs out or closes the shell to which the process is attached.

[root@CentOS]# nohup ping www.google.com & 
[1] 27299 
nohup: ignoring input and appending output to ‘nohup.out’

[root@CentOS]# pgrep ping 
27299

[root@CentOS]# kill -KILL `pgrep ping` 
[1]+  Killed                  nohup ping www.google.com

[root@CentOS rdc]# cat nohup.out  
PING www.google.com (216.58.193.68) 56(84) bytes of data. 
64 bytes from sea15s07-in-f4.1e100.net (216.58.193.68): icmp_seq = 1 ttl = 128
time = 51.6 ms 
64 bytes from sea15s07-in-f4.1e100.net (216.58.193.68): icmp_seq = 2 ttl = 128
time = 54.2 ms 
64 bytes from sea15s07-in-f4.1e100.net (216.58.193.68): icmp_seq = 3 ttl = 128
time = 52.7 ms

ps Command

The ps command is commonly used by administrators to investigate snapshots of a specific process. ps is commonly used with grep to filter out a specific process to analyze.

[root@CentOS ~]$ ps axw | grep python 
762   ?        Ssl    0:01 /usr/bin/python -Es /usr/sbin/firewalld --nofork -nopid 
1296  ?        Ssl    0:00 /usr/bin/python -Es /usr/sbin/tuned -l -P 
15550 pts/0    S+     0:00 grep --color=auto python

In the above command, we see all the processes using the python interpreter. Also included with the results were our grep command, looking for the string python.

Following are the most common command line switches used with ps.

SwitchAction
aExcludes constraints of only the reporting processes for the current user
xShows processes not attached to a tty or shell
wFormats wide output display of the output
eShows environment after the command
-eSelects all processes
-oUser-defined formatted output
-uShows all processes by a specific user
-CShows all processes by name or process id
--sortSorts the processes by definition

To see all processes in use by the nobody user −

[root@CentOS ~]$ ps -u nobody 
PID TTY          TIME CMD 
1853 ?        00:00:00 dnsmasq 

[root@CentOS ~]$

To see all information about the firewalld process −

[root@CentOS ~]$ ps -wl -C firewalld 
F   S   UID   PID   PPID   C   PRI   NI   ADDR   SZ   WCHAN   TTY   TIME      CMD
0   S     0   762      1   0    80   0     -   81786  poll_s   ?   00:00:01 firewalld 

[root@CentOS ~]$

Let's see which processes are consuming the most memory −

[root@CentOS ~]$ ps aux  --sort=-pmem | head -10 
USER       PID   %CPU   %MEM   VSZ     RSS   TTY   STAT   START   TIME   COMMAND 
cnetos     6130   0.7   5.7   1344512 108364  ?      Sl   02:16   0:29  /usr/bin/gnome-shell 
cnetos     6449   0.0   3.4   1375872 64440   ?      Sl   02:16   0:00  /usr/libexec/evolution-calendar-factory 
root       5404   0.6   2.1   190256  39920 tty1     Ssl+ 02:15   0:27  /usr/bin/Xorg :0 -background none -noreset -audit 4 -verbose -auth /run/gdm/auth-for-gdm-iDefCt/database -seat seat0 -nolisten tcp vt1 
cnetos     6296   0.0   1.7   1081944 32136   ?      Sl   02:16   0:00  /usr/libexec/evolution/3.12/evolution-alarm-notify 
cnetos     6350   0.0   1.5   560728  29844   ?      Sl   02:16   0:01  /usr/bin/prlsga 
cnetos     6158   0.0   1.4   1026956 28004   ?      Sl   02:16   0:00  /usr/libexec/gnome-shell-calendar-server 
cnetos     6169   0.0   1.4   1120028 27576   ?      Sl   02:16   0:00  /usr/libexec/evolution-source-registry 
root       762    0.0   1.4   327144  26724   ?      Ssl  02:09   0:01  /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid 
cnetos     6026   0.0  1.4 1090832 26376      ?      Sl   02:16   0:00  /usr/libexec/gnome-settings-daemon

[root@CentOS ~]$

See all the processes by user centos and format, displaying the custom output −

[cnetos@CentOS ~]$ ps -u cnetos -o pid,uname,comm 
   PID    USER     COMMAND 
   5802  centos   gnome-keyring-d 
   5812  cnetos   gnome-session 
   5819  cnetos   dbus-launch 
   5820  cnetos   dbus-daemon 
   5888  cnetos   gvfsd 
   5893  cnetos   gvfsd-fuse 
   5980  cnetos   ssh-agent   
   5996  cnetos   at-spi-bus-laun

pstree Command

pstree is similar to ps but is not often used. It displays the processes in a neater tree fashion.

[centos@CentOS ~]$ pstree 
  systemd─┬─ModemManager───2*[{ModemManager}] 
          ├─NetworkManager─┬─dhclient 
          │                └─2*[{NetworkManager}] 
          ├─2*[abrt-watch-log] 
          ├─abrtd 
          ├─accounts-daemon───2*[{accounts-daemon}] 
          ├─alsactl 
          ├─at-spi-bus-laun─┬─dbus-daemon───{dbus-daemon} 
          │                 └─3*[{at-spi-bus-laun}] 
          ├─at-spi2-registr───2*[{at-spi2-registr}] 
          ├─atd 
          ├─auditd─┬─audispd─┬─sedispatch 
          │        │         └─{audispd} 
          │        └─{auditd} 
          ├─avahi-daemon───avahi-daemon 
          ├─caribou───2*[{caribou}] 
          ├─cgrulesengd 
          ├─chronyd 
          ├─colord───2*[{colord}] 
          ├─crond 
          ├─cupsd

The total output from pstree can exceed 100 lines. Usually, ps will give more useful information.

top Command

top is one of the most often used commands when troubleshooting performance issues in Linux. It is useful for real-time stats and process monitoring in Linux. Following is the default output of top when brought up from the command line.

Tasks: 170 total,   1 running, 169 sleeping,   0 stopped,   0 zombie 
%Cpu(s):  2.3 us,  2.0 sy,  0.0 ni, 95.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
KiB Mem :  1879668 total,   177020 free,   607544 used,  1095104 buff/cache 
KiB Swap:  3145724 total,  3145428 free,      296 used.  1034648 avail Mem 
 
PID    USER     PR   NI    VIRT     RES   SHR    S  %CPU  %MEM   TIME+   COMMAND
5404   root     20   0    197832   48024  6744   S   1.3   2.6  1:13.22   Xorg
8013   centos   20   0    555316   23104  13140  S   1.0   1.2  0:14.89   gnome-terminal-
6339   centos   20   0    332336   6016   3248   S   0.3   0.3  0:23.71   prlcc
6351   centos   20   0    21044    1532   1292   S   0.3   0.1  0:02.66   prlshprof

Common hot keys used while running top (hot keys are accessed by pressing the key as top is running in your shell).

CommandAction
bEnables / disables bold highlighting on top menu
zCycles the color scheme
lCycles the load average heading
mCycles the memory average heading
tTask information heading
hHelp menu
Shift+FCustomizes sorting and display fields

Following are the common command line switches for top.

CommandAction
-oSorts by column (can prepend with - or + to sort ascending or descending)
-uShows only processes from a specified user
-dUpdates the delay time of top
-OReturns a list of columns which top can apply sorting

Sorting options screen in top, presented using Shift+F. This screen allows customization of top display and sort options.

Fields Management for window 1:Def, whose current sort field is %MEM 
Navigate with Up/Dn, Right selects for move then <Enter> or Left commits, 
 'd' or <Space> toggles display, 's' sets sort.  Use 'q' or <Esc> to end!
 
* PID     = Process Id             TGID    = Thread Group Id      
* USER    = Effective User Name    ENVIRON = Environment vars     
* PR      = Priority               vMj     = Major Faults delta   
* NI      = Nice Value             vMn     = Minor Faults delta   
* VIRT    = Virtual Image (KiB)    USED    = Res+Swap Size (KiB)  
* RES     = Resident Size (KiB)    nsIPC   = IPC namespace Inode  
* SHR     = Shared Memory (KiB)    nsMNT   = MNT namespace Inode
* S       = Process Status         nsNET   = NET namespace Inode  
* %CPU    = CPU Usage              nsPID   = PID namespace Inode  
* %MEM    = Memory Usage (RES)     nsUSER  = USER namespace Inode 
* TIME+   = CPU Time, hundredths   nsUTS   = UTS namespace Inode  
* COMMAND = Command Name/Line 
PPID    = Parent Process pid
UID     = Effective User Id

top, showing the processes for user rdc and sorted by memory usage −

 PID   USER  %MEM  PR  NI    VIRT    RES    SHR    S %CPU     TIME+    COMMAND
 6130  rdc    6.2  20   0  1349592  117160  33232  S  0.0   1:09.34    gnome-shell
 6449  rdc    3.4  20   0  1375872   64428  21400  S  0.0   0:00.43    evolution-calen
 6296  rdc    1.7  20   0  1081944   32140  22596  S  0.0   0:00.40    evolution-alarm
 6350  rdc    1.6  20   0   560728   29844   4256  S  0.0   0:10.16    prlsga
 6281  rdc    1.5  20   0  1027176   28808  17680  S  0.0   0:00.78    nautilus
 6158  rdc    1.5  20   0  1026956   28004  19072  S  0.0   0:00.20    gnome-shell-cal

Showing valid top fields (condensed) −

[centos@CentOS ~]$ top -O 
PID 
PPID 
UID 
USER 
RUID 
RUSER 
SUID 
SUSER 
GID 
GROUP 
PGRP 
TTY 
TPGID

kill Command

The kill command is used to kill a process from the command shell via its PID. When killing a process, we need to specify a signal to send. The signal lets the kernel know how we want to end the process. The most commonly used signals are −

  • SIGTERM is implied as the kernel lets a process know it should stop soon as it is safe to do so. SIGTERM gives the process an opportunity to exit gracefully and perform safe exit operations.

  • SIGHUP most daemons will restart when sent SIGHUP. This is often used on the processes when changes have been made to a configuration file.

  • SIGKILL since SIGTERM is the equivalent to asking a process to shut down. The kernel needs an option to end a process that will not comply with requests. When a process is hung, the SIGKILL option is used to shut the process down explicitly.

For a list off all signals that can be sent with kill the -l option can be used −

[root@CentOS]# kill -l 
1) SIGHUP           2) SIGINT         3) SIGQUIT        4) SIGILL         5) SIGTRAP
6) SIGABRT          7) SIGBUS         8) SIGFPE         9) SIGKILL       10) SIGUSR1
11) SIGSEGV        12) SIGUSR2       13) SIGPIPE       14) SIGALRM       15) SIGTERM
16) SIGSTKFLT      17) SIGCHLD       18) SIGCONT       19) SIGSTOP       20) SIGTSTP
21) SIGTTIN        22) SIGTTOU       23) SIGURG        24) SIGXCPU       25) SIGXFSZ
26) SIGVTALRM      27) SIGPROF       28) SIGWINCH      29) SIGIO         30) SIGPWR
31) SIGSYS         34) SIGRTMIN      35) SIGRTMIN+1    36) SIGRTMIN+2    37) SIGRTMIN+3
38) SIGRTMIN+4     39) SIGRTMIN+5    40) SIGRTMIN+6    41) SIGRTMIN+7    42) SIGRTMIN+8
43) SIGRTMIN+9     44) SIGRTMIN+10   45) SIGRTMIN+11   46) SIGRTMIN+12   47) SIGRTMIN+13 
48) SIGRTMIN+14    49) SIGRTMIN+15   50) SIGRTMAX-14   51) SIGRTMAX-13   52) SIGRTMAX-12 
53) SIGRTMAX-11    54) SIGRTMAX-10   55) SIGRTMAX-9    56) SIGRTMAX-8    57) SIGRTMAX-7
58) SIGRTMAX-6     59) SIGRTMAX-5    60) SIGRTMAX-4    61) SIGRTMAX-3    62) SIGRTMAX-2
63) SIGRTMAX-1     64) SIGRTMAX

[root@CentOS rdc]#

Using SIGHUP to restart system.

[root@CentOS]# pgrep systemd 
1 
464 
500 
643 
15071

[root@CentOS]# kill -HUP 1

[root@CentOS]# pgrep systemd
1 
464 
500 
643 
15196 
15197 
15198

[root@CentOS]#

pkill will allow the administrator to send a kill signal by the process name.

[root@CentOS]# pgrep ping 
19450 
[root@CentOS]# pkill -9 ping 
[root@CentOS]# pgrep ping 
[root@CentOS]#

killall will kill all the processes. Be careful using killall as root, as it will kill all the processes for all users.

[root@CentOS]# killall chrome

free Command

free is a pretty simple command often used to quickly check the memory of a system. It displays the total amount of used physical and swap memory.

[root@CentOS]# free 
             total       used      free      shared      buff/cache      available 
Mem:        1879668     526284    699796     10304        653588          1141412 
Swap:       3145724          0    3145724

[root@CentOS]# 

nice Command

nice will allow an administrator to set the scheduling priority of a process in terms of CPU usages. The niceness is basically how the kernel will schedule CPU time slices for a process or job. By default, it is assumed the process is given equal access to CPU resources.

First, let's use top to check the niceness of the currently running processes.

PID   USER   PR   NI    VIRT    RES    SHR   S  %CPU  %MEM     TIME+    COMMAND
28    root   39   19       0      0      0   S  0.0   0.0    0:00.17    khugepaged
690   root   39   19   16808   1396   1164   S  0.0   0.1    0:00.01    alsactl]
9598  rdc    39   19  980596  21904  10284   S  0.0   1.2    0:00.27    tracker-extract
9599  rdc    39   19  469876   9608   6980   S  0.0   0.5    0:00.04    tracker-miner-a
9609  rdc    39   19  636528  13172   8044   S  0.0   0.7    0:00.12    tracker-miner-f
9611  rdc    39   19  469620   8984   6496   S  0.0   0.5    0:00.02    tracker-miner-u
27    root   25    5       0      0      0   S  0.0   0.0    0:00.00    ksmd
637   rtkit  21    1  164648   1276   1068   S  0.0   0.1    0:00.11    rtkit-daemon
1     root   20    0  128096   6712   3964   S  0.3   0.4    0:03.57    systemd
2     root   20    0       0      0      0   S  0.0   0.0    0:00.01    kthreadd
3     root   20    0       0      0      0   S  0.0   0.0    0:00.50    ksoftirqd/0
7     root   20    0       0      0      0   S  0.0   0.0    0:00.00    migration/0
8     root   20    0       0      0      0   S  0.0   0.0    0:00.00    rcu_bh
9     root   20    0       0      0      0   S  0.0   0.0    0:02.07    rcu_sched

We want to focus on the NICE column depicted by NI. The niceness range can be anywhere between -20 to positive 19. -20 represents the highest given priority.

nohup nice --20 ping www.google.com &

PID USER    PR NI  VIRT    RES  SHR  S %CPU %MEM  TIME+    COMMAND
30727 root  0 -20  132108 1640  1264 S 0.0   0.1  0:00.06   ping

renice

renice allows us to change the current priority of a process that is already running.

renice 17 -p 30727

The above command will lower the priority of our ping process command.

Linux Admin - Resource Mgmt with crgoups

 cgroups or Control Groups are a feature of the Linux kernel that allows an administrator to allocate or cap the system resources for services and also group.

To list active control groups running, we can use the following ps command −

[root@localhost]# ps xawf -eo pid,user,cgroup,args 
8362 root     -                            \_ [kworker/1:2] 
1 root        -                           /usr/lib/systemd/systemd --switched-
   root --system --    deserialize 21 
507 root     7:cpuacct,cpu:/system.slice  /usr/lib/systemd/systemd-journald 
527 root     7:cpuacct,cpu:/system.slice  /usr/sbin/lvmetad -f 
540 root     7:cpuacct,cpu:/system.slice  /usr/lib/systemd/systemd-udevd 
715 root     7:cpuacct,cpu:/system.slice  /sbin/auditd -n 
731 root     7:cpuacct,cpu:/system.slice   \_ /sbin/audispd 
734 root     7:cpuacct,cpu:/system.slice       \_ /usr/sbin/sedispatch 
737 polkitd  7:cpuacct,cpu:/system.slice  /usr/lib/polkit-1/polkitd --no-debug 
738 rtkit    6:memory:/system.slice/rtki  /usr/libexec/rtkit-daemon 
740 dbus     7:cpuacct,cpu:/system.slice  /bin/dbus-daemon --system --
   address=systemd: --nofork --nopidfile --systemd-activation

Resource Management, as of CentOS 6.X, has been redefined with the systemd init implementation. When thinking Resource Management for services, the main thing to focus on are cgroupscgroups have advanced with systemd in both functionality and simplicity.

The goal of cgroups in resource management is -no one service can take the system, as a whole, down. Or no single service process (perhaps a poorly written PHP script) will cripple the server functionality by consuming too many resources.

cgroups allow resource control of units for the following resources −

  • CPU − Limit cpu intensive tasks that are not critical as other, less intensive tasks

  • Memory − Limit how much memory a service can consume

  • Disks − Limit disk i/o

**CPU Time: **

Tasks needing less CPU priority can have custom configured CPU Slices.

Let's take a look at the following two services for example.

Polite CPU Service 1

[root@localhost]# systemctl cat polite.service 
# /etc/systemd/system/polite.service 
[Unit] 
Description = Polite service limits CPU Slice and Memory 
After=remote-fs.target nss-lookup.target

[Service] 
MemoryLimit = 1M 
ExecStart = /usr/bin/sha1sum /dev/zero 
ExecStop = /bin/kill -WINCH ${MAINPID} 
WantedBy=multi-user.target

# /etc/systemd/system/polite.service.d/50-CPUShares.conf 
[Service] 
CPUShares = 1024 
[root@localhost]#

Evil CPU Service 2

[root@localhost]# systemctl cat evil.service 
# /etc/systemd/system/evil.service 
[Unit] 
Description = I Eat You CPU 
After=remote-fs.target nss-lookup.target

[Service] 
ExecStart = /usr/bin/md5sum /dev/zero 
ExecStop = /bin/kill -WINCH ${MAINPID} 
WantedBy=multi-user.target

# /etc/systemd/system/evil.service.d/50-CPUShares.conf 
[Service] 
CPUShares = 1024 
[root@localhost]#

Let's set Polite Service using a lesser CPU priority −

systemctl set-property polite.service CPUShares = 20  
/system.slice/polite.service
1   70.5   124.0K        -        -  

/system.slice/evil.service
1   99.5   304.0K        -        -

As we can see, over a period of normal system idle time, both rogue processes are still using CPU cycles. However, the one set to have less time-slices is using less CPU time. With this in mind, we can see how using a lesser time time-slice would allow essential tasks better access the system resources.

To set services for each resource, the set-property method defines the following parameters −

systemctl set-property name parameter=value

CPU SlicesCPUShares
Memory LimitMemoryLimit
Soft Memory LimitMemorySoftLimit
Block IO WeightBlockIOWeight
Block Device Limit (specified in /volume/path) )BlockIODeviceWeight
Read IOBlockIOReadBandwidth
Disk Write IOBlockIOReadBandwidth

Most often services will be limited by CPU useMemory limits and Read / Write IO.

After changing each, it is necessary to reload systemd and restart the service −

systemctl set-property foo.service CPUShares = 250 
systemctl daemon-reload 
systemctl restart foo.service

Configure CGroups in CentOS Linux

To make custom cgroups in CentOS Linux, we need to first install services and configure them.

Step 1 − Install libcgroup (if not already installed).

[root@localhost]# yum install libcgroup 
Package libcgroup-0.41-11.el7.x86_64 already installed and latest version 
Nothing to do 
[root@localhost]#

As we can see, by default CentOS 7 has libcgroup installed with the everything installer. Using a minimal installer will require us to install the libcgroup utilities along with any dependencies.

Step 2 − Start and enable the cgconfig service.

[root@localhost]# systemctl enable cgconfig 
Created symlink from /etc/systemd/system/sysinit.target.wants/cgconfig.service 
to /usr/lib/systemd/system/cgconfig.service. 
[root@localhost]# systemctl start cgconfig 
[root@localhost]# systemctl status cgconfig 
● cgconfig.service - Control Group configuration service 
Loaded: loaded (/usr/lib/systemd/system/cgconfig.service; enabled; vendor 
preset: disabled) 
Active: active (exited) since Mon 2017-01-23 02:51:42 EST; 1min 21s ago 
Main PID: 4692 (code=exited, status = 0/SUCCESS) 
Memory: 0B 
CGroup: /system.slice/cgconfig.service  

Jan 23 02:51:42 localhost.localdomain systemd[1]: Starting Control Group 
configuration service... 
Jan 23 02:51:42 localhost.localdomain systemd[1]: Started Control Group 
configuration service. 
[root@localhost]#

Resource Mgmt with systemctl

 systemctl is the utility used to control systemd. systemctl provides CentOS administrators with the ability to perform a multitude of operations on systemd including −

  • Configure systemd units
  • Get status of systemd untis
  • Start and stop services
  • Enable / disable systemd services for runtime, etc.

The command syntax for systemctl is pretty basic, but can tangle with switches and options. We will present the most essential functions of systemctl needed for administering CentOS Linux.

Basic systemctl syntax: 
systemctl [OPTIONS] COMMAND [NAME]

Following are the common commands used with systemctl −

  • start
  • stop
  • restart
  • reload
  • status
  • is-active
  • list-units
  • enable
  • disable
  • cat
  • show

We have already discussed startstopreloadrestartenable and disable with systemctl. So let's go over the remaining commonly used commands.

status

In its most simple form, the status command can be used to see the system status as a whole −

[root@localhost rdc]# systemctl status 
 ● localhost.localdomain 
  State: running 
  Jobs: 0 queued
  Failed: 0 units 
  Since: Thu 2017-01-19 19:14:37 EST; 4h 5min ago 
CGroup: / 
       ├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21 
       ├─user.slice 
       │ └─user-1002.slice 
       │   └─session-1.scope 
       │     ├─2869 gdm-session-worker [pam/gdm-password] 
       │     ├─2881 /usr/bin/gnome-keyring-daemon --daemonize --login 
       │     ├─2888 gnome-session --session gnome-classic 
       │     ├─2895 dbus-launch --sh-syntax --exit-with-session

The above output has been condensed. In the real-world systemctl status will output about 100 lines of treed process statuses.

Let's say we want to check the status of our firewall service −

[root@localhost rdc]# systemctl status firewalld 
● firewalld.service - firewalld - dynamic firewall daemon 
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled) 
Active: active (running) since Thu 2017-01-19 19:14:55 EST; 4h 12min ago 
 Docs: man:firewalld(1) 
Main PID: 825 (firewalld) 
CGroup: /system.slice/firewalld.service 
       └─825 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

As you see, our firewall service is currently active and has been for over 4 hours.

list-units

The list-units command allows us to list all the units of a certain type. Let's check for sockets managed by systemd −

[root@localhost]# systemctl list-units --type=socket 
UNIT                         LOAD     ACTIVE     SUB     DESCRIPTION     
avahi-daemon.socket          loaded   active  running    Avahi mDNS/DNS-SD Stack Activation Socket 
cups.socket                  loaded   active  running    CUPS Printing Service Sockets 
dbus.socket                  loaded   active  running    D-Bus System Message Bus Socket 
dm-event.socket              loaded   active  listening  Device-mapper event daemon FIFOs 
iscsid.socket                loaded   active  listening  Open-iSCSI iscsid Socket
iscsiuio.socket              loaded   active  listening  Open-iSCSI iscsiuio Socket 
lvm2-lvmetad.socket          loaded   active  running    LVM2 metadata daemon socket 
lvm2-lvmpolld.socket         loaded   active  listening  LVM2 poll daemon socket 
rpcbind.socket               loaded   active  listening  RPCbind Server Activation Socket 
systemd-initctl.socket       loaded   active  listening  /dev/initctl Compatibility Named Pipe 
systemd-journald.socket      loaded   active  running    Journal Socket 
systemd-shutdownd.socket     loaded   active  listening  Delayed Shutdown Socket 
systemd-udevd-control.socket loaded   active  running    udev Control Socket 
systemd-udevd-kernel.socket  loaded   active  running    udev Kernel Socket 
virtlockd.socket             loaded   active  listening  Virtual machine lock manager socket 
virtlogd.socket              loaded   active  listening  Virtual machine log manager socket

Now let’s check the current running services −

[root@localhost rdc]# systemctl list-units --type=service 
UNIT                      LOAD     ACTIVE     SUB     DESCRIPTION 
abrt-ccpp.service         loaded   active   exited    Install ABRT coredump hook 
abrt-oops.service         loaded   active   running   ABRT kernel log watcher 
abrt-xorg.service         loaded   active   running   ABRT Xorg log watcher 
abrtd.service             loaded   active   running   ABRT Automated Bug Reporting Tool 
accounts-daemon.service   loaded   active   running   Accounts Service 
alsa-state.service        loaded   active   running   Manage Sound Card State (restore and store) 
atd.service               loaded   active   running   Job spooling tools 
auditd.service            loaded   active   running   Security Auditing Service

is-active

The is-active command is an example of systemctl commands designed to return the status information of a unit.

[root@localhost rdc]# systemctl is-active ksm.service 
active

cat

cat is one of the seldomly used command. Instead of using cat at the shell and typing the path to a unit file, simply use systemctl cat.

[root@localhost]# systemctl cat firewalld 
# /usr/lib/systemd/system/firewalld.service
[Unit] 
Description=firewalld - dynamic firewall daemon 
Before=network.target 
Before=libvirtd.service 
Before = NetworkManager.service 
After=dbus.service 
After=polkit.service 
Conflicts=iptables.service ip6tables.service ebtables.service ipset.service 
Documentation=man:firewalld(1)

[Service] 
EnvironmentFile = -/etc/sysconfig/firewalld 
ExecStart = /usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS 
ExecReload = /bin/kill -HUP $MAINPID 
# supress to log debug and error output also to /var/log/messages 
StandardOutput = null 
StandardError = null

Type = dbus 
BusName = org.fedoraproject.FirewallD1

[Install] 
WantedBy = basic.target 
Alias = dbus-org.fedoraproject.FirewallD1.service

[root@localhost]#

Now that we have explored both systemd and systemctl in more detail, let's use them to manage the resources in cgroups or control groups.

Systemd Services Start and Stop

 systemd is the new way of running services on Linux. systemd has a superceded sysvinit. systemd brings faster boot-times to Linux and is now, a standard way to manage Linux services. While stable, systemd is still evolving.

systemd as an init system, is used to manage both services and daemons that need status changes after the Linux kernel has been booted. By status change starting, stopping, reloading, and adjusting service state is applied.

First, let's check the version of systemd currently running on our server.

[centos@localhost ~]$ systemctl --version 
systemd 219 
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP 
+GCRYPT +GNUTLS +ACL     +XZ -LZ4 -SECCOMP +BLKID +ELFUTILS +KMOD +IDN

[centos@localhost ~]$

As of CentOS version 7, fully updated at the time of this writing systemd version 219 is the current stable version.

We can also analyze the last server boot time with systemd-analyze

[centos@localhost ~]$ systemd-analyze    
Startup finished in 1.580s (kernel) + 908ms (initrd) + 53.225s (userspace) = 55.713s 
[centos@localhost ~]$

When the system boot times are slower, we can use the systemd-analyze blame command.

[centos@localhost ~]$ systemd-analyze blame 
   40.882s kdump.service 
   5.775s NetworkManager-wait-online.service 
   4.701s plymouth-quit-wait.service 
   3.586s postfix.service 
   3.121s systemd-udev-settle.service 
   2.649s tuned.service 
   1.848s libvirtd.service 
   1.437s network.service 
   875ms packagekit.service 
   855ms gdm.service 
   514ms firewalld.service 
   438ms rsyslog.service
   436ms udisks2.service 
   398ms sshd.service 
   360ms boot.mount 
   336ms polkit.service 
   321ms accounts-daemon.service

When working with systemd, it is important to understand the concept of unitsUnits are the resources systemd knows how to interpret. Units are categorized into 12 types as follows −

  • .service
  • .socket
  • .device
  • .mount
  • .automount
  • .swap
  • .target
  • .path
  • .timer
  • .snapshot
  • .slice
  • .scope

For the most part, we will be working with .service as unit targets. It is recommended to do further research on the other types. As only .service units will apply to starting and stopping systemd services.

Each unit is defined in a file located in either −

  • /lib/systemd/system − base unit files

  • /etc/systemd/system − modified unit files started at run-time

Manage Services with systemctl

To work with systemd, we will need to get very familiar with the systemctl command. Following are the most common command line switches for systemctl.

SwitchAction
-tComma separated value of unit types such as service or socket
-aShows all loaded units
--stateShows all units in a defined state, either: load, sub, active, inactive, etc..
-HExecutes operation remotely. Specify Host name or host and user separated by @.

Basic systemctl Usage

systemctl [operation]
example: systemctl --state [servicename.service]

For a quick look at all the services running on our box.

[root@localhost rdc]# systemctl -t service 
UNIT                       LOAD     ACTIVE      SUB     DESCRIPTION

abrt-ccpp.service          loaded   active   exited     Install ABRT coredump   hook 
abrt-oops.service          loaded   active   running    ABRT kernel log watcher 
abrt-xorg.service          loaded   active   running    ABRT Xorg log watcher 
abrtd.service              loaded   active   running    ABRT Automated Bug  Reporting Tool 
accounts-daemon.service    loaded   active   running    Accounts Service 
alsa-state.service         loaded   active   running    Manage Sound Card State (restore and store) 
atd.service                loaded   active   running    Job spooling tools 
auditd.service             loaded   active   running    Security Auditing Service 
avahi-daemon.service       loaded   active   running    Avahi mDNS/DNS-SD Stack 
blk-availability.service   loaded   active   exited     Availability of block devices 
bluetooth.service          loaded   active   running    Bluetooth service 
chronyd.service            loaded   active   running    NTP client/server

Stopping a Service

Let's first, stop the bluetooth service.

[root@localhost]# systemctl stop bluetooth

[root@localhost]# systemctl --all -t service | grep bluetooth      
bluetooth.service   loaded    inactive dead    Bluetooth service

[root@localhost]#

As we can see, the bluetooth service is now inactive.

To start the bluetooth service again.

[root@localhost]# systemctl start bluetooth

[root@localhost]# systemctl --all -t service | grep bluetooth 
bluetooth.service  loaded    active   running Bluetooth     service

[root@localhost]#

Note − We didn't specify bluetooth.service, since the .service is implied. It is a good practice to think of the unit type appending the service we are dealing with. So, from here on, we will use the .service extension to clarify we are working on service unit operations.

The primary actions that can be performed on a service are −

StartStarts the service
StopStops a service
ReloadReloads the active configuration of a service w/o stopping it (like kill -HUP in system v init)
RestartStarts, then stops a service
EnableStarts a service at boot time
DisableStops a service from automatically starting at run time

The above actions are primarily used in the following scenarios −

StartTo bring a service up that has been put in the stopped state.
StopTo temporarily shut down a service (for example when a service must be stopped to access files locked by the service, as when upgrading the service)
ReloadWhen a configuration file has been edited and we want to apply the new changes while not stopping the service.
RestartIn the same scenario as reload, but the service does not support reload.
EnableWhen we want a disabled service to run at boot time.
DisableUsed primarily when there is a need to stop a service, but it starts on boot.

To check the status of a service −

[root@localhost]# systemctl status network.service 
network.service - LSB: Bring up/down networking 
Loaded: loaded (/etc/rc.d/init.d/network; bad; vendor preset: disabled) 
Active: active (exited) since Sat 2017-01-14 04:43:48 EST; 1min 31s ago 
Docs: man:systemd-sysv-generator(8)

Process: 923 ExecStart = /etc/rc.d/init.d/network start (code=exited, status = 0/SUCCESS)

localhost.localdomain systemd[1]: Starting LSB: Bring up/down networking... 
localhost.localdomain network[923]: Bringing up loopback interface:  [  OK  ] 
localhost.localdomain systemd[1]: Started LSB: Bring up/down networking.

[root@localhost]#

Show us the current status of the networking service. If we want to see all the services related to networking, we can use −

[root@localhost]# systemctl --all -t service | grep -i network 
network.service                       loaded    active    exited    LSB: Bring up/ 
NetworkManager-wait-online.service    loaded    active    exited    Network Manager  
NetworkManager.service                loaded    active    running   Network Manager 
ntpd.service                          loaded    inactive  dead      Network Time  
rhel-import-state.service             loaded    active    exited    Import network      

[root@localhost]#

For those familiar with the sysinit method of managing services, it is important to make the transition to systemdsystemd is the new way starting and stopping daemon services in Linux.