Automated Process monitoring during high server load

Automated Process monitoring during high server load

root@myServer [/root]# cat load-process-monitor.sh

#!/bin/bash

# Define Variables

DT=`date +”%A %b %e %r”`

HOSTNAME=`hostname`

# Create dir to store data

mkdir -p /opt/loadcheck/

# Retrieve the load average of the past 1 minute

LAVG=`uptime | awk {‘print $10}’ | cut -d. -f1`

LCURRENT=`uptime | awk {‘print $10,$11,$12}’`

# Define Threshold. This value will be compared with the current load average. Set the value as per your wish.

LIMIT=-1

# Compare the current load average with Threshold and email the server administrator if threshold is greater.

if [ $LAVG -gt $LIMIT ]

then

#Save the current running processes in a file

/bin/ps -auxf >> /opt/ps_output

echo “Current Time :: $DT. >> /tmp/loadmon.txt

echo “Current Load Average :: $LCURRENT. >> /tmp/loadmon.txt

echo “current processes list attached with the email 1 instance. >> /tmp/loadmon.txt

echo “Also check loadps.txt :: loadtop.txt :: netstat_all.txt :: netstat_port80.txt inside /opt/loadcheck/ on the server” >> /tmp/loadmon.txt

# Send email to support

/usr/bin/mutt -s “Server Load ALERT!!! High 1 minute load average on ‘$HOSTNAME’” -a /opt/ps_output support@somedomain.com > /opt/ps_output

echo “Current Time :: $DT” >> /tmp/loadmon.txt

echo “Current Load Average :: $LCURRENT” >> /tmp/loadmon.txt

echo “current processes list attached with the email 1 instance” >> /tmp/loadmon.txt

echo “Also check loadps.txt :: loadtop.txt :: netstat_all.txt :: netstat_port80.txt inside /opt/loadcheck/ on the server” >> /tmp/loadmon.txt

# Send email to support

/usr/bin/mutt -s ” Server Load ALERT ::: High 1 minute load average on ‘$HOSTNAME’ ” -a /opt/ps_output support@integrityhost.com > /opt/loadcheck/loadps.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt

/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt

/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt

/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt

/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt

/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt

/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt

/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt

/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt

/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt

/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt

/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt

/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_port80.txt

/bin/ps -auxf >> /opt/loadcheck/loadps.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadps.txt

/bin/top -c -n1 >> /opt/loadcheck/loadtop.txt

echo “#########################################################################################################################” >> /opt/loadcheck/loadtop.txt

/bin/netstat -ntu | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_all.txt

echo “#########################################################################################################################” >> /opt/loadcheck/netstat_all.txt

/bin/netstat -alntp | grep :80 | awk ‘{print $5}’ | cut -d: -f1 | sort | uniq -c | sort -n >> /opt/loadcheck/netstat_port80.txt

echo ” ######################################################################################################################### ” >> /opt/loadcheck/netstat_port80.txt

fi

# Remove residue logs

/bin/rm -f /tmp/loadmon.txt

/bin/rm -f /opt/ps_output

root@myServer [/root]#

Add a cron to run load-process-monitor.sh ( * * * * * /bin/sh /path-to/load-process-monitor.sh ) every min and when server load goes beyond 4, it will send you email and log some important details, which can help to some extent to find some pointers to load issue from process and netstat listings.

Email Spamming Audit scripts on Cpanel

Some scripts that are used to send out spam emails :
[root@support ~]#grep “cwd=” /var/log/exim_mainlog|awk ‘{for(i=1;i<=10;i++){print $i}}’|sort|uniq -c|grep cwd|sort -n
Eximstats
[root@support ~]#eximstats -t5 /var/log/exim_mainlog > teststats
Script to know the mail count by various accounts
[root@support ~]#grep “cwd=” /var/log/exim_mainlog|awk ‘{for(i=1;i<=10;i++){print $i}}’|sort|uniq -c|grep cwd|sort -n
The number of mails by a domain
[root@support ~]#exigrep @domain.com /var/log/exim_mainlog|grep 2009-04-17|grep Completed|wc -l
[root@support ~]#ps -C exim -fH ewww |grep home, it shows the mails going from the server.
It shows from which user’s home the mail is going, so that you can easily trace it and block it if needed.
[root@support ~]#eximstats -ne -nr /var/log/exim_mainlog
It shows top 50 domains using mail server with options.
[root@support ~]#exim -bp | exiqsumm
It shows the main domains receiving and sending mails on the server.
[root@support ~]# netstat -plan|grep :25|awk {‘print $5′}|cut -d: -f 1|sort|uniq -c|sort -nk 1
It shows the IPs which are connected to server through port number 25. It one particular Ip is using more than 10 connection you can block it in the server firewall.
In order to find “nobody” spamming, issue the following command
[root@support ~]#ps -C exim -fH ewww|awk ‘{for(i=1;i<=40;i++){print $i}}’|sort|uniq -c|grep PWD|sort -n
It will give as like below result :-
Example :
6 PWD=/
347 PWD=/home/sample/public_html/test
Count the PWD and if it is a large value check the files in the directory listed in PWD
(Ignore if it is / or /var/spool/mail /var/spool/exim)
The above command is valid only if the spamming is currently in progress. If the spamming has happened some hours before, use the following command.
Command :
[root@support ~]#grep “cwd=” /var/log/exim_mainlog|awk ‘{for(i=1;i<=10;i++){print $i}}’|sort|uniq -c|grep cwd|sort -n
This will result in something like :
96 cwd=/root
4583 cwd=/home/sample/public_html/test
Count the cwd and if it is a large value check the files in the directory listed in cwd
(Ignore if it is / or /var/spool/mail /var/spool/exim)
Pass the below mentioned command at your command prompt to find the domain which is being used by spammers.
[root@support ~]#exim -bpr | exiqsumm -c | head
[root@support ~]#exiqgrep -ir <domain> | xargs -n1 exim -Mrm
That should remove any e-mail that is in the queue that is waiting to be delivered to POP accounts at <domain>.
Take below Precautions:
I)Turn on the SMTP tweak. It will block the users to bypass the mail server for sending out spam.
II)Turn on blacklisting ability in whm.
III)Use spamassassin to stop receiving spam mails.

Virtuzzo All commands

List vps’es on a node vzlist -a
Create a vps vzctl create 5001 –ostemplate centos-5-i386-default
When a vps is created the files are stored in main node in the location /vz/private/5001. The configuration file for vps 5001 will lie at /etc/vz/conf/5001.conf. The /etc/vz/conf is a link to /etc/sysconfig/vz-scripts/
[root@server ~]# ls -al /etc/vz/conf
lrwxrwxrwx 1 root root 23 Apr 10 2009 /etc/vz/conf -> ../sysconfig/vz-scripts
Destroy a vps
vzctl destroy 5001
Assigning host name to vps
vzctl set 5001 –hostname server.hostname.com –save
Add IP to vps
vzctl set 5001 –ipadd 11.22.33.44 –save
Create vps with hostnmae and IP set
vzctl create 5001 –ostemplate centos-5-i386-default –ipadd 11.22.33.44 –hostname server.hostname.com
Dump a vps to take backup. This will create a backup 5001.tgz in the path /backup/
vzdump –compress –dumpdir /backup/5001.tgz 5001
Restore a vps from backup
vzdump –restore /backup/vzdump-5001.tgz 5001
Change vps parameters, eg privvmpages
vzctl set 5001 –privvmpages 431072:441072 –save
Execute a command inside a vps from Node, Eg: to execute “free -m” inside the vps 5001
vzctl exec 5001 free -m
To set vps to be booted when the node boots up
vzctl set 5001 –onboot=yes –save
To set root password for a vps from Node
vzctl set 5001 –userpasswd root:newpassword –save
Install a package on vps. To install “yum” on vps 5001.
vzyum 5001 install yum
To set capability to change eg: MAC addresses
vzctl set 5001 –capability net_admin:on –save
Basic OpenVZ commands:

To set up a VPS from the default CentOS 5 template you have to use the below command:

vzctl create 101 –ostemplate centos-5-x86 –config basic

The 101 must be a uniqe ID – each virtual machine must have its own unique ID. If you want to have the vm started at boot, run
vzctl set 101 –onboot yes –save

To set a hostname and IP address for the vm, run:

vzctl set 101 –hostname test.example.com –save
vzctl set 101 –ipadd 192.168.1.101 –save
To start the vm
vzctl start 101

To set a root password for the vm, execute

vzctl exec 101 passwd

To login into vm

vzctl enter 101

To leave the vm’s console, type

exit

To stop a vm, run

vzctl stop 101

To restart a vm, run

vzctl restart 101

To delete a vm from the hard drive (it must be stopped before you can do this), run

vzctl destroy 101

To get a list of your vms and their statuses, run

vzlist -a

To find out about the resources allocated to a vm, run

vzctl exec 101 cat /proc/user_beancounters

Command to check the load of all VPS from main server node.

vzlist -o veid,laverage

It will show the below output:

CTID LAVERAGE

101 3.02/3.32/3.31
102 5.35/3.30/3.38
103 03.34/3.03/3.50
104 0.93/0.61/0.66
105 1.67/1.67/1.85
106 0.02/0.01/0.00

vzctl status VEID
To view the status of the particular VPS

vzctl stop VEID –fast
To stop the VPS quickly and forcefully

vzctl enter VEID
To enter in a particular VPS

vzctl set VEID –hostname vps.domain.com –save
To set the Hostname of a VPS

vzctl set VEID –ipadd 1.2.3.4 –save
To add a new IP to the hosting VPS

vzctl set VEID –ipdel 1.2.3.4 –save
To delete the IP from VPS

vzctl set VEID –userpasswd root:new_password –save
To reset root password of a VPS

vzctl set VEID –nameserver 1.2.3.4 –save
To add the nameserver IP’s to the VPS

vzctl exec VEID command
To run any command on a VPS from Node

vzyum VEID install package_name
To install any package/Software on a VPS from Node

**VEID refers to the ID of the Particular VPS**
vz -> recommends ext3 file system
PIM -> IP:4643
PMC -> mangaing h/w node
PPP -> for managing a single container
All the above things need a license from parallel
====================

Templates

====================
OS template -> vzpkg (cache needs to created before creating the container)
EZ template -> points to the repository that contains packages that constitute the template
Application template -> mysql template
====================

To create a container

====================
vzctl create
the container ID > 100
32 -bit integer
ID – 0 -> h/w node
ID -1 -> service container

/etc/vz/conf -> for the sample file names (only use the main part of the file name)

vzctl set – configure the container
vzctl exec – run anything on the container
vzctl status/vzlist 101

vzpkg – for adding application templates on h/w node (/vz/templates)
- the same command can be used for installing/removing the template into a the container.

Virtuozzo – User’s guide

OS virtualization – (1-2% of is spent on the virtualization s/w)
VZFS – allows sharing of common files among containers
Hardware node & Containers

/etc/vz/vz.conf
/etc/vz/conf/
/etc/vzbackup.conf

Standard migration
vzmigrate

Zero downtime migration
vzmigrate –online –require-realtime my_node.com 101

Move the container within the h/w node
vzmlocal

Backup & restore
vzabackup/vzarestore

Reinstall container
vzctl recover -> restores the VZFS symlinks
vzctl reinstall -> creates new private area for the container, copies the old private area to /old dir
- vps.configure, vps.reinstall

Delete the container
vzctl destroy

Disabling the container
vzctl set

Suspending the container
vzctl suspend

Setting up resources
disk quotas – first level(how much files or disk space the container can use) and second level (quotaugidlimit > no of entries in /etc/passwd or group files)
Container must be restarted to come this changes into effect

vzquota – to check the status of the quota info

Cleaning up containers – to move identical files from contrainers to /vz/template/vc folder
vzcache

Linking container files with application templates
vzpkg link
vzpkglink
vzpkgls

Managing CPU – cpu share and the number of processors the node can make use of.
vzcpucheck

Network traffic
/etc/vz/conf/networks_classes – after any changes, you need to service vz accrestart
class 0 – no accounting will be performed
class 1 – defined by containers to match any IP address
vznetstat

Bandwidth management (outgoing traffic)
service vz shaperon, shaperrestart, shaperoff
BANDWIDTH -> kilobites per second
TOTALRATE -> ::

For container
RATE -> ::
RATEBOUND -> to specify if the bandwidth is limited to the RATEBOUND
====================

Monitoring tools

====================
vzstat
vzps
vzpid
vztop
vzsetxinetd

Virtuozzo networks
vznetcfg if list

vznetcfg vlan add eth0 5
vznetcfg vlan del eth0.5

vznetcfg net addif vznetwork1 eth0
vznetcfg net delif eth0

vznetcfg net new vznetwork1
vznetcfg net del vznetwork1
vznetcfg net list

venet0 –?

License
vzlicload
vzlicupdate
vzlicview

Keeping system up2date
The h/w node can be updated using the normal update procedures without affecting virtuozzo
vzup2date – /etc/sysconfig/vzup2date/vzup2date.conf
vzup2date -m batch install –core
vzup2date -t -m batch install –all-os

Updating the containers
vzpkg update 101 redhat-el5-x86
vzpkg update 101
vzpkg update cache fedora-core-8-x86

Loading iptables modules
/etc/sysconfig/iptables-config
/etc/vz/vz.conf
/etc/vz/conf/.conf

VZFS-v2

Virtuozzo – Templates management guide

/vz/template
/vz/private/
/vz/root/

vzpkg – for managing VZ template
vzpkgls – for listing standard template

vzpkg install -p 110 yum

Virtuozzo – Managing UBC Resources

vzcfgvalidate

VE0CPUUNITS
SLM parameters

i-node setups

ext3 – 1 inode per every 4K
# vzlist -a

To list all the running VPS in the node

# vzlist

To display the templates present in the server

# vzpkgls

Creating a VPS To create a VPS with VEID 101 and ostemplate fedora-core-4 with vps.basic configuration

# vzctl create 101 –ostemplate fedora-core-4 -.config vps.basic

Deleting a VPS To destroy a VPS with VEID 101

# vzctl destroy 101

Configuring VPS (The changes are saved in /etc/vz/conf/.conf) To automatically boot when a node is up

# vzctl set 101 –onboot yes –save

To set hostname

# vzctl set 101 –hostname test101.my.org –save

To add an IP address

# vzctl set 101 –ipadd 10.0.186.1 –save

To delete an IP address

# vzctl set 101 –ipdel 10.0.186.1 –save

To set the name servers

# vzctl set 101 –nameserver 192.168.1.165 –save

To set the root password of VPS 101

# vzctl set 101 –userpasswd root:password

To set shortname for VPS

# vzctl set 101 –name test101 –save

Start/Stop/Restart VPS To start a VPS

# vzctl start 101

To start a disabled VPS

# vzctl start 101 –force

To stop a VPS

vzctl stop 101

To restart a VPS

# vzctl restart 101

To know the status of a VPS

# vzctl status 101

To get the details of the VPS like VEID, ClassID, number of processes inside each VPS and the IP addresses of VPS

# cat /proc/vz/veinfo

To enter into a VPS 101

# vzctl enter 101

To execute a command in VPS 101

# vzctl exec 101 command (replace command with the command you need to execute)
# vzctl exec 101 df -h

Managing Disk Quotas To assign disk quotas – First limit is soft limit, second limit is hard limit

# vzctl set 101 –diskspace 10485760 –save ==>> for setting 10GB
OR
# vzctl set 101 –diskspace 1048576 –save ==>> for setting 1GB

To assign disk inodes

# vzctl set 101 –diskinodes 90000:91000 –save

To check the disk quota of a VPS

# vzquota stat 101 -t

Managing CPU quota To display the available CPU power

# vzcpucheck

To set the number of CPUs available to a VPS

# vzctl set 101 –cpus 2 –save

To set the minimum and maximum CPU limits

# vzctl set 101 –cpuunits nnnn –cpulimit nn –save
(cpuunits is a an absolute number (fraction of power of the node) and cpulimit is taken as percentage)

Managing memory quota To display memory usage

# vzmemcheck -v

To set kmem

# vzctl set 101 –kmemsize 2211840:2359296 –save

To set privvmpages

# vzctl set 101 –privvmpages 2G:2G –save

Other Commands To copy/clone a VPS

# vzmlocal -C :

To disable a VPS

# vzctl set 101 –disabled yes

To enable a VPS

# vzctl set 101 –disabled no

To suspend a VPS

# vzctl suspend 101

To resume a VPS

# vzctl resume 101

To run yum update on a VPS

# vzyum 101 -y update

To install a package using yum on VPS

# vzyum 101 -y install package

To install a package using rpm on VPS

# vzrpm 101 -ivh package
VZ -> Vir­tuozzo –> rec­om­mends ext3 file sys­tem
PIM -> Par­al­lels Infra­struc­ture Man­ager –> IP:4643
PMC -> Par­al­lels Man­age­ment Con­sole –> man­ag­ing hard­ware node
PPP -> Par­al­lels Power Panel –> for man­ag­ing a sin­gle con­tainer
All the above things need a license from parallel

Tem­plates

OS tem­plate -> vzpkg (cache needs to cre­ated before cre­at­ing the con­tainer)
EZ tem­plate -> points to the repos­i­tory that con­tains pack­ages that con­sti­tute the tem­plate
Appli­ca­tion tem­plate -> mysql tem­plate

Vir­tuozzo — User’s guide

OS vir­tu­al­iza­tion — (1–2% of is spent on the vir­tu­al­iza­tion s/w) VZFS — allows shar­ing of com­mon files among con­tain­ers Hard­ware node & Containers

/etc/vz/vz.conf
/etc/vz/conf/
/etc/vzbackup.conf

Delete the container

# vzctl destroy

Dis­abling the container

# vzctl set

Sus­pend­ing the container

# vzctl suspend

To check the sta­tus of the quota info

# vzquota

Man­ag­ing CPU — cpu share and the num­ber of proces­sors the node can make use of

# vzcpucheck

Mon­i­tor­ing tools

# vzs­tat
# vzps
# vzpid
# vztop
# vzsetx­inetd

Vir­tuozzo networks

# vznetcfg if list
# vznetcfg vlan add eth0 5
# vznetcfg vlan del eth0.5
# vznetcfg net addif vznetwork1 eth0
# vznetcfg net delif eth0
# vznetcfg net new vznetwork1
# vznetcfg net del vznetwork1
# vznetcfg net list

Keep­ing sys­tem up2date : The h/w node can be updated using the nor­mal update pro­ce­dures with­out affect­ing virtuozzo

# vzup2date — /etc/sysconfig/vzup2date/vzup2date.conf
# vzup2date –m batch install –core
# vzup2date –t –m batch install –all-os

Updat­ing the con­tain­ers

# vzpkg update 101 redhat-el5-x86
# vzpkg update 101
# vzpkg update cache fedora-core-8-x86

1) vzlist -a : Shows list of all the VPS’s hosted on the Node.
2) vzctl start VPS_ID: To start the VPS.
3) vzctl stop VPS_ID : To stop (Shut Down) the VPS.
4) vzctl status VPS_ID : To view the status of the particular VPS.
5) vzctl stop VPS_ID –fast : to stop the VPS quickly and forcefully
6) vzctl enter VPS_ID : To enter in a particular VPS

Configuration Commands

1) vzctl set VPS_ID –hostname vps.domain.com –save: To set the Hostname of a VPS.

2) vzctl set VPS_ID –ipadd 1.2.3.4 –save : To add a new IP to the hosting VPS

3) vzctl set VPS_ID –ipdel 1.2.3.4 –save : To delete the IP from VPS.

4) vzctl set VPS_ID –userpasswd root:new_password –save : to reset root password of a VPS.

5) vzctl set VPS_ID –nameserver 1.2.3.4 –save : To add the nameserver IP’s to the VPS

6) vzctl exec VPS_ID command : To run any command on a VPS from Node.

6) vzyum VPS_ID install package_name : To install any package/Software on a VPS from Node.

VPS_ID refers to the ID of the Particular VPS.

Unable to back up large container: Acronis Error: Module=0 Code=0, Tag=0

Resolution

There are two ways to resolve the issue:

• Bind-mount the /tmp partition onto a larger partition:

  # mkdir /vz/temp
  # mount --bind /vz/temp /tmp
  

  In order to preserve such behavior after a server reboot, add the corresponding record to /etc/fstab, e.g.:

  # echo "/vz/temp /tmp none bind 0 0" >> /etc/fstab
  

• Change temporary storage location in PVA Agent backup plugin configuration (as of PVA 4.6.4 update 4 or 6.0):
  1. Edit /opt/pva/agent/plugins/VZABackupLibClient/VZABackupLibClient.voc
  2. Append parameter/load/params section for vzlin_backup_serializer11 with envp variables so the resulting section will look in a following way:

         <!-- vzlpl stuff -->
         <parameter>
                 <id>vzlin_backup_serializer11</id>
                 <category>plugins</category>
                 <short>Virtuozzo Backup plugin</short>
                 <load>
                         <params xsi:type="plugin_params_externType">
                                 <type>extern</type>
                                 <path>vzlpl</path>
                                 <arguments>
                                         <argv>VZABackupLibClient/VZABackupLibClient.so</argv>
                                         <argv>vzlin_backup_serializer11</argv>
                                 </arguments>
                                 <!-- Environment variables-->
                                 <envp>
                                         <item>
                                                 <name>ACRONIS_TMP_PATH</name>
                                                 <value>/vz/tmp</value>
                                         </item>
                                 </envp>
                         </params>
                 </load>
     

  3. Restart PVA Agent

18 COMMANDS TO MONITOR NETWORK BANDWIDTH ON LINUX SERVER

The tools have different mechanisms of generating the traffic report. Some of the tools like nload read the "/proc/net/dev" file to get traffic stats, whereas some tools use the pcap library to capture all packets and then calculate the total size to estimate the traffic load.

 1. Nload

Nload is a commandline tool that allows users to monitor the incoming and outgoing traffic separately. It also draws out a graph to indicate the same, the scale of which can be adjusted. Easy and simple to use, and does not support many options.

So if you just need to take a quick look at the total bandwidth usage without details of individual processes, then nload will be handy.

$ nload
nload command linux

Installing Nload - Fedora and Ubuntu have got it in the default repos. CentOS users need to get nload from Epel repositories.

# fedora or centos
$ yum install nload -y

# ubuntu/debian
$ sudo apt-get install nload
2. iftop

Iftop measures the data flowing through individual socket connections, and it works in a manner that is different from Nload. Iftop uses the pcap library to capture the packets moving in and out of the network adapter, and then sums up the size and count to find the total bandwidth under use.

Although iftop reports the bandwidth used by individual connections, it cannot report the process name/id involved in the particular socket connection. But being based on the pcap library, iftop is able to filter the traffic and report bandwidth usage over selected host connections as specified by the filter.

$ sudo iftop -n
The n option prevents iftop from resolving ip addresses to hostname, which causes additional network traffic of its own.

iftop command linux

Install iftop - Ubuntu/Debian/Fedora users get it from default repos. CentOS users get it from Epel.

# fedora or centos
yum install iftop -y

# ubuntu or debian
$ sudo apt-get install iftop
3. iptraf

Iptraf is an interactive and colorful IP Lan monitor. It shows individual connections and the amount of data flowing between the hosts. Here is a screenshot

$ sudo iptraf
iptraf linux command

Install iptraf

# Centos (base repo)
$ yum install iptraf

# fedora or centos (with epel)
$ yum install iptraf-ng -y

# ubuntu or debian
$ sudo apt-get install iptraf iptraf-ng
4. nethogs

Nethogs is a small 'net top' tool that shows the bandwidth used by individual processes and sorts the list putting the most intensive processes on top. In the event of a sudden bandwidth spike, quickly open nethogs and find the process responsible. Nethogs reports the PID, user and the path of the program.

$ sudo nethogs
nethogs command linux

Install Nethogs - Ubuntu, Debian, Fedora users get from default repos. CentOS users need Epel

# ubuntu or debian (default repos)
$ sudo apt-get install nethogs

# fedora or centos (from epel)
$ sudo yum install nethogs -y
5. bmon

Bmon (Bandwidth Monitor) is a tool similar to nload that shows the traffic load over all the network interfaces on the system. The output also consists of a graph and a section with packet level details.

bmon linux network monitor

Install Bmon - Ubuntu, Debian and Fedora users can install from default repos. CentOS users need to setup repoforge, since its not available in Epel.

# ubuntu or debian
$ sudo apt-get install bmon

# fedora or centos (from repoforge)
$ sudo yum install bmon
Bmon supports many options and is capable of producing reports in html format. Check the man page for more information

6. slurm

Slurm is 'yet' another network load monitor that shows device statistics along with an ascii graph. It supports 3 different styles of graphs each of which can be activated using the c, s and l keys. Simple in features, slurm does not display any further details about the network load.

$ slurm -s -i eth0
slurm command linux

Install slurm

# debian or ubuntu
$ sudo apt-get install slurm

# fedora or centos
$ sudo yum install slurm -y
7. tcptrack

Tcptrack is similar to iftop, and uses the pcap library to capture packets and calculate various statistics like the bandwidth used in each connection. It also supports the standard pcap filters that can be used to monitor specific connections.

tcptrack command linux

Install tcptrack - Ubuntu, Debian and Fedora have it in default repos. CentOS users need to get it from RepoForge as it is not available in Epel either.

# ubuntu, debian
$ sudo apt-get install tcptrack

# fedora, centos (from repoforge repository)
$ sudo yum install tcptrack
8. Vnstat

Vnstat is bit different from most of the other tools. It actually runs a background service/daemon and keeps recording the size of data transfer all the time. Next it can be used to generate a report of the history of network usage.

$ service vnstat status
 * vnStat daemon is running
Running vnstat without any options would simply show the total amount of data transfer that took place since the date the daemon is running.

$ vnstat
Database updated: Mon Mar 17 15:26:59 2014

   eth0 since 06/12/13

          rx:  135.14 GiB      tx:  35.76 GiB      total:  170.90 GiB

   monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Feb '14      8.19 GiB |    2.08 GiB |   10.27 GiB |   35.60 kbit/s
       Mar '14      4.98 GiB |    1.52 GiB |    6.50 GiB |   37.93 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated      9.28 GiB |    2.83 GiB |   12.11 GiB |

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
     yesterday    236.11 MiB |   98.61 MiB |  334.72 MiB |   31.74 kbit/s
         today    128.55 MiB |   41.00 MiB |  169.56 MiB |   24.97 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated       199 MiB |      63 MiB |     262 MiB |
To monitor the bandwidth usage in realtime, use the '-l' option (live mode). It would then show the total bandwidth used by incoming and outgoing data, but in a very precise manner without any internal details about host connections or processes.

$ vnstat -l -i eth0
Monitoring eth0...    (press CTRL-C to stop)

   rx:       12 kbit/s    10 p/s          tx:       12 kbit/s    11 p/s
Vnstat is more like a tool to get historic reports of how much bandwidth is used everyday or over the past month. It is not strictly a tool for monitoring the network in real time.

Vnstat supports many options, details about which can be found in the man page.

Install vnstat

# ubuntu or debian
$ sudo apt-get install vnstat

# fedora or centos (from epel)
$ sudo yum install vnstat
9. bwm-ng

Bwm-ng (Bandwidth Monitor Next Generation) is another very simple real time network load monitor that reports a summary of the speed at which data is being transferred in and out of all available network interfaces on the system.

$ bwm-ng
bwm-ng v0.6 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  /         iface                   Rx                   Tx                T
ot==========================================================================
==           eth0:           0.53 KB/s            1.31 KB/s            1.84
KB             lo:           0.00 KB/s            0.00 KB/s            0.00
KB--------------------------------------------------------------------------
--          total:           0.53 KB/s            1.31 KB/s            1.84
KB/s
If the console size is sufficiently large, bwm-ng can also draw bar graphs for the traffic using the curses2 output mode.

$ bwm-ng -o curses2
Install Bwm-NG - On CentOS bwm-ng can be installed from Epel.

# ubuntu or debian
$ sudo apt-get install bwm-ng

# fedora or centos (from epel)
$ sudo apt-get install bwm-ng
10. cbm - Color Bandwidth Meter

A tiny little simple bandwidth monitor that displays the traffic volume through network interfaces. No further options, just the traffic stats are display and updated in realtime.

cbm linux network monitor

$ sudo apt-get install cbm
11. speedometer

Another small and simple tool that just draws out good looking graphs of incoming and outgoing traffic through a given interface.

$ speedometer -r eth0 -t eth0
speedometer linux network monitor

Install speedometer

# ubuntu or debian users
$ sudo apt-get install speedometer
12. Pktstat

Pktstat displays all the active connections in real time, and the speed at which data is being transferred through them. It also displays the type of the connection, i.e. tcp or udp and also details about http requests if involved.

$ sudo pktstat -i eth0 -nt
pktstat linux network monitor

$ sudo apt-get install pktstat
13. Netwatch

Netwatch is part of the netdiag collection of tools, and it too displays the connections between local host and other remote hosts, and the speed at which data is transferring on each connection.

$ sudo netwatch -e eth0 -nt
netwatch linux network monitor

$ sudo apt-get install netdiag
14. Trafshow

Like netwatch and pktstat, trafshow reports the current active connections, their protocol and the data transfer speed on each connection. It can filter out connections using pcap type filters.

Monitor only tcp connections

$ sudo trafshow -i eth0 tcp
trafshow linux network monitor

$ sudo apt-get install netdiag
15. Netload

The netload command just displays a small report on the current traffic load, and the total number of bytes transferred since the program start. No more features are there. Its part of the netdiag.

$ netload eth0
netload linux network monitor

$ sudo apt-get install netdiag
16. ifstat

The ifstat reports the network bandwidth in a batch style mode. The output is in a format that is easy to log and parse using other programs or utilities.

$ ifstat -t -i eth0 0.5
  Time           eth0      
HH:MM:SS   KB/s in  KB/s out
09:59:21      2.62      2.80
09:59:22      2.10      1.78
09:59:22      2.67      1.84
09:59:23      2.06      1.98
09:59:23      1.73      1.79
Install ifstat - Ubuntu, Debian and Fedora users have it in the default repos. CentOS users need to get it from Repoforge, since its not there in Epel.

# ubuntu, debian
$ sudo apt-get install ifstat

# fedora, centos (Repoforge)
$ sudo yum install ifstat
17. dstat

Dstat is a versatile tool (written in python) that can monitor different system statistics and report them in a batch style mode or log the data to a csv or similar file. This example shows how to use dstat to report network bandwidth

$ dstat -nt
-net/total- ----system----
 recv  send|     time    
   0     0 |23-03 10:27:13
1738B 1810B|23-03 10:27:14
2937B 2610B|23-03 10:27:15
2319B 2232B|23-03 10:27:16
2738B 2508B|23-03 10:27:17
Install dstat


18. collectl

Collectl reports system statistics in a style that is similar to dstat, and like dstat it is gathers statistics about various different system resources like cpu, memory, network etc. Over here is a simple example of how to use it to report network usage/bandwidth.

$ collectl -sn -oT -i0.5
waiting for 0.5 second sample...
#         <----------Network---------->
#Time       KBIn  PktIn  KBOut  PktOut
10:32:01      40     58     43      66
10:32:01      27     58      3      32
10:32:02       3     28      9      44
10:32:02       5     42     96      96
10:32:03       5     48      3      28
Install Collectl

# Ubuntu/Debian users
$ sudo apt-get install collectl

#Fedora
$ sudo yum install collectl

Xenserver VM force reboot not working?

xe vm-shutdown force=true vm=
xe vm-reset-powerstate force=true vm=



Log in to the host
Get the UUID of the VM that is hung
Run list_domains to get the domain ID of the VM
Run a destroy_domain on the domain ID
Now do a force reboot of the VM
Example CLI steps

xe vm-list name-label=vm001
list_domains|grep 6512c54b-4f3f-6737-3bc7-68b377f15a6a
/opt/xensource/debug/destroy_domain -domid 450
xe vm-reboot force=true vm=vm001
The VM should now be up and running again.

Xenserver 6.1 CLI commands specific for Storage Migration

Check SR uuid
xe sr-list name-label=mystorage
List disks for vm
xe vm-disk-list vm=myvmname
List specific VDI
xe vdi-list name-label=myvdiname
offline migrate disk
xe vdi-copy sr-uuid=mystorageuuid uuid=myvdiuuid
live migrate disk
xe vdi-pool-migrate sr-uuid=mystorageuuid uuid=myvdiuuid
check copy task
xe task-list name-label=VDI.copy
check migrate task
xe task-list name-label=VDI.pool_migrate