10 useful tricks to use Sed command for file Manipulation

Commands and Tools to get network statistics                                                  Netstat Command

If you are working or using unix operating system, then you must be proficient in sed language. Sed command in unix is very helpful for manipulation of file. While writing this article, I am considering this you are aware about the sed’s little basics. Sed is a great utility which solves complex tasks by few lines of code.

Sed stands for Stream Editor. It is very powerful command to file manipulatation, filter lines while reading file and transform text. Sed works both the way i.e. Sed can read file and do operations on it or Sed can take inputs from pipe.  If you are new learner then you will face difficulty to understand the sed command but once you understood then It will be easier to use as you need.

We are going to use below file in all 10 examples.

root@hello:~/test# cat salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000

The above salary database contains the columns as

•           Employee ID
•        Name
•          Designation
•          Salary

Basic Syntax of sed command:

sed [options] {sed-commands} {input-file} sed [options] -e {sed-command-1} -e {sed-command-2}{input-file}

Sed reads one line at a time from input file and executes sed - command on that particular line. If we want to execute multiple commands in a single line using several –e arguments, you can split them into multiple lines using a backslash as below:

sed -n \ -e '/^mysql/ p' \ -e '/^other/ p' \ /etc/groups

10 special tricks of Sed as follows :

1.       Print the lines of input file after specific interval i.e. print 1^st, 3^rd, 5^th line of file. Operator   ~ used in an address range. Its special meaning is to skip lines between commands. Below command you can use to print the odd numbered lines or to print the even numbered lines.

root@hello:~/test# sed -n '1~2 p' salary.txt 1001,Ajay,Manager,25000 302,Atul,CEO,26000 1231,Kalyan,Human Resource,20000

2.       Print the lines matching with keyword.

root@hello:~/test# sed -n '/Atul/p' salary.txt 302,Atul,CEO,26000

3.       Print the lines matching with keyword and 2 lines immediately after that. i.e. If matched keyword found then print 3 lines including matched keyword line and 2 lines immediately after that line.

root@hello:~/test# sed -n '/Atul/, +2p' salary.txt 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000

You can use above command for deleting lines in output by using “d” instead of “p” option. i.e. if you want to delete lines starting from 1^st match of “Atul” keyword till 4^th line of file then you can use it as

sed '/Atul/,4 d' salary.txt

4.       Write the content of salary.txt file to new_salary.txt file. i.e. simple copy the salary.txt file into new_salary.txt file.

root@hello:~/test# sed 'w new_salary.txt' salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@hello:~/test# root@hello:~/test# cat new_salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@hello:~/test#

5.       Write the lines matching pattern “Atul” to pattern_found.txt file. Below command can be used for write only odd numbered line (1~2) or write only even numbered lines (2~2).

root@ctier:~/test# sed -n '/Atul/ w pattern_found.txt' salary.txt root@ctier:~/test# cat pattern_found.txt 302,Atul,CEO,26000 root@ctier:~/test#

 You can all above commands by replacing “p” or “d” by “w” option.

Want to know more about : How to run Windows softwares on Linux

6.       Mostly, sed used to substitute pattern by another pattern.

sed '[address-range|pattern-range] s/originalstring/replacement-string/[substitute-flags]' inputfile

Replace all occurrences of Atul with Mohan:

root@ctier:~/test# sed 's/Atul/Mohan/' salary.txt 1001,Ajay,Manager,25000 101,Satish,Founder,30000 302,Mohan,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@ctier:~/test#

Note: We are not changing the file. Sed only changes the display output not a file. If you want to ignore the case the use “I” option at end pattern. i.e. ‘s/Atul/Mohan/i’

Replace Satish with Pravin only on lines that contain thekeyword 'Founder':

root@ctier:~/test# sed '/Founder/s/Satish/Pravin/' salary.txt 1001,Ajay,Manager,25000 101,Pravin,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 1231,Kalyan,Human Resource,20000 root@ctier:~/test#

7.       Replace the 1st occurrence of lower case “a” with upper case “A” in a line

root@ctier:~/test# sed 's/a/A/' salary.txt 1001,AjAy,Manager,25000 101,SAtish,Founder,30000 302,Atul,CEO,26000 434,RAj,Senior Manager, 26000 1231,KAlyan,Human Resource,20000 root@ctier:~/test#

                Same command we can use for replace all occurrences of “a” in a line by upper case “A” with below command:

               

root@ctier:~/test# sed 's/a/A/g' salary.txt 1001,AjAy,MAnAger,25000 101,SAtish,Founder,30000 302,Atul,CEO,26000 434,RAj,Senior MAnAger, 26000 1231,KAlyAn,HumAn Resource,20000 root@ctier:~/test#

If you want to replace the second occurrence of “a” with “A” in each line then use following syntax: sed 's/a/A/2' salary.txt

Sed flags in short:

g-            global

i-                    Ignore case

p-            Print

w-           Write

You can use this flags as per your need in sed command.

8.       Suppress the meaning of special characters in sed while mataching pattern. Use backslash to remove special meaning.

[root@rhel Documents]# cat test.text Hello How are you ? Hello /usr/local/bin directory used to store the commands script [root@rhel Documents]# [root@rhel Documents]# sed 's/\/usr\/local\/bin/\/usr\/bin/g' test.text Hello How are you ? Hello /usr/bin directory used to store the commands script [root@rhel Documents]#

9.       Get matched pattern in “&”. Lets see the use of “&” operator.

Enclose the employee id in [EMP ID] and show output.

[root@rhel Documents]# sed 's/^[0-9][0-9][0-9]/[&]/' salary.txt [100]1,Ajay,Manager,25000 [101],Satish,Founder,30000 [302],Atul,CEO,26000 [434],Raj,Senior Manager, 26000 [123]1,Kalyan,Human Resource,20000 [root@rhel Documents]# [root@rhel Documents]# sed 's/^[0-9][0-9][0-9][0-9]/[&]/' salary.txt [1001],Ajay,Manager,25000 101,Satish,Founder,30000 302,Atul,CEO,26000 434,Raj,Senior Manager, 26000 [1231],Kalyan,Human Resource,20000 [root@rhel Documents]# sed -e  's/^[0-9][0-9][0-9]/[&]/' -e 's/^[0-9][0-9][0-9][0-9]/[&]/' salary.txt [100]1,Ajay,Manager,25000 [101],Satish,Founder,30000 [302],Atul,CEO,26000 [434],Raj,Senior Manager, 26000 [123]1,Kalyan,Human Resource,20000 [root@rhel Documents]#

10.       Use execute flag (e). Run commands inside the files by sed command.

Suppose, we add the ls –l command in file.txt file at starting of each line. Then we want to run these lines then how to run :

[root@rhel Documents]# cat > file.txt /etc/passwd /etc/group [root@rhel Documents]# root@rhel Documents]# sed -i 's/^/ls -l /' file.txt [root@rhel Documents]# [root@rhel Documents]# cat file.txt ls -l /etc/passwd ls -l /etc/group [root@rhel Documents]# sed 's/^/ /e' file.txt -rw-r--r--. 1 root root 1527 Mar 20 18:18 /etc/passwd -rw-r--r--. 1 root root 786 Mar 20 18:18 /etc/group [root@rhel Documents]#

Sed has many more uses. This is the basic sed commands use. I have given here most of the sed Command in Unix with examples. If you need more details you can post on my facebook page.

Jinfo Command                                         Jstat Command           File System in Linux

Check Port Details by netstat Command: TCP, UDP port’s status, network connections, routing tables, interface statistics, masquerade connections, and multicast memberships

Generally, netstat command is used to check port is listening or not. But actually, there are lots of things we can check with the netstat command in Linux. Let’s understand the some examples of the netstat command to get desired result.

1.       List out all open ports of Linux server:   netstat –a

root@hello:~#netstat -a Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State tcp        0              0         localhost:mysql         *:*                     LISTEN tcp        0              0           *:www                   *:*                     LISTEN tcp        0              0           *:webmin                *:*                     LISTEN tcp        0              0         ctier.local:domain      *:*                     LISTEN tcp        0              0         localhost:domain        *:*                     LISTEN tcp        0              0               *:ssh                   *:*                     LISTEN

2.       List only open TCP ports of server: netstat –at

root@hello:~#netstat -at   Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State tcp        0              0            localhost:mysql         *:*                     LISTEN tcp        0              0              *:www                   *:*                     LISTEN

3.       List only open UDP ports of server: netstat –au

root@ctier:~# netstat -au Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State udp        0             0            *:10000                 *:* udp        0             0            *:50715                 *:* udp        0             0            *:bootpc                *:* udp        0             0             *:snmp                  *:*

4.       Now, you can see the statistics of UDP and TCP ports by commands as below : netstat –st  or netstat –su

[redhat@localhost ~]$ netstat -st IcmpMsg:     InType3: 1     InType8: 1     OutType0: 1     OutType3: 11 Tcp:     681 active connections openings     1 passive connection openings     8 failed connection attempts     4 connection resets received     3 connections established     15811 segments received     13669 segments send out     8 segments retransmited     0 bad segments received.     147 resets sent UdpLite: TcpExt:     564 TCP sockets finished time wait in fast timer     4 time wait sockets recycled by time stamp     236 delayed acks sent     1 packets directly queued to recvmsg prequeue.     8564 packets header predicted                   818 acknowledgments not containing data received

5.       Now,  if you want to see the all ports which are in LISTENING mode only type command as

netstat  -l and only for TCP listening ports netstat –lt and for UDP : netstat –lu

[redhat@localhost ~]$ netstat -l | head Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address               Foreign Address             State tcp        0             0           *:ssh                                *:*                         LISTEN tcp        0             0        localhost.localdomain:smtp  *:*                         LISTEN tcp        0            0           *:44626                            *:*                         LISTEN tcp        0            0           *:ssh                                 *:*                         LISTEN                ……

[redhat@localhost ~]$ netstat -lu Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address               Foreign Address             State udp        0            0          *:mdns                           *:* udp        0            0          *:bootpc                        *:* udp        0            0          *:58959                         *:* [redhat@localhost ~]$

[redhat@localhost ~]$ netstat -lt Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address               Foreign Address             State tcp        0            0              *:ssh                            *:*                         LISTEN tcp        0            0       localhost.localdomain:smtp  *:*                         LISTEN tcp        0            0             *:44626                        *:*                         LISTEN tcp        0            0             *:ssh                             *:*                         LISTEN [redhat@localhost ~]$

6.       List the ports which are listening unix ports using netstat –lx

[redhat@localhost ~]$ netstat -lx | head Active UNIX domain sockets (only servers) Proto RefCnt Flags       Type       State         I-Node Path unix  2      [ ACC ]     STREAM     LISTENING     11846  /var/run/acpid.socket unix  2      [ ACC ]     STREAM     LISTENING     12271  /var/run/sdp unix  2      [ ACC ]     STREAM     LISTENING     12564  public/cleanup unix  2      [ ACC ]     STREAM     LISTENING     12572  private/tlsmgr unix  2      [ ACC ]     STREAM     LISTENING     12576  private/rewrite unix  2      [ ACC ]     STREAM     LISTENING     11573  /var/run/dbus/system_bus_socket unix  2      [ ACC ]     STREAM     LISTENING     12580  private/bounce                unix  2      [ ACC ]     STREAM     LISTENING     12584  private/defer

7.       If you want to know the program which using specific port then use : netstat -p

[redhat@localhost ~]$ netstat -p | head -20 (Not all processes could be identified, non-owned process info  will not be shown, you would have to be root to see it all.) Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name tcp        1      0 192.168.136.128:57484       201-215-187-203.static:http CLOSE_WAIT  1978/clock-applet tcp        0      0 192.168.136.128:ssh         192.168.136.1:51507         ESTABLISHED - tcp        0      0 192.168.136.128:58239       bom03s02-in-f21.1e100:https ESTABLISHED 2047/firefox tcp        0      0 192.168.136.128:42256       bom03s02-in-f21.1e100:https ESTABLISHED 2047/firefox Active UNIX domain sockets (w/o servers) Proto RefCnt Flags       Type       State         I-Node PID/Program name    Path unix  2      [ ]         DGRAM                    8569   -                   @/org/kernel/udev/udevd unix  2      [ ]         DGRAM                    11911  -                   @/org/freedesktop/hal/udev_event                unix  22     [ ]         DGRAM                    11408  -                   /dev/log

8.       If you want to list all network interfaces use this command: netstat –i

[redhat@localhost ~]$ netstat -i Kernel Interface table Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg eth0       1500   0    18773      0      0      0    15719      0      0      0 BMRU                lo        16436   0       18      0      0      0       18      0      0      0 LRU

9.       If you know the ifconfig command then ifconfig –a and netstat –ie command’s output will be same. Both commands are providing extended network interface information

Above mentioned state’s details as follows:

State

       The state of the socket. Since there are no states in raw mode and usually no states used in UDP, this column may be left blank.  Normally  this can be one of several values:

ESTABLISHED: The socket has an established connection.

SYN_SENT: The socket is actively attempting to establish a connection.

SYN_RECV: A connection request has been received from the network.

FIN_WAIT1: The socket is closed, and the connection is shutting down.

FIN_WAIT2: Connection is closed, and the socket is waiting for a shutdown from the remote end.

TIME_WAIT: The socket is waiting after close to handle packets still in the network.

CLOSED:  The socket is not being used.

CLOSE_WAIT: The remote end has shut down, waiting for the socket to close.

LAST_ACK: The remote end has shut down, and the socket is closed. Waiting for acknowledgement.

LISTEN: The socket is listening for incoming connections.

CLOSING: Both sockets are shut down but we still donât have all our data sent.

UNKNOWN: The state of the socket is unknown.

Check Port details by nc command in Linux: Port Scanning, Talking to servers, Data Transfer, Client server Model

How to recover deleted files in Linux                              How to Run Unix commands on Windows     

Nc command is different as compared to netstat command. It comes under the netcat utility. It is helpful to open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, scanning of ports and it deals with IPv4 and IPv6 as well.

Common uses of this command includes as :

·         To check simple TCP proxies

·         Use it in shell-script based HTTP clients and servers

·         For network daemon testing

·         For a SOCKETS or HTTP ProxyCommand

Practical Uses

There are different practical usages of this command as follows.

1.       Open a TCP connection to port 42 of my.home.org, using port 31447 as the source port, with a timeout of 5 seconds:

$ nc -p 31447 -w 5 my.home.org 42

2.       Open a UDP connection to port 53 of my.home.org

                                        $ nc -u my.home.org 53

3.       Open a TCP Connection to port 42 of my.home.org using 10.1.2.3 as the IP for local end of the connection:

$nc –s   10.1.2.3 my.home.org 42

4.       Connect to port 42 of my.home.org via an HTTP proxy at 10.2.3.4 port 8080.

$ nc -x10.2.3.4:8080 -Xconnectmy.home.org 42

5.       The same example again, this time enabling proxy authentication with username user1

                                        $ nc -x10.2.3.4:8080 -Xconnect –Puser1my.home.org 42

 Other uses of nc command as follows:

PORT SCANNING

     It is useful to check which ports are open and which services are running on a target machine.  The -z flag can be used to tell nccommand to report open ports, rather than initiate a connection.  For example:

           $ nc -z host.example.com 20-30

Connection to host.example.com 22 port [tcp/ssh] succeeded!

Connection to host.example.com 25 port [tcp/smtp] succeeded!

In case of Ubuntu machine you need to specify protocol like for TCS use –t option, for UDP use –u option as follows:

root@hello:~# nc -t my.home.org 20-30

SSH-2.0-OpenSSH_5.1p1 Debian-5ubuntu1

Click here to know more about : How to install windows softwares on Linux

20-30 means port range which we wanted to scan. It might be useful to know which server software is running and which versions. This information is often contained in greeting banners. In order to retrieve these, it is necessary to first make connection, and then break the connection when the banner has been retrieved. This can be accomplished by specifying a small timeout with –w flag or by issuing a “QUIT” command to the server.

TALKING TO SERVER

It is useful to talk to servers for what data a server is sending in response to commands issued by the client.  For example, to retrieve the home page of a web site:

   $ echo -n "GET / HTTP/1.0\r\n\r\n" | nc host.example.com 80

Note that this also displays the headers sent by the web server.  They can be filtered. More complicated examples can be built up when the user knows the format of requests required by the server.  As another example, an email may be submitted to

 A SMTP server using:

           $ nc [-C] localhost 25 << EOF

           HELO host.example.com

           MAIL FROM:<user@host.example.com>

           RCPT TO:<user2@host.example.com>

           DATA

Body of email.

           .

           QUIT

           EOF

DATA TRANSFER

The example in the previous section can be expanded to build a basic data transfer model.  Any information input into one end of the connection will be output to the other end. Input and output can be easily captured in order to emulate file transfer.

We can use nc command to listen on a specific port and capture output into a file :

$ nc –l 1234 >nccommand.out

Now, use second machine and try to connect to listening nc process feeding it the file which is to be transferred.

$nc my.home.org 1234 < nccommand.in

After the command execution complete i.e. file transfer process completes, connection will close automatically.

CLIENT/SERVR MODEL

It is very easy to create client/server model using nc command. There are some steps as follows:

Step 1: On one console, start nc command to listen on a specific port for a connection. e.g.

$ nc –l 1235

Nc command is now listening on port 1235 for a connection.

Step 2: On a second console (or second machine) connect to the machine and port being listened on :

$ nc 127.0.0.1 1235                                           (Here we used second Console)

Now, there is connection is present between the ports. Anything typed at the second console will be concatenated to the first and vice versa. We can terminate this connection using an EOF character. After the connection establishment, nc command doesn’t take care of which machine is working as server or which machine is working as client.

vmstat command To check CPU performance                    Commands to check CPU usage

Linux Commands & Tools for getting network statistics and port details

mpstat command to check cpu usage                                        vmstat command to check cpu performance

There are many commands available to get network statistics and particular port details. In this article, we are not going in deep to understand these commands but we will just understand the overall functionality of these command.

Nmap Command

Nmap is, especially network exploration tool but it can be used as a security or port scanner. We can scan entire network or selected machine or single server by nmap command. It is also useful for scanning firewall rules.

Nmap is designed to allow system administrators:

·         To scan large networks to determine which servers or hosts are running and what services they are offering. 

·         Nmap supports a large number of scanning techniques such as: UDP, TCP, TCP SYN (half open), ftp proxy (bounce attack), ICMP (ping sweep), FIN,  ACK  sweep,  Xmas Tree, SYN sweep, IP Protocol, and Null scan. 

·         Nmap also offers a number of advanced features such as remote OS detection via TCP/IP fingerprinting, stealth scanning, dynamic delay and retransmission calculations, parallel scanning, detection  of down hosts via parallel pings, decoy scanning, port filtering detection, direct (non-portmapper) RPC scanning, fragmentation scanning, and flexible target and port specification.

Netstat Command

A netstat is a command line tool which provides information about incoming and outgoing network connections, routing tables, network protocol statistics and network interface. Following are some of the features of netstat command.

Features of netstat command:

• Displays routing information maintained by Kernel. (netstat –r)
• Display multicast group membership information.(netstat –g)
• It is displays information related to all network statistics/interfaces. (netstat –s/netstat –i)
• It can monitor system continuously by netstat –tc command. (c – Option most important.)
• It displays the network time related information.

nc Command in Linux (netcat command)

A nc is a command line tool to check TCP and UDP connections and listens it. It can open TCP connections, send UDP packets, listens on TCP and UDP ports, do port scanning and source routing.

Common uses of nc command include:

Ø  Simple TCP proxies
Ø  Shell script based HTTP clients and servers
Ø   Source routing based connectivity testing

Following are some of the best network scanners as :

Unicornscan

Unicornscan is a new information gathering and correlation engine built for and by members of the security research and testing communities. It was designed to provide an engine that is Scalable, Accurate, Flexible, and Efficient. It is released for the community to use under the terms of the GPL license.

Get more information related to this tool by clicking here: unicornscan 

Zenmap – The Network mapper GUI

Zenmap is the Nmap Security Scanner GUI. It is available in multi-platform  for Linux, Windows, Mac OS X, BSD, etc. It is free and open source application and it is designed to make Nmap easy for beginners to use while providing advanced features for experienced Nmap users. Frequently used scans can be saved as profiles to make them easy to run repeatedly. A command creator allows interactive creation of nmap command lines. Scan results can be saved and viewed later. Saved scan results can be compared with one another to see how they differ. The results of recent scans are stored in a searchable database.

Nast – Network Analyzer Sniffer Tool

Nast is a packet sniffer and a LAN analyzer based on Libnet and Libpcap. It can sniff in normal mode or in promiscuous mode the packets on a network interface and log it. It dumps the headers of packets and  the payload in ascii or ascii-hex format. You can apply a filter. The sniffed data can be saved in a parated file.

As analyzer Tool, it has many features like as:

·         Follows a TCP-DATA stream
·         Find LAN Internet gateways
·         Discover promiscuous nodes
·         Reset an established connection
·         Perform a single half-open portscanner
·         Perform a multi half-open portscanner
·         Find link type (hub or switch)
·         Catch daemon banner of LAN nodes
·         Build LAN hosts list

Knocker

Knocker is a simple and easy to use TCP security port scanner written in C to analyze hosts and all of the different services started on them. It is available for Linux, FreeBSD, HP-UX, and Windows9x/2000/NT and it is licensed under the GNU General Public License GPL.

Click here to  know more about Knocker

Linux boot Loader                                                                        file system in Linux