Scripting tricks and one liners
This page is useful for those who know scripting logic and looking for one liners.
Shell Built in Variables and Meanings
$# Number of command line arguments. Useful to test no. of command line args in shell script.
$* All arguments to shell
$@ Same as above
$- Option supplied to shell
$$ PID of shell
$! PID of last started background process (started with &)
awk one liners
1. Remove empty or blank lines with the following command
awk NF file > newfile
2. Remove first 2 or 3 lines from the output using awk
awk 'NR > 3 { print } ' filename
3. Print Horizontal lines to vertical
awk ' { for(i=1;i<=NF;i++) { print $i } } ' filename
4. Print Vertical lines into Horizontal
nawk 'ORS=" "' filename && echo
OR
cat file | xargs echo
4. Column wise awk , if column 2 has word "JUDI", print the entire line
awk '$2 == "JUDI" {print $0}' filename.log
5. Use awk command to separate the fields using field separator
grep -i judi /etc/passwd |awk -F":" '{print $6}'
6. Print the line immediately before the pattern, not the line containing the pattern
awk '/^DD/{print x};{x=$0}' myfile.txt
7. Print the line where pattern is found,print previous line too using awk,
awk '/^DD/{print x;print};{x=$0}' myfile.txt
8. Print the line immediately after the pattern, not the line containing the pattern
awk '/^DD/{f=1;next}f{print;exit}' myfile.txt
awk '/PATTERN/{i=1;next;}{if(i){i--; print;}}'
9. Print the line where pattern is found,print next line too
awk '/^DD/{f=1;print;next}f{print;exit}' myfile.txt
10. Print the line where pattern is found, print the previous line and print the next 4 lines
/usr/xpg4/bin/awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=1 a=4 s="Retryable" /var/adm/messages.3
11. Print the next four lines after the pattern using awk
cat /etc/passwd | awk '/judi/ {for(i=1; i<=4; i++) {getline; print}}'
12. Multiple word grep pattern in awk, prints all records in 'filename' that contain both `2400' and `foo'.
awk '/2400/ && /foo/' filename
13. Print if any one of the word exists in awk like egrep, prints all records in 'filename' that contain either `2400' or `foo', or both.
awk '/2400/ || /foo/' filename
14. Prints all the words expect the given word in awk ,Like grep -v ; prints all records in 'filename' that do not contain the string `foo'.
awk '! /foo/' filename
15. Print last word using awk
awk '{print $NF}' /tmp/123 # Prints last word from the file
16. Print last but one word using awk
awk '{print $(NF-2)}' /tmp/123 # Prints last but one word from the file
17. Compare two files line by line and print the difference line alone
nawk 'NR==FNR{name[$1]++;next} !($1 in name)' file1 file2
18. Compare two files line by line and print the matching lines alone
nawk 'NR==FNR{name[$1]++;next}$1 in name' file2 file1
19. Compare tow file line by line and print the difference of both files
nawk '{ h[$0] = ! h[$0] } END { for (k in h) if (h[k]) print k }' file1 file2
20. Print outputs between two patterns
awk '/Cpu Compatibility Group/{flag=1;next}/Server Pool =/{flag=0}flag'
21. Print the previous line where pattern is found, {Find the NIC interface wby knowing the IP Address
ifconfig -a | awk '/192.168.1.10/{print x};{x=$0}'
22. Do not print the pattern matching line and next line.
iostat -E | awk '/NET/{matched=1;next} 1==matched{matched=0;next} {print}'
iostat -E | awk '/NET|SEAGATE|TEAC/{matched=1;next} 1==matched{matched=0;next} {print}'
23. Print all records from some regexp:
nawk '/regexp/{f=1}f' file OR (ex) nawk '/SunOS/{f=1}f' /var/adm/messages
24. Print all records after some regexp:
nawk 'f;/regexp/{f=1}' file
(ex) nawk 'f;/SunOS/{f=1}' /var/adm/messages
25. Print the Nth record after some regexp:
nawk 'c&&!--c;/regexp/{c=N}' file
(ex) nawk 'c&&!--c;/SunOS/{c=N}' /var/adm/messages
26. Print every record except the Nth record after some regexp:
nawk 'c&&!--c{next}/regexp/{c=N}1' file
nnawk 'c&&!--c{next}/SunOS/{c=5}1' /var/adm/messages.2 (Every 5th line after SunOS will not display)
27. Print the N records after some regexp: Not the regexp
nawk 'c&&c--;/regexp/{c=5}' file
nawk 'c&&c--;/regexp/{c=5}' /var/adm/messages.2
28. Print every record except the N records after some regexp:
nawk 'c&&c--{next}/regexp/{c=N}1' file
29. Print every record except the N records after some regexp:
nawk '/regexp/{c=N}c&&c--' file
General one liners
1. Print characters in bold letters
echo -e "I am \033[1m BOLD \033[0m Person"
2. Black foreground and white background in default color scheme
echo -e "\033[7m Linux OS! Best OS!! \033[0m"
3. Print characters in colours 30m - 37m(black)
echo -e "\033[31m I am in Red"
4. set background color 40m - 47m
echo -e "\033[44m Wow!!!"
5. will show all files beginning with letters a,b,c
ls [abc]*
6. Print from line number 20 to line number 30 and store this result to file called 'hlist'
tail +20 < hotel.txt | head -n30 >hlist
7. passing command to another server using 'Single Quote'
ssh -q -o ConnectTimeout=10 root@$i 'uname -a; /usr/platform/`uname -i`/sbin/prtdiag -v | head -1'
8. Find how many times a line has repeated in a file
cat /tmp/1234 | sort | uniq -c | sort -nr | more
9. Remove duplicate words from a line
echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs
10. Remove/delete the first 8 words(includes spaces) in a file's every line
cut -d' ' -f8- <filename>
11. Find any string has alpha
echo "IamARobot" | grep '[a-zA-Z]'
12. Find any string has numeric
echo "IamARobot7" | grep '[0-9]'
13. Find any string has alpha umeric
echo "IamARobot7" | grep '[a-zA-Z0-9]'
14. Print words in a straight line giving a tab space
echo Mountpoint\tVolume\tFilesystem\tTotal\tBlocks
15. To get previous day in solaris (86400 seconds for one day) Calculate accordingly
perl -MPOSIX=strftime -le 'print strftime("%d", localtime(time-86400))'
16. To get previous two days date in solaris
perl -MPOSIX=strftime -le 'print strftime("%d", localtime(time-172800))'
17. To get previous date in solaris (24 Hours per day) Calculate accordiangly
TZ=GMT+24 date +%d-%m-%Y
TZ=GMT+48 date +%d-%m-%Y
18. Remove first 5 words from a file
cat filename | cut -d " " -f6-
19. Find the duplicate entries from a file
cat /tmp/1234 | sort | uniq -c | sort -nr
20. Division /Divide using expr command
expr $(df -k /var | tail -1 | awk '{print $2}') \/ 1024 \/ 1024
21. Remove blank/empty line in a file
grep '^..' <filename>
22. Find exact pattern match using grep
grep '\<judi07\>' file.txt
23. Get epoch seconds of current time in solaris
nawk "BEGIN{print srand}"
24. Set Prompt after login in UNIX - Update the profile file
PS1=`hostname`:'$PWD# '
25. Read a file and print the lines one by one
while IFS= read -r var; do
echo "$var"
sleep 1
done < filename
26. A file has multiple lines, Every three lines will be send to a separate file, file name starts with "x"
split -l 3 filename
27. Read filename1 by line by line ; Get a string from that line as var "A" ; if that var available/matches in any of the line in filename2,
Aappend the entire line of filename1 to filename2 next to the line matches the string
#!/usr/bin/ksh
while IFS= read -r var
do
A=`echo "$var" | awk -F"/" '{print $4}' | awk '{print $1}'`
printf '%s\n' /$A/a "$var" . w q | ex -s filename2
done < filename1
28. Append a character at the end of each line in unix file
awk '{print $0 "," } file > outFile
sed 's/$/,/' file > outFile
29. WC Command Examples to Count Number of Lines, Words, Characters in UNIX
for loop
- This for loop is used to fetch the outputs from a list of servers
- with conditions provided,
- Any one of the server struck with timeout- No Problem
- Password issue - No Problem
- HostKeyChecking - No Problem
#!/bin/ksh
> NotReach.txt
> Outputs.txt
for i in `cat server_list` ; do
ssh -nq -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=15 judi@$i "uname -a" >> Outputs.txt
if [ $? -ne 0 ];then
echo "$i - Not Reachable" >> NotReach.txt
fi
done
sed one liners
1. Remove last n characters - 1 or 2 or 3 or n characters
sed 's/.\{1\}$//'
2. remove empty lines using sed
sed '/^$/d'
3. print the last line using sed
sed '$!d'
4. Replace a Character using sed (Replace M with a space)
sed 's/\M/ /g'
5. Replace a Character using sed (Replace M with a G)
sed 's/\M/G/g'
6. print the line immediately before the pattern, not the line containing the pattern
sed -n '/^DD/{g;1!p;};h' myfile.txt
7. print the line immediately after the pattern, not the line containing the pattern
sed -n '/^DD/{n;p;}' myfile.txt
8. Print the line where pattern is found,print next line too
sed -n '/^DD/{p;n;p;}' myfile.txt
9. Remove round Brackets using sed
sed 's/[)(]//g'
10. To remove last four characters from a string or a file
echo "diana123" | sed "s/....$//" #(dian - the dots refers each characters)
sed "s/....$//" file1
11. To remove first 2 characters from a string or file
sed 's/^..//' file1
12. To display last four characters from a string or a file
echo "diana123" | sed "s/.*\(..$\)/\1/"
sed "s/.*\(....$\)/\1/" file1
13. To display the last four characters from a file line by line
while read -r line
do
echo ${line:(-4)}
done < "file"
14. Exact pattern match of a variable using awk
awk '$1=="'$ServerName'" {print $1}' file.txt
15. Remove only the mentioned special characters ex: " and ,
9. Print the line where pattern is found,print next line too
awk '/^DD/{f=1;print;next}f{print;exit}' myfile.txt
10. Print the line where pattern is found, print the previous line and print the next 4 lines
/usr/xpg4/bin/awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=1 a=4 s="Retryable" /var/adm/messages.3
11. Print the next four lines after the pattern using awk
cat /etc/passwd | awk '/judi/ {for(i=1; i<=4; i++) {getline; print}}'
12. Multiple word grep pattern in awk, prints all records in 'filename' that contain both `2400' and `foo'.
awk '/2400/ && /foo/' filename
13. Print if any one of the word exists in awk like egrep, prints all records in 'filename' that contain either `2400' or `foo', or both.
awk '/2400/ || /foo/' filename
14. Prints all the words expect the given word in awk ,Like grep -v ; prints all records in 'filename' that do not contain the string `foo'.
awk '! /foo/' filename
awk '{print $NF}' /tmp/123 # Prints last word from the file
16. Print last but one word using awk
awk '{print $(NF-2)}' /tmp/123 # Prints last but one word from the file
17. Compare two files line by line and print the difference line alone
nawk 'NR==FNR{name[$1]++;next} !($1 in name)' file1 file2
18. Compare two files line by line and print the matching lines alone
nawk 'NR==FNR{name[$1]++;next}$1 in name' file2 file1
19. Compare tow file line by line and print the difference of both files
nawk '{ h[$0] = ! h[$0] } END { for (k in h) if (h[k]) print k }' file1 file2
20. Print outputs between two patterns
awk '/Cpu Compatibility Group/{flag=1;next}/Server Pool =/{flag=0}flag'
21. Print the previous line where pattern is found, {Find the NIC interface wby knowing the IP Address
ifconfig -a | awk '/192.168.1.10/{print x};{x=$0}'
22. Do not print the pattern matching line and next line.
iostat -E | awk '/NET/{matched=1;next} 1==matched{matched=0;next} {print}'
iostat -E | awk '/NET|SEAGATE|TEAC/{matched=1;next} 1==matched{matched=0;next} {print}'
23. Print all records from some regexp:
nawk '/regexp/{f=1}f' file OR (ex) nawk '/SunOS/{f=1}f' /var/adm/messages
24. Print all records after some regexp:
nawk 'f;/regexp/{f=1}' file
(ex) nawk 'f;/SunOS/{f=1}' /var/adm/messages
25. Print the Nth record after some regexp:
nawk 'c&&!--c;/regexp/{c=N}' file
(ex) nawk 'c&&!--c;/SunOS/{c=N}' /var/adm/messages
26. Print every record except the Nth record after some regexp:
nawk 'c&&!--c{next}/regexp/{c=N}1' file
nnawk 'c&&!--c{next}/SunOS/{c=5}1' /var/adm/messages.2 (Every 5th line after SunOS will not display)
27. Print the N records after some regexp: Not the regexp
nawk 'c&&c--;/regexp/{c=5}' file
nawk 'c&&c--;/regexp/{c=5}' /var/adm/messages.2
28. Print every record except the N records after some regexp:
nawk 'c&&c--{next}/regexp/{c=N}1' file
29. Print every record except the N records after some regexp:
nawk '/regexp/{c=N}c&&c--' file
General one liners
1. Print characters in bold letters
echo -e "I am \033[1m BOLD \033[0m Person"
2. Black foreground and white background in default color scheme
echo -e "\033[7m Linux OS! Best OS!! \033[0m"
3. Print characters in colours 30m - 37m(black)
echo -e "\033[31m I am in Red"
4. set background color 40m - 47m
echo -e "\033[44m Wow!!!"
5. will show all files beginning with letters a,b,c
ls [abc]*
6. Print from line number 20 to line number 30 and store this result to file called 'hlist'
tail +20 < hotel.txt | head -n30 >hlist
7. passing command to another server using 'Single Quote'
ssh -q -o ConnectTimeout=10 root@$i 'uname -a; /usr/platform/`uname -i`/sbin/prtdiag -v | head -1'
8. Find how many times a line has repeated in a file
cat /tmp/1234 | sort | uniq -c | sort -nr | more
9. Remove duplicate words from a line
echo "zebra ant spider spider ant zebra ant" | xargs -n1 | sort -u | xargs
10. Remove/delete the first 8 words(includes spaces) in a file's every line
cut -d' ' -f8- <filename>
11. Find any string has alpha
echo "IamARobot" | grep '[a-zA-Z]'
12. Find any string has numeric
echo "IamARobot7" | grep '[0-9]'
13. Find any string has alpha umeric
echo "IamARobot7" | grep '[a-zA-Z0-9]'
14. Print words in a straight line giving a tab space
echo Mountpoint\tVolume\tFilesystem\tTotal\tBlocks
15. To get previous day in solaris (86400 seconds for one day) Calculate accordingly
perl -MPOSIX=strftime -le 'print strftime("%d", localtime(time-86400))'
16. To get previous two days date in solaris
perl -MPOSIX=strftime -le 'print strftime("%d", localtime(time-172800))'
17. To get previous date in solaris (24 Hours per day) Calculate accordiangly
TZ=GMT+24 date +%d-%m-%Y
TZ=GMT+48 date +%d-%m-%Y
18. Remove first 5 words from a file
cat filename | cut -d " " -f6-
19. Find the duplicate entries from a file
cat /tmp/1234 | sort | uniq -c | sort -nr
20. Division /Divide using expr command
expr $(df -k /var | tail -1 | awk '{print $2}') \/ 1024 \/ 1024
21. Remove blank/empty line in a file
grep '^..' <filename>
22. Find exact pattern match using grep
grep '\<judi07\>' file.txt
23. Get epoch seconds of current time in solaris
nawk "BEGIN{print srand}"
24. Set Prompt after login in UNIX - Update the profile file
PS1=`hostname`:'$PWD# '
25. Read a file and print the lines one by one
while IFS= read -r var; do
echo "$var"
sleep 1
done < filename
26. A file has multiple lines, Every three lines will be send to a separate file, file name starts with "x"
split -l 3 filename
27. Read filename1 by line by line ; Get a string from that line as var "A" ; if that var available/matches in any of the line in filename2,
Aappend the entire line of filename1 to filename2 next to the line matches the string
#!/usr/bin/ksh
while IFS= read -r var
do
A=`echo "$var" | awk -F"/" '{print $4}' | awk '{print $1}'`
printf '%s\n' /$A/a "$var" . w q | ex -s filename2
done < filename1
28. Append a character at the end of each line in unix file
awk '{print $0 "," } file > outFile
sed 's/$/,/' file > outFile
29. WC Command Examples to Count Number of Lines, Words, Characters in UNIX
wc -w : Prints the number of words in a file.
wc -c : Displays the count of bytes in a file.
wc -m : prints the count of characters from a file.
30. Find full process name using ps command
/usr/ucb/ps -auwx
31. How to find the length of a string in Linux
$ x="judi"
$ echo ${#x}
4
30. Find full process name using ps command
/usr/ucb/ps -auwx
31. How to find the length of a string in Linux
$ x="judi"
$ echo ${#x}
4
32. Remove all special characters
tr -dc '[:alnum:]\n\r'
tr -dc '[:alnum:]\n\r'
The first tr deletes special characters. d means delete, c means complement (invert the character set). So, -dc means delete all characters except those specified. The \n and \r are included to preserve linux or windows style newlines
33. Translate upper characters to lower lower case
tr '[:upper:]' '[:lower:]'
tr '[:upper:]' '[:lower:]'
34. Translate upper characters to lower lower case
tr '[:upper:]' '[:lower:]'
tr '[:upper:]' '[:lower:]'
for loop
- This for loop is used to fetch the outputs from a list of servers
- with conditions provided,
- Any one of the server struck with timeout- No Problem
- Password issue - No Problem
- HostKeyChecking - No Problem
#!/bin/ksh
> NotReach.txt
> Outputs.txt
for i in `cat server_list` ; do
ssh -nq -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=15 judi@$i "uname -a" >> Outputs.txt
if [ $? -ne 0 ];then
echo "$i - Not Reachable" >> NotReach.txt
fi
done
sed one liners
1. Remove last n characters - 1 or 2 or 3 or n characters
sed 's/.\{1\}$//'
2. remove empty lines using sed
sed '/^$/d'
3. print the last line using sed
sed '$!d'
4. Replace a Character using sed (Replace M with a space)
sed 's/\M/ /g'
5. Replace a Character using sed (Replace M with a G)
sed 's/\M/G/g'
6. print the line immediately before the pattern, not the line containing the pattern
sed -n '/^DD/{g;1!p;};h' myfile.txt
7. print the line immediately after the pattern, not the line containing the pattern
sed -n '/^DD/{n;p;}' myfile.txt
8. Print the line where pattern is found,print next line too
sed -n '/^DD/{p;n;p;}' myfile.txt
9. Remove round Brackets using sed
sed 's/[)(]//g'
10. To remove last four characters from a string or a file
echo "diana123" | sed "s/....$//" #(dian - the dots refers each characters)
sed "s/....$//" file1
11. To remove first 2 characters from a string or file
sed 's/^..//' file1
12. To display last four characters from a string or a file
echo "diana123" | sed "s/.*\(..$\)/\1/"
sed "s/.*\(....$\)/\1/" file1
13. To display the last four characters from a file line by line
while read -r line
do
echo ${line:(-4)}
done < "file"
14. Exact pattern match of a variable using awk
awk '$1=="'$ServerName'" {print $1}' file.txt
15. Remove only the mentioned special characters ex: " and ,
sed 's/[\",]//g'
find command one liners
1. Find symbolic links in a directory.
find /opt/home/judi/ -type l -ls
2. Find files more than 270 days old
/usr/xpg4/bin/find /opt/home/judi/ -mtime +270 -exec ls -l {} \;
3. Find the directories alone under the path /var/tmp/
find /var/tmp/ -type d -exec ls -ld {} \;
find command one liners
1. Find symbolic links in a directory.
find /opt/home/judi/ -type l -ls
2. Find files more than 270 days old
/usr/xpg4/bin/find /opt/home/judi/ -mtime +270 -exec ls -l {} \;
3. Find the directories alone under the path /var/tmp/
find /var/tmp/ -type d -exec ls -ld {} \;
4. grep contents in file using find command.
find / -xdev -type f -exec grep -l bmc {} \;
~Judi~
Comments are welcome :)
find / -xdev -type f -exec grep -l bmc {} \;
~Judi~
Comments are welcome :)
No comments:
Post a Comment