Showing posts with label file handling. Show all posts
Showing posts with label file handling. Show all posts

Monday, April 12, 2010

if flags in UNIX shell scripting

/bin/[ is another name for /bin/test. It evaluates its arguments as a boolean expression, and exits with an exit code of 0 if it is true, or 1 if it is false.
If test is invoked as [, then it requires a closing bracket ] as its last argument. Otherwise, there must be no closing bracket.
test understands the following expressions, among others:
-e filename
True if filename exists.
-d filename
True if filename exists and is a directory.
-f filename
True if filename exists and is a plain file.
-h filename
True if filename exists and is a symbolic link.
-r filename
True if filename exists and is readable.
-w filename
True if filename exists and is writable.
-n string
True if the length of string is non-zero.
-z string
True if the length of string is zero.
string
True if string is not the empty string.
s1 = s2
True if the strings s1 and s2 are identical.
s1 != s2
True if the strings s1 and s2 are not identical.
n1 -eq n2
True if the numbers n1 and n2 are equal.
n1 -ne n2
True if the numbers n1 and n2 are not equal.
n1 -gt n2
True if the number n1 is greater than n2.
n1 -ge n2
True if the number n1 is greater than or equal to n2.
n1 -lt n2
True if the number n1 is less than n2.
n1 -le n2
True if the number n1 is less than or equal to n2.
! expression
Negates expression, that is, returns true iff expression is false.
expr1 -a expr2
True if both expressions, expr1 and expr2 are true.
expr1 -o expr2
True if either expression, expr1 or expr2 is true.
( expression )
True if expression is true. This allows one to nest expressions.

Tuesday, March 23, 2010

Temporary Files


A script may need to create temporary files to hold intermediate results. The safest way to do this is to use mktemp which returns a currently unused name. The following command creates a new file in /tmp.
newfile=$(mktemp -c -d /tmp)
If a script is prematurely aborted (the user may press ^C for example) it's good practise to remove any temporary files. The trap command can be used to run a tidy-up routine when the script (for whatever reason) exits. To see this in action start the following script then press ^C
newfile=$(mktemp -c -d /tmp)
  trap "echo Removing $newfile ; rm -f $newfile" 0
  sleep 100

grep : Examples

^ (Caret)=match expression at the start of a line, as in ^A.
$ (Question)=match expression at the end of a line, as in A$.
\ (Back Slash)=turn off the special meaning of the next character, as in \^.
[ ] (Brackets)=match any one of the enclosed characters, as in [aeiou]. Use Hyphen "-" for a range, as in [0-9].
[^ ]=match any one character except those enclosed in [ ], as in [^0-9].
. (Period)=match a single character of any value, except end of line.
* (Asterisk)=match zero or more of the preceding character or expression.
\{x,y\}=match x to y occurrences of the preceding.
\{x\}=match exactly x occurrences of the preceding.
\{x,\}=match x or more occurrences of the preceding.

grep smug files{search files for lines with 'smug'}
grep '^smug' files{'smug' at the start of a line}
grep 'smug$' files{'smug' at the end of a line}
grep '^smug$' files{lines containing only 'smug'}
grep '\^s' files{lines starting with '^s', "\" escapes the ^}
grep '[Ss]mug' files{search for 'Smug' or 'smug'}
grep 'B[oO][bB]' files{search for BOB, Bob, BOb or BoB }
grep '^$' files{search for blank lines}
grep '[0-9][0-9]' file{search for pairs of numeric digits}

grep '^From: ' /usr/mail/$USER{list your mail}
grep '[a-zA-Z]'{any line with at least one letter}
grep '[^a-zA-Z0-9]{anything not a letter or number}
grep '[0-9]\{3\}-[0-9]\{4\}'{999-9999, like phone numbers}
grep '^.$'{lines with exactly one character}
grep '"smug"'{'smug' within double quotes}
grep '"*smug"*'{'smug', with or without quotes}
grep '^\.'{any line that starts with a Period "."}
grep '^\.[a-z][a-z]'{line start with "." and 2 lc letters}

grep "unix" *.htm
search all .htm files in the current directory for any reference of  unix and give results similar to the below example text.

Search /etc/passwd for boo user:
$ grep boo /etc/passwd

force grep to ignore word case i.e match boo, Boo, BOO and all other combination with -i option:
$ grep -i "boo" /etc/passwd

search recursively i.e. read all files under each directory for a string "192.168.1.5"
$ grep -r "192.168.1.5" /etc/

select only those lines containing matches that form whole words i.e. match only boo word:
$ grep -w "boo" /path/to/file  

the number of times that the pattern has been matched for each file using -c (count) option:
$ grep -c 'word' /path/to/file  

use -n option, which causes grep to precede each line of output with the number of the line in the text file from which it was obtained:
$ grep -n 'word' /path/to/file 


matches only those lines that do not contain the given word. For example print all line that do not contain the word bar:
$ grep -v bar /path/to/file

print name of hard disk devices:
# dmesg | egrep '(s|h)d[a-z]' 


Display cpu model name:
# cat /proc/cpuinfo | grep -i 'Model'

the -l option to list file name whose contents mention main():
$ grep -l 'main' *.c 

force grep to display output in colors:
$ grep --color vivek /etc/passwd 

prints all lines in the file that begin with the letter a, followed by any one character, then the letters ple.
grep ^a.ple fruitlist.txt

grep for both "FS" and "HR" at the same time, but return lines that contain either entry.
egrep 'HR|FS' myfile

print all lines containing strings "abc" or "def" or both:
grep -E 'abc|def'

print all lines matching exactly "abc" or "def" :
grep -E '^abc$|^def$'

Monday, March 22, 2010

Split a file into multiple files depending on a key value

You have, e.g., a log file to collect logs of different sources;
then you
want to separate the log records depending on a key value defined in the
log data records (the key might identify the source); records with equal
keys should go into the same file.
The program can, of course, be used for other applications than log files,
too.

A nice task for a one-liner... !!

# splitlog.sh - split logfile into multiple logfiles
#
# Depending on some key value in the first column of the logfile defined
# in the argument, it will be split into a set of logfiles, one for each
# key.  If no argument is specified, stdin is used.
#
# Usage: splitlog.sh [ logfile ]
#
# Janis Papanagnou, 2002-11-22

awk -v logfile=${1:-"stdin"} '{ print > logfile"-"$1 }' "$1"

: '--- An example to illustrate...
A file "logfile" with keys A, B, C, and containing the lines, e.g.:
        A 489257 8957 38tgzg75ßhg g5hg 5gh27hg 75gh 5hg    0
        C 8 c83h5g 85gh 5hg5hg h 8h8gh t2h gtj2            1
        B 459 wef2 eruhg uiregn euignutibngtnb ioj         2
        B 489257 8957 38tgzg75ßhg g5hg 5gh27hg 75gh 5hg    3
        A 459 wef2 eruhg uiregn euignutibngtnb ioj         4

will be split into three files "logfile-A" (only lines with key A):
        A 489257 8957 38tgzg75ßhg g5hg 5gh27hg 75gh 5hg    0
        A 459 wef2 eruhg uiregn euignutibngtnb ioj         4
"logfile-B" (only lines with key B):
        B 459 wef2 eruhg uiregn euignutibngtnb ioj         2
        B 489257 8957 38tgzg75ßhg g5hg 5gh27hg 75gh 5hg    3
and "logfile-C" (only lines with key C):
        C 8 c83h5g 85gh 5hg5hg h 8h8gh t2h gtj2            1
----'
      

To reverse a file

#If @ is used anywhere in the file to be reversed, use any other
special character that is not used 

ctr=`cat filename|wc -l `
line=`sed 's/$/@/g' filename |tr -d "\n"`
while test $ctr -ne 0
do
        echo $line|cut -d"@" -f$ctr
        ctr=`expr $ctr - 1`
done
 
-----------------------------------------------------------------------
nl -ba FILE | sort -nr | cut -f2-

wild card characters use & pattern matching on UNIX Shell

As the shell reads each line, it "handles" any special characters. This includes variable evaluation (variables start with a "$)," and filename expansion. Expansion of filenames occurs when the characters "*," "?," or "[" occur in a word. A question mark matches a single character. An asterisk matches any number of characters, including none. Square brackets are used to specify a range or particular combination of characters. Inside square brackets, a hyphen is used to specify a range or characters. Also, if the first character inside the square brackets is an exclamation point, the complement of the range is used. Let me give some examples:
+-------------------------------------------------------------------------------+
|        Table 1     |
|    Examples of filename expansion    |
+-------------------------------------------------------------------------------+
|Pattern Matches        |
|*  Every file in the current directory    |
|?  Files consisting of one character    |
|??  Files consisting of two characters    |
|??*  Files consisting of two or more characters   |
|[abcdefg] Files consisting of a single letter from a to g.  |
|[gfedcba] Same as above       |
|[a-g]  Same as above       |
|[a-cd-g] Same as above       |
|[a-zA-Z0-9] Files that consist of a single letter or number   |
|[!a-zA-Z0-9] Files that consist of a single character not a letter or number |
|[a-zA-Z]* Files that start with a letter     |
|?[a-zA-Z]* Files whose second character matches a letter.   |
|*[0-9]  Files that end with a number     |
|?[0-9]  Two character filename that end with a number   |
|*.[0-9] Files that end with a dot and a number    |
+-------------------------------------------------------------------------------+
As you can see, the dot is not a special character. Filenames may or may not have a dot. UNIX Programers use the dot to standardize on the type of source code of each file, but that is just a convention. There is another convention, which concerns the shell: