Lect11 / 12 - Basic Analysis Flashcards
Explain the following ls commands:
- # ls -l
- # ls -R
- # ls -lt
- # ls -Rlth
- # ls -l => long list (size and dates)
- # ls -R => recursive list
- # ls -lt => sort by modification date
- # ls -Rlth => show human readable size
How does this command looks like:
“find, starting in /mnt/analysis, by name, file with .jpg extension”
find /mnt/analysis -name *jpg
Other Parameters:
- -type f : files
- -type d : directories
- -name : by name
- -iname : by name (case insensitive)
- -many others : read the man page
How can you execute a command based on the find result (e.g. md5hash)?
# find /mnt/analysis -type f -exec md5sum {} \;
or
# find /mnt/analysis -name *.jpg -exec md5sum {} \;
How can you determine the filetype by showing the first line in hex?
# xxd cat_Warmer.jpg | head -n 1
Explain the following grep options:
- # grep -i
- # grep -v
- # grep ^string
- # grep string$
- # grep [char]
- # grep ^$
- :case insensitive
- :reverse grep (exclude)
- :string at beginning of a line
- :string at the end of a line
- :group of characters
- :blank line
How can you use a keyword list with grep?
# grep -abif analysis/searchlist.txt fat_fs.raw > analysis/hits.txt
Parameters are:
- -f <filename> : Use filename as keyword list</filename>
- -i : Case insensitive
- -a : Search binary as text
- -b : Return bit offset of keyword hits
How can you replace control characters with new line characters using grep?
# tr ’[:cntrl:]’ ’\n’ < fat_fs.raw | grep -abif analysis/searchlist.txt
Viewing of different file types. How can you:
- stream the contents of a file to STDOUT
- paging viewer for documents (and command output)
- view MS Office docs from the command line (install first)
- view MS Office xml format files (.docx, etc.)
- for PDF files
- to view graphics from the command line
- cat
- less
- catdoc
- catdocx
- xpdf (or evince)
- xv or display
How to seek into the file to a specified number of bytes?
# xxd -s 75441 fat_fs.raw | head
Name commands to parse structured data:
- display contents of a file to STDOUT
- same, but reverse the contents
- search for patterns and strings in an object
- sort the contents [reverse, unique, etc]
- counts the number of words, lines and bytes in output
- “an output processing tool”.
- Replace or “translate” characters and sets
- “Stream editor” - processes text in a stream
- cat
- tac
- grep
- sort
- wc
- awk
- tr
- sed
How can you sort in alphabetical order and removes duplicates?
# cat names.txt | sort -u
How can you change the field delimiter in awk?
# cat file.txt | awk -F “,” ‘{print $1 $2 $3”\t”$NF}’
or
# awk -F “,” ‘{print $1 $2 $3”\t”$NF}’ <filename></filename>
$NF is normally the last field of a line as it stands for “number of fields”.
awk examples:
How to add tabulator in the output?
cat text.txt | awk ‘{print $1 “\t” $2}’
awk examples?
How to omit the header record?
# awk ‘NR!=1{print $1}’ file1
awk examples:
Print entire file content?
# awk ‘{print $0}’ file1
or
# awk ‘1’ file1
awk examples:
Use special field separator
# awk ‘{print $1,$3}’ FS=”,” file1
awk examples:
Use comma to separate output
awk -F “,” ‘NR!=1{print $1,$3}’ OFS=”,” file1
What is sed?
sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipeline). While in some ways similar to an editor which permits scripted edits (such as ed ), sed works by making only one pass over the input(s), and is consequently more efficient.
sed examples:
Add “fruit” to the beginning of every line
$ sed ‘s/^/Fruit: /’ sample1.txt
sed examples:
Add something to the end of the line
$ sed ‘s/$/ Fruit/’ sample1.txt
sed examples:
To replace a particular char (a => A)
$ sed ‘s/a/A/’ sample1.txt
sed examples:
Replace all occurrences (a => A)
$ sed ‘s/a/A/g’ sample1.txt
sed examples:
Replace the 2nd occurrence (a => A)
$ sed ‘s/a/A/2’ sample1.txt
sed examples:
Replace all occurrences from 2nd occurrence onwards
$ sed ‘s/a/A/2g’ sample1.txt
sed examples:
Replace ‘a’ only in a specific line say 3rd line
$ sed ‘3s/a/A/g’ sample1.txt
sed examples:
Replace or substitute ‘a’ on a range of lines
$ sed ‘1,3s/a/A/g’ sample1.txt
sed examples:
To replace the entire line with something. For example, to replace ‘apple’ with ‘apple is a Fruit’.
$ sed ‘s/.*/& is a Fruit/’ sample1.txt
sed examples:
Using sed, we can also do multiple substitutions. For example, say to replace all ‘a’ to ‘A’, and ‘p’ to ‘P’:
$ sed ‘s/a/A/g; s/p/P/g’ sample1.txt
or
$ sed -e ‘s/a/A/g’ -e ‘s/p/P/g’ sample1.txt
sed examples:
Add header into first line
$ sed -i ‘1i Employee, EmpId’ empFile
sed examples:
Add a line ‘——-‘ after the header line or the 1st line
$ sed -i ‘1a —————’ empFile
sed examples:
Add a trailer line to this file
$ sed -i ‘$a —————’ empFile
sed examples:
Add a record after a particular record
$ sed -i ‘/Hilesh/a Bharti, 1002’ empFile
Logfile analysis:
Show only one filed of the log file
$ cat cisco.log | awk ‘{print $5}’
Output:
%DHCPD-4-PING_CONFLICT:
%DHCPD-4-PING_CONFLICT:
%SYS-5-CONFIG_I:
%SYS-5-CONFIG_I:
Logfile analysis:
Group them together, count them, then sort them from the greatest to least number of occurrences
cat cisco.log | awk ‘{print $5}’| sort | uniq -c | sort -rn
Output:
8 %LINEPROTO-5-UPDOWN:
3 %SYS-5-CONFIG_I:
2 %DHCPD-4-PING_CONFLICT:
1 by
1 Software
1 Cisco
1 %SYS-5-RESTART:
Logfile analysis:
Exclude garbage from output
cat cisco.log | grep %[a-zA-Z]*-[0-9]-[a-zA-Z]* | awk ‘{print $5}’ | sort | uniq -c | sort -rn
Output:
8 %LINEPROTO-5-UPDOWN:
3 %SYS-5-CONFIG_I:
2 %DHCPD-4-PING_CONFLICT:
1 %SYS-5-RESTART:
1 %SSH-5-ENABLED:
