Skip to content

mtumilowicz/linux-terminal-bash-workshop

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

52 Commits
files
files
 
 
.gitignore
.gitignore
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 
answers
answers
 
 
answers-script.sh
answers-script.sh
 
 
script.sh
script.sh
 
 
workshop
workshop
 
 

Repository files navigation

bash-workshop

preface

  • goals of this workshop
    1. introduction to linux filesystem
      • files
      • directories
      • permissions
      • symbolic links
      • hard links
    2. introduction to command line
      • standard commands with examples
      • data streams (with redirection)
    3. workshop tasks are in workshop file and the answers are in answers file

linux filesystem

  • file is a collection of data items stored on disk
    • is always associated with devices like hard disk, floppy disk, USB pen drive and more
    • inode
      • is a file data structure that stores information about any file except its name and data
      • stores the file’s metadata, including all the storage blocks on which the file’s data can be found
        • if you save a file that exceeds a standard block, your computer will find the next available segment on which to store the rest of your file
          • over time, that can get super confusing
  • directory is a group of files
    • two types
      • root directory
        • only one root directory
        • denoted by /
        • root of your entire file system tree
      • sub directory
        • directory under root
    • inside every directory, you will find out two default sub-directories
      • . – current directory
      • .. – pointer to previous directory
        • directory immediately above the current one I am in now
    • some typical directories
      • / - root directory
      • /bin - executable programs which are needed in single user mode
        • contains common commands like cat, cp, cd, ls, etc.
      • /dev - files, which refer to physical devices such as hard disk, keyboard, monitor, mouse and modem etc
      • /etc- all your system configuration files in it
      • /home - for a particular user of the system and consists of individual files
      • /lib - hold shared libraries that are necessary to boot the system and to run the commands
      • /opt - contains packages not part of the Operating System distribution, but provided by an independent source
      • /root - home directory for the root user
      • /tmp - temporary files which may be deleted with no notice
  • permissions
    • four groups X(XXX)(XXX)(XXX)
      • example: drwxr-xr-x
    • group 1: type
      • -: regular file
      • d: directory
      • l: symbolic link
      • s: socket
    • group 2: owner permissions
      • r: read only file permission
      • w: write only file permission
      • x: execute only file permission
      • –: no permission
    • group 3: group permissions
    • group 4: other permissions
  • symbolic link vs hard link
    • symbolic link
      • shortcut that reference to a file instead of its inode value
      • like a shortcut in Windows
      • can be seen as a static link to the last known location of the original file
    • hard links
      • direct reference to a file via its inode
      • only files and not directories

terminal

  • terminal: user@computerName:~$
    • ~ shortcut for current user home directory
  • commands are just text you type in the terminal
  • commands are interpreted by the shell
    • shell - program that interprets the commands and passes them on to the operating system
  • the terminal is the window to the shell
  • commands structure
    • commandName options inputs
    • echo $PATH
      • in what directories the shell will search for a specific command
      • example: which echo
        • echo is /usr/bin/echo
        • you could type /usr/bin/echo 'Hello' or echo 'Hello'
        • type -a echo
          • echo is a shell builtin
      • that's how the shell knows where to find your commands
      • commandNames need to be on the shell's search path
    • there can be input for options
      • cal -A 1 12 2017
        • cal (-A 1) 12 2017: 1 is input for A option
        • cal -A=1 -B=1 12 2017
  • open: ctrl alt t
  • close: ctrl d
  • terminal: user@computerName:~$
    • ~ shortcut for current user home directory
  • search in history
    • ctrl r
  • suggestions: double tap TAB

data streams

  • types
    • standard input
    • standard output
    • standard error
  • redirections
    • data streams can be redirected from their default locations to wherever you wish
    • piping: redirecting the standard output of one commands to the standard input of another
    • example
      • standard input: 0
        • wc -l < test.txt
      • standard output: 1
        • discard stdout: > /dev/null // or 1> /dev/null > is just a shortcut
          • /dev/null is known as the null device
          • whatever you write to /dev/null will be discarded
      • standard error: 2
        • cat -k bla 2>> error.txt // k not existing option
        • no error in terminal
      • &>> result.txt
        • redirect and append both stdout and stderr to file result.txt
  • ways to get data into and out of a command
    • standard input (2)
      • cmd arguments and standard
    • standard output (2)
      • error and standard
    • standard output is connected by default to terminal
    • standard input is connected by default to the keybord

commands

  • awk
    • operations
      1. scans a file line by line
      2. splits each input line into fields
      3. compares line/fields to pattern
      4. performs action(s) on matched lines
    • used for
      • transform (processing, manipulating, formatting) data files
      • searching
      • produce reports
    • awk '{print $1}' file
    • awk '{print $1 "," $2}' file
    • awk -F '.' '{print $1 "," $2}' file // separator
    • awk 'NR>=122 && NR<=129 { print }' // for row numbers
    • awk '$1>5 { print }' // filtering by column
  • cat
    • name derived from its function to concatenate files
    • view content
      • cat file
    • create new file with content
      • cat > FileWithContent then enter then insert content ctr c
    • copy content of 1 file to another
      • cat file > file2
      • cat file1 file2 > file3
    • append data
      • cat file1 >> file2
  • cd
    • change directory
    • cd dirName
    • cd
      • return to home
  • cp source target
    • used for copying files and directories to another location
    • cp a.txt b.txt
    • cp -r source target
      • for copying dirs
  • cut
    • used to extract sections from each line of input — usually from a file
    • cut -d "." -f1 // delimited by . and only first column
      • default delimiter is "TAB"
      • split only by single character
    • cut -d "." -f1-3 // delimited by . and range
    • cut c5- file // since 5 to the end
  • file filename
    • shows type of the file
    • if you change the name of the file to xxx.jpg it still prints png
      • in linux file extensions doesn't matter
  • find haystack/ -type f -name "needle.txt" -exec command {} ;
    • used to find files and directories and perform subsequent operations on them
    • for each result, command {} is executed
      • all occurences of {} are replaced by the filename
  • grep textToSearch file
    • used for searching plain-text data sets for lines that match a regular expression
    • grep -v textToSearch fileToSearchIn // line that does not have textToSearch
    • grep -f - fileToSearchIn
      • -f tells grep to obtain its search pattern from a file and - tells it that this file is actually stdin
  • less file
    • displays the contents of a file or a command output, one page at a time
  • locate *.log
    • search for files
      • doesn't read the file system for the searched file or directory name
      • refers to a database (prepared by the command updatedb)
    • very fast but requires updating
      • updatedb
        • sudo updatedb (to run as an root)
      • sudo - super user do
      • done automatically daily
  • ls
    • list files or directories
    • ls -F (shows differences between dirs and files)
      • directories: Downloads/
      • files: file.txt - without "/"
    • ls -a
      • plus hidden files / directories
  • head/tail file
    • display the beginning of a text file or piped data
  • history
    • list history of commands
    • then !1 - chose first and executes
    • !! - most recent cmd
    • !v latest cmd that starts with v
    • !. latest cmd that start with .
  • mkdir folder
    • create folder
    • mkdir -p /a/b/c
      • create whole path even if dir doesn't exist
      • mkdir a/b/c - error
  • mv source target // for files and directories
    • moves files or directories from one place to another
    • mv foo.txt bar.txt
      • rename foo.txt to bar.txt
    • mv file1.txt file.2.txt file3.txt folder
  • pwd
    • writes the full pathname of the current working directory
    • $oldpwd - prev directory
    • ~+ for pwd
    • ~- for oldpwd
  • rm
    • remove file
  • rm -r
    • remove directory
    • rmdir - removes if empty
  • sed 's/search-pattern/replacement-string/flags' file
    • display and edit data
    • edit
      • sed 's/a/b/' 'a'
        • result: 'b'
      • doesn't modify the original file
      • replace only first occurrence
      • g flag
        • replace all
    • display
      • sed -n '3p' file // 3rd row
      • sed -n '2,4p' file // range of lines
    • sed vs awk
      • sed is used to process and modify text, awk is used for analysis and reporting
  • sort words.txt
    • sort -r words.txt // sort words.txt | tac
    • sort -n numbers.txt // sort by number not lexicographical
    • sort -u number.txt // unique
    • sort -k 2 employee.txt
    • sort -t : -k 2n employee.txt
      • set delimiter to : // default is blank spaces
  • tar
    • create, extract, or list files from a tar file
    • tar -cvf filename.tar file[1..3].txt
      • c – create an archive file
      • v – show the progress of the archive file
      • f – filename of the archive file
      • t – viewing the content of the archive file
    • tar -xvf ourarchive.tar
      • x - extract
    • compression
      • gzip
        • gzip ourarchive.tar // compressed in place
        • gunzip ourarchive.tar // uncompress in place
        • tar -cvzf archive.tar.gz ... // z - gzip
        • tar -xvzf archive.tar.gz // extract gzip
  • tee
    • save a data snapshot without breaking pipeline
      • redirection of stdout breaks pipeline
    • find "4DOS" wikipedia.txt | tee 4DOS.txt | sort > 4DOSsorted.txt
      • searches the file wikipedia.txt for any lines containing the string "4DOS"
      • makes a copy of the matching lines in 4DOS.txt
      • sorts the lines
      • writes them to the output file 4DOSsorted.txt
  • touch filename
    • create empty file
  • wildcards
    • *
      • matches zero or more occurrences
      • ls Documents/ Downloads/ Pictures/
        • same as: ls Do* Pi* // assuming that there are only
    • ?
      • matches single occurrence
      • ls file?.txt
    • [123456]
      • matches 1 place
      • ls file[12345].txt
  • xargs
    • converts input from standard input into arguments to a command
    • cat file | xargs echo // vs echo file

bash

  • conventions
  • bash = most commonly used linux shell today
  • what is script
    • a shell script is a file containing commands for the shell
    • different shells can interpret same text different ways
  • how should bash script begin 
    `#!/bin/bash`

set -o errexit
set -o pipefail
set -o nounset
    • #!/bin/bash
      • first line; means: use bash to interpret this file
      • path to the bash shell
        • to obtain that path type which bash

      • = sharp, ! = bang, #! = shebang

    • set -o errexit
      • exit immediately if a command exits with a non-zero status
    • set -o pipefail
      • return value is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status
    • set -o nounset
      • treat unset variables as an error when substituting
    • other flags: help set
  • declaring/calling structures
    • variable: str="Hello World!"
      • echo ${STR}
    • function 
      hello() {
    local first_arg=$1 // access to first argument
}
      then calling: hello "arg"
  • control structures
    • if 
      if [[ condition ]] // ex. $NUM -gt 5
then
  command // ex. echo "Hello word!"
fi
    • for 
      for i in {0..3}
do
  echo "Number: $i"
done
    • while 
      while [ $i -le 2 ]
do
  echo Number: $i
  ((i++))
done
  • arguments
    • $1 // without default
    • ${1:-foo} // with default
  • options
    • if an invalid option is provided, the option variable is assigned the value ?
      • this behaviour is only true when you prepend the list of valid options with : to disable the default error handling of invalid options
      • it is recommended to always disable the default error handling in your scripts
    • shift $((OPTIND -1))
      • it is common practice to call the shift command at the end of your processing loop to remove options that have already been handled from $@
    • options that themselves have arguments are signified with a :
      • if no argument is provided getopts will set opt to :
    • example 
      while getopts ":d:h" opt; do
  case ${opt} in
    d )
      dir_count=$OPTARG
      ;;
    h )
      echo "Usage:"
      echo "-h                      help"
      echo "-d                      how many directories to create"
      exit 0
      ;;
    \? )
      echo "use -h option for help"
      exit 1
      ;;
    : )
      echo "Invalid option: $OPTARG requires an argument" 1>&2
      exit 1
      ;;
  esac
done
shift $((OPTIND -1))
  • intercepting output of commands
    • $(command)
    • called command substitution
      • bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command
    • subshell starts out as an almost identical copy of the original shell process
      • under the hood, the shell calls the fork system call, which creates a new process whose code and memory are copies
      • in particular, they have the same variables
      • subshell is thus different from executing a script
  • debugging
    • add lines before section you want to debug 
      set -x // print commands and their arguments as they are executed
set -v // print shell input lines as they are read
  • traps
    • allows you to catch signals and execute code when they occur
      • signals are asynchronous notifications that are sent to your script when certain events occur
    • bash provides a psuedo-signal called "EXIT", which is executed when your script exits
      • this can be used to make sure that your script executes some cleanup on exit

About

Introduction to bash scripting.

Resources

Readme

License

GPL-3.0 license
Activity

Stars

1 star

Watchers

2 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Languages