LAB 03 - Disks, Filesystems, filesystem tools, bash and simple scripting, pipes, std in out err

From JaxHax
Jump to: navigation, search

links: hard vs soft

Hard links: hard links are filesystem pointers that point at a particular piece of data. These links cannot cross filesystem boundries.

usage: ln <source> <destination>

Soft links: soft links are pointers to other files on the filesystem but do not point directly at a piece of data. These links can cross filesystem boundaries.

usage: ln -s <source> <destination>

character and block devices

Character devices: if you do a ls -l /dev, you will see devices like the /dev/tty devices that have a type of 'c' or character. These are devices that have no buffers and the driver communicates via characters with.

crw-rw-rw-  1 root root        5,   0 Nov 14 20:21 tty

Block devices: if you do a ls -l /dev, you will see devices like /dev/sda that have the type of 'b' or block. These devices have buffers and are written to by the driver with entire blocks of data.

brw-rw----  1 root disk        8,   0 Nov 11 20:41 sda

disks, partitions, filesystems, mounting and options

there are two types of disks we will discuss here, hda and sda:

  • hda: hard disk a, or the first hard disk, IDE
  • sda: scsi/sata disk a, or the first sata or scsi disk

You can have up to 4 primary partitions, one of which can be extended: Here we will first check out what our blank disk contains, (p)artition information:

# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x9c68108f.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
 
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').
 
Command (m for help): p
 
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x9c68108f
 
   Device Boot      Start         End      Blocks   Id  System

Now we can make a new DOS disklabel (this process is for 2TB and smaller disks, you need to use gdisk/parted/gparted and GPT for disks larger then 2TB), new d(o)s label, (n)ew partition, (p)rimary partition, (l)ogical partition, (e)xtended partition

Command (m for help): o
Building a new DOS disklabel with disk identifier 0x402e399b.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
 
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
 
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Now we need to create some partitions, we will create 1(linux), 2(extended for logicals), 5(swap) and 6(LVM)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1044, default 1): 
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1044, default 1044): +3G
 
Command (m for help): p
 
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x402e399b
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         393     3156741   83  Linux
 
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Partition number (1-4): 2
First cylinder (394-1044, default 394): 
Using default value 394
Last cylinder, +cylinders or +size{K,M,G} (394-1044, default 1044): 
Using default value 1044
 
Command (m for help): p
 
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x402e399b
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         393     3156741   83  Linux
/dev/sdb2             394        1044     5229157+   5  Extended
 
Command (m for help): n
Command action
   l   logical (5 or over)
   p   primary partition (1-4)
l
First cylinder (394-1044, default 394): 
Using default value 394
Last cylinder, +cylinders or +size{K,M,G} (394-1044, default 1044): +2G
 
Command (m for help): n
Command action
   l   logical (5 or over)
   p   primary partition (1-4)
l
First cylinder (656-1044, default 656): 
Using default value 656
Last cylinder, +cylinders or +size{K,M,G} (656-1044, default 1044): 
Using default value 1044
 
Command (m for help): p
 
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x402e399b
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         393     3156741   83  Linux
/dev/sdb2             394        1044     5229157+   5  Extended
/dev/sdb5             394         655     2104483+  83  Linux
/dev/sdb6             656        1044     3124611   83  Linux

Now that the partitions are created, we will be able to change them to be the right type, (t)ype

Command (m for help): t
Partition number (1-6): 5
Hex code (type L to list codes): L
 
 0  Empty           24  NEC DOS         81  Minix / old Lin bf  Solaris        
 1  FAT12           39  Plan 9          82  Linux swap / So c1  DRDOS/sec (FAT-
 2  XENIX root      3c  PartitionMagic  83  Linux           c4  DRDOS/sec (FAT-
 3  XENIX usr       40  Venix 80286     84  OS/2 hidden C:  c6  DRDOS/sec (FAT-
 4  FAT16 <32M      41  PPC PReP Boot   85  Linux extended  c7  Syrinx         
 5  Extended        42  SFS             86  NTFS volume set da  Non-FS data    
 6  FAT16           4d  QNX4.x          87  NTFS volume set db  CP/M / CTOS / .
 7  HPFS/NTFS       4e  QNX4.x 2nd part 88  Linux plaintext de  Dell Utility   
 8  AIX             4f  QNX4.x 3rd part 8e  Linux LVM       df  BootIt         
 9  AIX bootable    50  OnTrack DM      93  Amoeba          e1  DOS access     
 a  OS/2 Boot Manag 51  OnTrack DM6 Aux 94  Amoeba BBT      e3  DOS R/O        
 b  W95 FAT32       52  CP/M            9f  BSD/OS          e4  SpeedStor      
 c  W95 FAT32 (LBA) 53  OnTrack DM6 Aux a0  IBM Thinkpad hi eb  BeOS fs        
 e  W95 FAT16 (LBA) 54  OnTrackDM6      a5  FreeBSD         ee  GPT            
 f  W95 Ext'd (LBA) 55  EZ-Drive        a6  OpenBSD         ef  EFI (FAT-12/16/
10  OPUS            56  Golden Bow      a7  NeXTSTEP        f0  Linux/PA-RISC b
11  Hidden FAT12    5c  Priam Edisk     a8  Darwin UFS      f1  SpeedStor      
12  Compaq diagnost 61  SpeedStor       a9  NetBSD          f4  SpeedStor      
14  Hidden FAT16 <3 63  GNU HURD or Sys ab  Darwin boot     f2  DOS secondary  
16  Hidden FAT16    64  Novell Netware  af  HFS / HFS+      fb  VMware VMFS    
17  Hidden HPFS/NTF 65  Novell Netware  b7  BSDI fs         fc  VMware VMKCORE 
18  AST SmartSleep  70  DiskSecure Mult b8  BSDI swap       fd  Linux raid auto
1b  Hidden W95 FAT3 75  PC/IX           bb  Boot Wizard hid fe  LANstep        
1c  Hidden W95 FAT3 80  Old Minix       be  Solaris boot    ff  BBT            
1e  Hidden W95 FAT1
Hex code (type L to list codes): 82
Changed system type of partition 5 to 82 (Linux swap / Solaris)
 
Command (m for help): t
Partition number (1-6): 6
Hex code (type L to list codes): 8e
Changed system type of partition 6 to 8e (Linux LVM)
 
Command (m for help): p
 
Disk /dev/sdb: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x402e399b
 
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1         393     3156741   83  Linux
/dev/sdb2             394        1044     5229157+   5  Extended
/dev/sdb5             394         655     2104483+  82  Linux swap / Solaris
/dev/sdb6             656        1044     3124611   8e  Linux LVM

Now that everything is correct, we will now need to write this. Up to now, all changes were written to memory, if you just quit no changes will be made (fdisk only), (w)rite

Command (m for help): w
The partition table has been altered!
 
Calling ioctl() to re-read partition table.
Syncing disks.

And now we can create the filesystems, for regular filesystems, you can use mkfs.FILESYSTEM to create it

# mkfs.ext4 /dev/sdb1
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
197600 inodes, 789185 blocks
39459 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=809500672
25 block groups
32768 blocks per group, 32768 fragments per group
7904 inodes per group
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912
 
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
 
This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

To create swap, you use mkswap

# mkswap /dev/sdb5
Setting up swapspace version 1, size = 2104476 KiB
no label, UUID=8b76eb77-7fe8-4c9f-95dd-769e4d1fded6

And now we want to mount the newly created filesystem and swap for use, then check to see that they are there and mounted properly.

# mount /dev/sdb1 /mnt
# swapon -a /dev/sdb5
# swapon -s
Filename				Type		Size	Used	Priority
/dev/dm-1                               partition	4128760	0	-1
/dev/sdb5                               partition	2104472	0	-2
# mount | grep /mnt
/dev/sdb1 on /mnt type ext4 (rw)
# df -h /mnt
Filesystem            Size  Used Avail Use% Mounted on
/dev/sdb1             3.0G   69M  2.8G   3% /mnt

usage of filesystem utilities: find, locate, updatedb, e2fsck, tar, gzip, bzip, tee

Find:
find is a utility to find and execute actions. Simple usage is find <root directory to search> <what to search for> <other actions> so to find in the current directory and all directories below for a file that is named foo: find . -name foo

Locate:
locate is a utility that is like find in its ability to find files, but uses a database instead of spidering the entire filesystem and is generally updated nightly. This allows you do find a file quickly without causing large disk io, usage is locate <file to locate>. updatedb is the update utility that builds and updates the locate database. If you say just installed a program, and want to find where it put stuff, you can manually run updatedb and it will update the database to current and then you can run locate with the newest set of data.

e2fsck:
e2fsck is the ext filesystem checker. Generally you run it on a offline filesystem by doing either e2fsck -y /dev/sda1 or e2fsck /dev/sda1. the -y just says yes to everything it wants to do, and generally e2fsck is smarter then the user and just let it do its thing. e2fsck is a system utility that you shouldnt usually have to run unless you are in a recovery situation.

tar:
tar rolls up multiple files into a single 'tarball', but does not compress them. usage: tar <options> file.tar files to be added options: (c)reate, e(x)tract, (f)ile, (j)bzip2, g(z)ip (compression discussed later)

gzip:
gnu zip, a standard utility in unix and linux that allows good compression on files and in linux can be combined with tar (see above)

bzip2:
better compression but slower then gzip, available in unix and linux, and can be combined with tar (see above)

tee:
tee allows you to split output between std out and a file, and is great for logging. usage: std out source | tee filename

bash: variables, expansion, customization, basic scripting (for loop, variables, comments, case statements)

Bash:
Bash is the Bourne-again shell and allows for interactive use of the command line, as well as scripting and many core functions to the use of the computer. Parent shells vs child shells: in some of the examples below, there are references to parent and child shells, a parent shell is the shell you start with, a child shell is one that is called by the parent shell. Example of this below:

$ pstree 20500
bash───pstree
$ bash
$ pstree 20500
bash───bash───pstree ## here we are inside of the child shell, in the parent shell, as you can see with the pstree
$ ps -ef | grep 20500
dan      20500  3119  0 17:18 pts/5    00:00:00 bash  ## 3119 is the parent process to our first shell, 20500 is our parent's pid
dan      20570 20500  0 17:19 pts/5    00:00:00 bash  ## current shell running on pid 20570, parent is 20500


Bash variables:
Bash variables are pretty simple things, they allow you to store a value in a place for later use or abuse. Below is an example of a variable: foo = 3 this sets the variable 'foo' to equal the value of '3'. This value can then be used by putting a $ at the beginning of the variable: echo $foo this above statement would yeild the value of 3 Variables are also used to store information for bash, these are called 'environment variables' and can all be seen by envoking the `env` command or "environment". Below I am editing the PATH variable, which is where bash searches for executables when they are not referenced via absolute or relative paths (IE: if you invoke `ls` you do not need to type `/bin/ls` because of the PATH variable)

$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
$ PATH=$PATH:/usr/sbin
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin
$ which ls
/bin/ls

Exporting variables:
There are local variables (as described above), and global variables. Global variables are allowed to move from parent shell -> child shell, where local variables do not. To build a global variable, you can first build a local variable then export it, or do it in one fell swoop. Example of the difference is below:

$ foo=3
$ echo $foo
3
$ bar=5
$ echo $bar
5
$ export bar
$ bash
$ echo $foo
 
$ echo $bar
5

More information on bash variables Special variables: These are variables that do special things that are especially useful for scripting, our friends $<number> and $? $1 (or any other number) allows you pull variables from the shell script execution and allows for non-hard-coded variables to be inserted in during each run of the script. $? is the exit status of the previously run command, 0 is good, non-zero is bad. Examples of these are shown in the shell scripting area later.

Bash expansion:
Bash has the ability to interpret special characters and expand values contained within. One of the more useful ones is the {} demonstrated below

$ echo 1 2 3 4 5
1 2 3 4 5
$ echo {1..5}
1 2 3 4 5
$ echo foo{1..5}
foo1 foo2 foo3 foo4 foo5

Then we have in-line execution, here we are doing an echo, but then executing the `date` command using back-tics

$ echo "Today's date is: `date`"
Today's date is: Thu Nov 21 17:03:45 EST 2013

More information on shell expansion


logic:
the bash shell allows for basic to fairly complex logic using different logical structures such as for loops, if statement, case statements and many other things.

  • for loop: the example below counts from 1 -> 5 using a for loop, 'for' is the command, 'i' is the variable that 'in 1 2 3 4 5' places each value upon each iteration of the loop. do is the beginning of the loop, done is the end of the loop and the echo is the one line in the loop (there can be more)
$ for i in 1 2 3 4 5; do
> echo $i
> done
1
2
3
4
5

More on loops

  • if statement: This is a logic test statement, if allows you to do a test, then upon the outcome of that test, do one thing or another. The first if below with the >'s is an inline script with bash allowing us to just hit return each line and it handles the breaks. The second one with no >'s is the if statement from the history, which is the same statement but without the nice newlines in it.
$ foo=3
$ echo $foo
3
$ if [ $foo -eq 3 ]
> then
> echo $foo is equal to 3
> else
> echo $foo is not equal to 3
> fi
3 is equal to 3
$ foo=44223551
$ if [ $foo -eq 3 ]; then echo $foo is equal to 3; else echo $foo is not equal to 3; fi
44223551 is not equal to 3

More on if statements

  • Case statements: Case statements allow you to make things like menus and finite logical structures and are sometimes quite useful.


Comments:
Comments are pretty simple in bash, just put a # at the beginning of anything you do not want bash to interpret and it is now a comment! However if you want to have a # in something that you're doing like a echo statement, you will have to delimit it, and you can do that by putting a \ in front of the special character. This tells the shell to ignore the following character and to just treat it like a normal character.


Shell Scripts:
Shell scripts are in their simplest form, a executable file that contains a list of commands and logic to be run over and over again, shell scripting is especially easy because you can test your code on the command line, before executing it in a shell script. Here we have a shell script that tries to cat a file, if it fails, the script bails out, if it passes it moves on, then displays our two variables. Note that we spawn an child process with the #!/bin/bash . Also remember that executable files such as our script foo.sh need to have the execute bit for whomever is running it.

#!/bin/bash
 
cat foo.test
 
if [ $? -eq "0" ]
then 
echo previous command ran without a hitch, lets go!
else
echo bail! bail!
exit 1
fi
 
echo $1 is a fine number but $2 is a better variable

now for the run

$ ./foo.sh 32 bob
cat: foo.test: No such file or directory
bail! bail!
$ touch foo.test
$ ./foo.sh 32 bob
previous command ran without a hitch, lets go!
32 is a fine number but bob is a better variable

other shells then bash: /bin/false /bin/sh,ksh,csh,other

There are a variety of other shells to choose from, however BASH is the default shell for CentOS linux.

sh or "shell" in centos is just a symlink to bash

# sh --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
 
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
# ls -lh /bin/sh
lrwxrwxrwx. 1 root root 4 Oct  8 10:50 /bin/sh -> bash

/bin/false is one that is also sometimes use, and it allows a user account to be active, but with a unusable shell, that will kick a user out immediatly upon attempting to log in.

csh(c shell)/ksh (korn shell)/other you are able to install on a system, and all have their plusses and minuses, but are not covered in this material, just know that they exist.

pipes, std in out and error

| is what is known as a pipe, and allows std out to move to the next command down the line

std out is the output of the program

std error is the error output of the program

std in is the input stream to the program