LAB 03 - Disks, Filesystems, filesystem tools, bash and simple scripting, pipes, std in out err
Contents
- 1 links: hard vs soft
- 2 character and block devices
- 3 disks, partitions, filesystems, mounting and options
- 4 usage of filesystem utilities: find, locate, updatedb, e2fsck, tar, gzip, bzip, tee
- 5 bash: variables, expansion, customization, basic scripting (for loop, variables, comments, case statements)
- 6 other shells then bash: /bin/false /bin/sh,ksh,csh,other
- 7 pipes, std in out and error
links: hard vs soft
Hard links: hard links are filesystem pointers that point at a particular piece of data. These links cannot cross filesystem boundries.
usage: ln <source> <destination>
Soft links: soft links are pointers to other files on the filesystem but do not point directly at a piece of data. These links can cross filesystem boundaries.
usage: ln -s <source> <destination>
character and block devices
Character devices: if you do a ls -l /dev, you will see devices like the /dev/tty devices that have a type of 'c' or character. These are devices that have no buffers and the driver communicates via characters with.
crw-rw-rw- 1 root root 5, 0 Nov 14 20:21 tty
Block devices: if you do a ls -l /dev, you will see devices like /dev/sda that have the type of 'b' or block. These devices have buffers and are written to by the driver with entire blocks of data.
brw-rw---- 1 root disk 8, 0 Nov 11 20:41 sda
disks, partitions, filesystems, mounting and options
there are two types of disks we will discuss here, hda and sda:
- hda: hard disk a, or the first hard disk, IDE
- sda: scsi/sata disk a, or the first sata or scsi disk
You can have up to 4 primary partitions, one of which can be extended: Here we will first check out what our blank disk contains, (p)artition information:
# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x9c68108f. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x9c68108f Device Boot Start End Blocks Id System
Now we can make a new DOS disklabel (this process is for 2TB and smaller disks, you need to use gdisk/parted/gparted and GPT for disks larger then 2TB), new d(o)s label, (n)ew partition, (p)rimary partition, (l)ogical partition, (e)xtended partition
Command (m for help): o Building a new DOS disklabel with disk identifier 0x402e399b. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u').
Now we need to create some partitions, we will create 1(linux), 2(extended for logicals), 5(swap) and 6(LVM)
Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1044, default 1): Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-1044, default 1044): +3G Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x402e399b Device Boot Start End Blocks Id System /dev/sdb1 1 393 3156741 83 Linux Command (m for help): n Command action e extended p primary partition (1-4) e Partition number (1-4): 2 First cylinder (394-1044, default 394): Using default value 394 Last cylinder, +cylinders or +size{K,M,G} (394-1044, default 1044): Using default value 1044 Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x402e399b Device Boot Start End Blocks Id System /dev/sdb1 1 393 3156741 83 Linux /dev/sdb2 394 1044 5229157+ 5 Extended Command (m for help): n Command action l logical (5 or over) p primary partition (1-4) l First cylinder (394-1044, default 394): Using default value 394 Last cylinder, +cylinders or +size{K,M,G} (394-1044, default 1044): +2G Command (m for help): n Command action l logical (5 or over) p primary partition (1-4) l First cylinder (656-1044, default 656): Using default value 656 Last cylinder, +cylinders or +size{K,M,G} (656-1044, default 1044): Using default value 1044 Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x402e399b Device Boot Start End Blocks Id System /dev/sdb1 1 393 3156741 83 Linux /dev/sdb2 394 1044 5229157+ 5 Extended /dev/sdb5 394 655 2104483+ 83 Linux /dev/sdb6 656 1044 3124611 83 Linux
Now that the partitions are created, we will be able to change them to be the right type, (t)ype
Command (m for help): t Partition number (1-6): 5 Hex code (type L to list codes): L 0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 39 Plan 9 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 3c PartitionMagic 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 40 Venix 80286 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 4 FAT16 <32M 41 PPC PReP Boot 85 Linux extended c7 Syrinx 5 Extended 42 SFS 86 NTFS volume set da Non-FS data 6 FAT16 4d QNX4.x 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS 4e QNX4.x 2nd part 88 Linux plaintext de Dell Utility 8 AIX 4f QNX4.x 3rd part 8e Linux LVM df BootIt 9 AIX bootable 50 OnTrack DM 93 Amoeba e1 DOS access a OS/2 Boot Manag 51 OnTrack DM6 Aux 94 Amoeba BBT e3 DOS R/O b W95 FAT32 52 CP/M 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 53 OnTrack DM6 Aux a0 IBM Thinkpad hi eb BeOS fs e W95 FAT16 (LBA) 54 OnTrackDM6 a5 FreeBSD ee GPT f W95 Ext'd (LBA) 55 EZ-Drive a6 OpenBSD ef EFI (FAT-12/16/ 10 OPUS 56 Golden Bow a7 NeXTSTEP f0 Linux/PA-RISC b 11 Hidden FAT12 5c Priam Edisk a8 Darwin UFS f1 SpeedStor 12 Compaq diagnost 61 SpeedStor a9 NetBSD f4 SpeedStor 14 Hidden FAT16 <3 63 GNU HURD or Sys ab Darwin boot f2 DOS secondary 16 Hidden FAT16 64 Novell Netware af HFS / HFS+ fb VMware VMFS 17 Hidden HPFS/NTF 65 Novell Netware b7 BSDI fs fc VMware VMKCORE 18 AST SmartSleep 70 DiskSecure Mult b8 BSDI swap fd Linux raid auto 1b Hidden W95 FAT3 75 PC/IX bb Boot Wizard hid fe LANstep 1c Hidden W95 FAT3 80 Old Minix be Solaris boot ff BBT 1e Hidden W95 FAT1 Hex code (type L to list codes): 82 Changed system type of partition 5 to 82 (Linux swap / Solaris) Command (m for help): t Partition number (1-6): 6 Hex code (type L to list codes): 8e Changed system type of partition 6 to 8e (Linux LVM) Command (m for help): p Disk /dev/sdb: 8589 MB, 8589934592 bytes 255 heads, 63 sectors/track, 1044 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x402e399b Device Boot Start End Blocks Id System /dev/sdb1 1 393 3156741 83 Linux /dev/sdb2 394 1044 5229157+ 5 Extended /dev/sdb5 394 655 2104483+ 82 Linux swap / Solaris /dev/sdb6 656 1044 3124611 8e Linux LVM
Now that everything is correct, we will now need to write this. Up to now, all changes were written to memory, if you just quit no changes will be made (fdisk only), (w)rite
Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
And now we can create the filesystems, for regular filesystems, you can use mkfs.FILESYSTEM to create it
# mkfs.ext4 /dev/sdb1 mke2fs 1.41.12 (17-May-2010) Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks 197600 inodes, 789185 blocks 39459 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=809500672 25 block groups 32768 blocks per group, 32768 fragments per group 7904 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done This filesystem will be automatically checked every 35 mounts or 180 days, whichever comes first. Use tune2fs -c or -i to override.
To create swap, you use mkswap
# mkswap /dev/sdb5 Setting up swapspace version 1, size = 2104476 KiB no label, UUID=8b76eb77-7fe8-4c9f-95dd-769e4d1fded6
And now we want to mount the newly created filesystem and swap for use, then check to see that they are there and mounted properly.
# mount /dev/sdb1 /mnt # swapon -a /dev/sdb5 # swapon -s Filename Type Size Used Priority /dev/dm-1 partition 4128760 0 -1 /dev/sdb5 partition 2104472 0 -2 # mount | grep /mnt /dev/sdb1 on /mnt type ext4 (rw) # df -h /mnt Filesystem Size Used Avail Use% Mounted on /dev/sdb1 3.0G 69M 2.8G 3% /mnt
usage of filesystem utilities: find, locate, updatedb, e2fsck, tar, gzip, bzip, tee
Find:
find is a utility to find and execute actions. Simple usage is find <root directory to search> <what to search for> <other actions>
so to find in the current directory and all directories below for a file that is named foo:
find . -name foo
Locate:
locate is a utility that is like find in its ability to find files, but uses a database instead of spidering the entire filesystem and is generally updated nightly. This allows you do find a file quickly without causing large disk io, usage is locate <file to locate>. updatedb is the update utility that builds and updates the locate database. If you say just installed a program, and want to find where it put stuff, you can manually run updatedb and it will update the database to current and then you can run locate with the newest set of data.
e2fsck:
e2fsck is the ext filesystem checker. Generally you run it on a offline filesystem by doing either e2fsck -y /dev/sda1 or e2fsck /dev/sda1. the -y just says yes to everything it wants to do, and generally e2fsck is smarter then the user and just let it do its thing. e2fsck is a system utility that you shouldnt usually have to run unless you are in a recovery situation.
tar:
tar rolls up multiple files into a single 'tarball', but does not compress them.
usage: tar <options> file.tar files to be added
options: (c)reate, e(x)tract, (f)ile, (j)bzip2, g(z)ip (compression discussed later)
gzip:
gnu zip, a standard utility in unix and linux that allows good compression on files and in linux can be combined with tar (see above)
bzip2:
better compression but slower then gzip, available in unix and linux, and can be combined with tar (see above)
tee:
tee allows you to split output between std out and a file, and is great for logging. usage: std out source | tee filename
bash: variables, expansion, customization, basic scripting (for loop, variables, comments, case statements)
Bash:
Bash is the Bourne-again shell and allows for interactive use of the command line, as well as scripting and many core functions to the use of the computer.
Parent shells vs child shells: in some of the examples below, there are references to parent and child shells, a parent shell is the shell you start with, a child shell is one that is called by the parent shell. Example of this below:
$ pstree 20500 bash───pstree $ bash $ pstree 20500 bash───bash───pstree ## here we are inside of the child shell, in the parent shell, as you can see with the pstree $ ps -ef | grep 20500 dan 20500 3119 0 17:18 pts/5 00:00:00 bash ## 3119 is the parent process to our first shell, 20500 is our parent's pid dan 20570 20500 0 17:19 pts/5 00:00:00 bash ## current shell running on pid 20570, parent is 20500
Bash variables:
Bash variables are pretty simple things, they allow you to store a value in a place for later use or abuse. Below is an example of a variable:
foo = 3
this sets the variable 'foo' to equal the value of '3'. This value can then be used by putting a $ at the beginning of the variable:
echo $foo
this above statement would yeild the value of 3
Variables are also used to store information for bash, these are called 'environment variables' and can all be seen by envoking the `env` command or "environment". Below I am editing the PATH variable, which is where bash searches for executables when they are not referenced via absolute or relative paths (IE: if you invoke `ls` you do not need to type `/bin/ls` because of the PATH variable)
$ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games $ PATH=$PATH:/usr/sbin $ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/sbin $ which ls /bin/ls
Exporting variables:
There are local variables (as described above), and global variables. Global variables are allowed to move from parent shell -> child shell, where local variables do not. To build a global variable, you can first build a local variable then export it, or do it in one fell swoop. Example of the difference is below:
$ foo=3 $ echo $foo 3 $ bar=5 $ echo $bar 5 $ export bar $ bash $ echo $foo $ echo $bar 5
More information on bash variables Special variables: These are variables that do special things that are especially useful for scripting, our friends $<number> and $? $1 (or any other number) allows you pull variables from the shell script execution and allows for non-hard-coded variables to be inserted in during each run of the script. $? is the exit status of the previously run command, 0 is good, non-zero is bad. Examples of these are shown in the shell scripting area later.
Bash expansion:
Bash has the ability to interpret special characters and expand values contained within. One of the more useful ones is the {} demonstrated below
$ echo 1 2 3 4 5 1 2 3 4 5 $ echo {1..5} 1 2 3 4 5 $ echo foo{1..5} foo1 foo2 foo3 foo4 foo5
Then we have in-line execution, here we are doing an echo, but then executing the `date` command using back-tics
$ echo "Today's date is: `date`" Today's date is: Thu Nov 21 17:03:45 EST 2013
More information on shell expansion
logic:
the bash shell allows for basic to fairly complex logic using different logical structures such as for loops, if statement, case statements and many other things.
- for loop: the example below counts from 1 -> 5 using a for loop, 'for' is the command, 'i' is the variable that 'in 1 2 3 4 5' places each value upon each iteration of the loop. do is the beginning of the loop, done is the end of the loop and the echo is the one line in the loop (there can be more)
$ for i in 1 2 3 4 5; do > echo $i > done 1 2 3 4 5
- if statement: This is a logic test statement, if allows you to do a test, then upon the outcome of that test, do one thing or another. The first if below with the >'s is an inline script with bash allowing us to just hit return each line and it handles the breaks. The second one with no >'s is the if statement from the history, which is the same statement but without the nice newlines in it.
$ foo=3 $ echo $foo 3 $ if [ $foo -eq 3 ] > then > echo $foo is equal to 3 > else > echo $foo is not equal to 3 > fi 3 is equal to 3 $ foo=44223551 $ if [ $foo -eq 3 ]; then echo $foo is equal to 3; else echo $foo is not equal to 3; fi 44223551 is not equal to 3
- Case statements: Case statements allow you to make things like menus and finite logical structures and are sometimes quite useful.
Comments:
Comments are pretty simple in bash, just put a # at the beginning of anything you do not want bash to interpret and it is now a comment! However if you want to have a # in something that you're doing like a echo statement, you will have to delimit it, and you can do that by putting a \ in front of the special character. This tells the shell to ignore the following character and to just treat it like a normal character.
Shell Scripts:
Shell scripts are in their simplest form, a executable file that contains a list of commands and logic to be run over and over again, shell scripting is especially easy because you can test your code on the command line, before executing it in a shell script.
Here we have a shell script that tries to cat a file, if it fails, the script bails out, if it passes it moves on, then displays our two variables. Note that we spawn an child process with the #!/bin/bash . Also remember that executable files such as our script foo.sh need to have the execute bit for whomever is running it.
#!/bin/bash cat foo.test if [ $? -eq "0" ] then echo previous command ran without a hitch, lets go! else echo bail! bail! exit 1 fi echo $1 is a fine number but $2 is a better variable
now for the run
$ ./foo.sh 32 bob cat: foo.test: No such file or directory bail! bail! $ touch foo.test $ ./foo.sh 32 bob previous command ran without a hitch, lets go! 32 is a fine number but bob is a better variable
other shells then bash: /bin/false /bin/sh,ksh,csh,other
There are a variety of other shells to choose from, however BASH is the default shell for CentOS linux.
sh or "shell" in centos is just a symlink to bash
# sh --version GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu) Copyright (C) 2009 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software; you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. # ls -lh /bin/sh lrwxrwxrwx. 1 root root 4 Oct 8 10:50 /bin/sh -> bash
/bin/false is one that is also sometimes use, and it allows a user account to be active, but with a unusable shell, that will kick a user out immediatly upon attempting to log in.
csh(c shell)/ksh (korn shell)/other you are able to install on a system, and all have their plusses and minuses, but are not covered in this material, just know that they exist.
pipes, std in out and error
| is what is known as a pipe, and allows std out to move to the next command down the line
std out is the output of the program
std error is the error output of the program
std in is the input stream to the program
