Posted & filed under Web Development.

Disclaimer: my notes use content that I researched on the web, filtered and assembled, not exclusively original content written (or rewritten) entirely by me.

 

Basic concepts

 

UNIX is an operating system first developed in the 1960s, and under constant development ever since. UNIX systems also have a graphical user interface (GUI) similar to Microsoft Windows which provides an easy to use environment. However, knowledge of UNIX is required for tasks for which no GUI program is available, for example inside a ssh session with a web server. There are many different versions of UNIX, all sharing common similarities. The most popular varieties of UNIX are Sun Solaris, GNU/Linux, and MacOS X.

 

Linux Shell

 

The shell acts as an interface between the user and the kernel. When a user logs in, e.g. via ssh, the login program checks the username and password, and then starts another program called the shell. The shell is a command line interpreter of the commands the user types in and arranges for them to be carried out. The commands are themselves programs: when they terminate, the shell gives the user another prompt.

 

Why command line?

 

  1. It is powerful: you can use a huge number of existing commands and programs as basic building blocks and combine them to solve complex tasks that would be almost impossible to do using only GUI tools or time consuming to do from scratch with another programming language, like PHP.

  2. It Increases productivity: you can do more with less (tasks automation).

  3. It is flexible: commands can be piped together, e.g. output from one program can be saved in a file or redirected to another program, etc.

  4. It is extensible: you can create your own scripts using the shell scripting language or an interpreted language like PHP or Perl.

  5. It is efficient: you can automate repetitive tasks. Bash scripts allow long, complicated, and/or repetitive tasks to either be simplified or automated completely, Cron will run the scripts at predefined intervals, etc.

  6. It is standardized, so a script you write for one computer will (likely) work on all compliant machines

  7. UNIX was built from the terminal up, almost everything is configurable from the command line.

  8. UNIX utilities are designed to do one thing well.

  9. Use less resources on the server (no graphical drivers).

  10. Accessible from any ssh capable device, with low bandwidth (you can manage a server from the iphone).

  11. Using the command line force you to dig deeper and usually gives you a better understanding of what you are doing and what is going on.

  12. You need to manage and configure a Linux web server (Webmin web interface is not enough)

 

An example of a task that would be longer to do with a GUI interface or a programming language: list and sort all the links inside an html page.

With the shell it just takes one line of code:

 

# wget –quiet -O – http://stackoverflow.com | grep -o 'http://[^”]*' | sort

 

This will output:

 

http://android.stackexchange.com

http://apple.stackexchange.com

http://area51.stackexchange.com

 

Or, even better, this command will download all the images from a site, recursively, and save them in a folder:

 

# wget -r -l 2 -nd -P images/ -A jpg,jpeg,gif,png http://www.guardian.co.uk/

 

Shell prompt

 

root@luigi-N55SL:/home/luigi#

 

Let’s see the different parts of the shell prompt:

 

root: user I am currently logged in the shell environment

@luigi-N55SL: hostname of the server

/home/luigi: the path of the folder I am currently in

# command prompt (it changes if you are a superuser or not)

 

Files and processes

 

Everything in UNIX is either a file or a process. Configuration is saved in text files.

A process is an executing program identified by a unique PID (process identifier).

A file is a collection of data. They are created by users using text editors, running compilers etc.

Examples of files:

 

  • a document

  • the text of a program written in some high-level programming language

  • a directory, containing information about its contents, which may be a mixture of other directories (subdirectories) and ordinary files.


 

Directory structure

 

All the files are grouped together in the directory structure. The file system is arranged in a hierarchical structure, like an inverted tree. The top of the hierarchy is traditionally called root (written as a slash / )

When you first login, your current working directory is your home directory and it is where your personal files and subdirectories are saved.


 

Linux folders structure

 

The most important directories for the developers are:

/ – Root

  • Every single file and directory starts from the root directory.

  • Only root user has write privilege under this directory.

  • Please note that /root is root user’s home directory, which is not same as /.

/etc – Configuration Files

  • Contains configuration files required by all programs, e.g. Apache, PHP, Mysql.

  • This also contains startup and shutdown shell scripts used to start/stop individual programs.

  • For example: /etc/resolv.conf, /etc/logrotate.conf

/var – Variable Files

  • var stands for variable files.

  • Content of the files that are expected to grow can be found under this directory.

  • This includes — system log files (/var/log); packages and database files (/var/lib); emails (/var/mail); print queues (/var/spool); lock files (/var/lock); temp files (/var/tmp); websites (var/www);

/tmp – Temporary Files

  • Directory that contains temporary files created by system and users.

  • Files under this directory are deleted when system is rebooted.

/home – Home Directories

  • Home directories for all users to store their personal files.

 

Basic Linux commands

 

Linux cheatsheet:

http://files.fosswire.com/2007/08/fwunixref.pdf


 

Connect to a Linux box

 

Use an ssh connection command to log in a Linux machine. The login can be user/password based, e.g. Rackspace machines, or private key based, e.g. Amazon machines.

Use Putty to simplify the connection and manage the servers informations.

 

From the shell, run the ssh command to log into the server:

 

Rackspace:

# ssh root@312.213.23.56

 

Amazon

# ssh -i “path/to/pem/file” ubuntu@232.43.123.21


 

Get to know your Linux environment

 

Show logged in users and system info

# w

 

Show system and kernel info

# uname -a

 

Show distribution version

# cat /etc/*release*

 

Show system date

# date

 

Become superuser (root)

# sudo su

 

Getting help

 

All commands offer a basic user guide:

 

# <command> –help

or

# man <command>

 

Copy and paste

 

To copy and paste text from and into a GUI application (browser, text editor, etc), first select with the mouse the text in the shell:

 

CTRL + SHIFT + c = copy

CTRL + SHIFT + v = paste

 

Or use the dropdown menu in the shell window.

 

File management

 

ls: The ls command ( lowercase L and lowercase S ) lists the contents of your current working directory.

 

ls lists only those ones whose name does not begin with a dot (.) Files beginning with a dot (.) are known as hidden files.

 

To list all files in your home directory including those whose names begin with a dot, type

ls -a

 

In Ubuntu you usually have a predefined “alias” command:

ll (2 lowercase L)

This will list in a cleaner format all files and folders in the working directory (excluding hidden files).

 

ls options

-a

Show all (including hidden)

-R

Recursive list

-r

Reverse order

-t

Sort by last modified

-S

Sort by file size

-l

Long listing format

-1

One file per line

-m

Comma-­sep­arated output

 

cd: change the current working directory

The current working directory may be thought of as the directory you are in, i.e. your current position in the file-system tree.

Move to the parent folder

# cd ..

Move to the parent of the parent folder

# cd ../../

 

pwd: Print working directory path

 

mkdir: Creates a new directory:

# mkdir <directory>

Create nested folders

# mkdir -p dir1/dir2/dir3

 

cp: Copy a file, without touching the original file:

# cp <file1> <file2>

# cp -r dir1/* dir2/  

(will copy recursively  all files in dir1 to dir2)

 

mv: Move a file, or rename

# mv file1 file2

(rename the file)

# mv file1 dir/file2

(move the file in dir and rename it)

 

rm: delete a file or folder (caution: you cannot restore a deleted file in Linux)

# rm file1

# rm -r myfolder

(delete the directory myfolder and all its content)

 

Print the folder structure and content in a hierarchical way:

# tree

The tree command could not be installed by default, to install:

# sudo ap-get install tree

 

File editing

 

cat: The cat command, short for concatenate, is useful for viewing and adding to text files. The simple command cat FILENAME displays the contents of the file. Using cat FILENAME file adds the contents of the first file to the second.

 

nano: Nano is a simple text editor for the command line. To open a file, use nano filename. Commands listed at the bottom of the screen are accessed via pressing Ctrl followed by the letter.

 

Nano Shortcuts

Files

 

Ctrl-R

Read file

Ctrl-O

Save file

Ctrl-X

Close file

Cut and Paste

 

ALT-A

Start marking text

CTRL-K

Cut marked text or current line

CTRL-U

Paste text

Navigate File

 

ALT-/

End of file

CTRL-A

Beginning of line

CTRL-E

End of line

CTRL-C

Show line number

CTRL-_

Go to line number

Search File

 

CTRL-W

Find

ALT-W

Find next

CTRL-\

Search and replace

 

less: The less command is used for viewing text files as well as standard output. A common usage is to pipe another command through less to be able to see all the output, such as ls | less. It shows one page at a time, press <space> or pg keys to move between pages.

 

head: Show only the first n lines of a file.

# head -n myfile

Useful for big files, since it won’t load the file in memory, for example to check the header of a mysql db dump file.

 

tail: Show the last n lines of a file.

# tail -n myfile

Useful for logs, that append new rows at the end.


 

Search

 

find: Search files recursively based on a name pattern, starting from the current folder

# find . -name mystring -print

Search and print all files named mystring

 

grep: The grep command allows you to search inside a number of files for a particular search pattern and then print matching lines. For example, grep blah file will search for the text "blah" in the file and then print any matching lines.

 

sed: The sed (or Stream EDitor) command allows search and replace of a particular string in a file.

 

Options

grep pattern files

Search for pattern in files

grep -i

Case insens­itive search

grep -r

Recursive search

grep -v

Inverted search

grep -o

Show matched part of file only

find /dir/ -name name*

Find files starting with name in dir

find /dir/ -user name

Find files owned by name in dir

find /dir/ -mmin num

Find files modifed less than num minutes ago in dir

whereis command

Find binary / source / manual for command

locate file

Find file (quick search of system index)

 

Using Wildcards

 

Sometimes you need to look at or use multiple files at the same time. You can use a series of wildcards:

 

* matches any number of characters. For example, *.rar matches any file with the ending .rar.

 

? matches any single character. For example, ?.rar matches a.rar but not ab.rar.

 

[characters] matches any of the characters within the brackets. For example, [ab].rar matches a.rar and b.rar but not c.rar.

 

[!characters] matches any characters that are not listed. For example, [!ab].rar matches c.rar but not a.rar or b.rar.

 

Redirection and piping

 

# cmd > file

Command standard output written to file

 

# cmd >> file

Command standard output appended at the end of file

 

# ls -la > directory_content.txt

Saves the list of files in the the file directory_content.txt

 

# cmd1 | cmd2

Output of cmd1 is the input for cmd2

 

# ls | sort -r

Takes the file listing and sort in reverse order


 

File permission

 

Each file (and directory) has access rights, which may be found using ls -l :

 

-rwxrw-r– 1 userx groupx 2450 Sept12 11:23 filex

 

In the left-hand column is a 10 symbol string consisting of the symbols d, r, w, x, -, and, occasionally, s or S. If d is present, it will be at the left hand end of the string, and indicates a directory: otherwise – will be the starting symbol of the string.

The 9 remaining symbols indicate the permissions, or access rights, and are taken as three groups of 3.

  • The left group of 3 gives the file permissions for the user that owns the file (or directory) (userx in the above example);

  • the middle group gives the permissions for the group of people to whom the file (or directory) belongs (groupx in the above example);

  • the rightmost group gives the permissions for all others.

The symbols r, w, etc., have slightly different meanings depending on whether they refer to a simple file or to a directory.

Access rights on files

  • r (or -), indicates read permission (or otherwise), that is, the presence or absence of permission to read and copy the file

  • w (or -), indicates write permission (or otherwise), that is, the permission (or otherwise) to change a file

  • x (or -), indicates execution permission (or otherwise), that is, the permission to execute a file, where appropriate

Access rights on directories

  • r allows users to list files in the directory;

  • w means that users may delete files from the directory or move files into it;

  • x means the right to access files in the directory. This implies that you may read files in the directory provided you have read permission on the individual files.

So, in order to read a file, you must have execute permission on the directory containing that file, and hence on any directory containing that directory as a subdirectory, and so on, up the tree.

 

Examples

 

rwxrwxrwx

a file that everyone can read, write and execute (and delete).

-rw——-

a file that only the owner can read and write – no-one else

can read or write and no-one has execution rights.

 

 

chmod: change file or folder permission

 

For example, to remove read write and execute permissions on the file biglist for the group and others, type

# chmod go-rwx biglist

 

To give read and write permissions on the file biglist to all

# chmod a+rw biglist


 

File Permission Numbers

 

The first digit is the owner permis­sion, the second the group and the third for everyone.

Calculate each of the three permission digits by adding the numeric values of the permis­sions below.

 

4

read (r)

2

write (w)

1

execute (x)

 

 

# chmod 775 file

Change mode of file to 775

 

# chmod -R 600 folder

Recurs­ively chmod folder to 600

 

chown:Change file owner to user and group to group

# chown user­:g­roup file

Change recursively owner and group  

# chwon -R user:group <folder>

 

System administration

 

df: The df command displays filesystem disk space usage for all partitions. The command df -h is probably the most useful. It uses megabytes (M) and gigabytes (G) instead of blocks to report. (-h means "human-readable.")

 

free: The free command displays the amount of free and used memory in the system. For example, free -m gives the information using megabytes, which is probably most useful for current computers.

 

top: The top command displays information on your Linux system, running processes, and system resources, including the CPU, RAM, swap usage, and total number of tasks being run. To exit top, press Q.

 

The program will then generate a whole screen display with a listing of the current processes, which is constantly updated. The list is by default sorted according to how much CPU cycles the processes are using. At the top of the screen are statistics of CPU and memory usage, as well as the number of users and how long the system has been up.

With the top utility running, you can now change the format of the display and the ordering of the processes with simple key strokes. For example, if you hit "M" (capital m) the processes will be sorted by memory usage. Hitting "P" will list the processes by CPU usage.

 

Variables

 

uptime

This line displays the time the system has been up, and the three load averages for the system. The load averages are the average number of process ready to run during the last 1, 5 and 15 minutes. This line is just like the output of uptime. The uptime display may be toggled by the interactive l command.

processes

The total number of processes running at the time of the last update. This is also broken down into the number of tasks which are running, sleeping, stopped, or undead. The processes and states display may be toggled by the t interactive command.

CPU states

Shows the percentage of CPU time in user mode, system mode, niced tasks, iowait and idle.

Mem

Statistics on memory usage, including total available memory, free memory, used memory, shared memory, and memory used for buffers. The display of memory information may be toggled by the m interactive command.

Swap

Statistics on swap space, including total swap space, available swap space, and used swap space. This and Mem are just like the output of free.

 

ps: The ps command allows you to view the processes running on the machine:

 

List all processes currently running

# ps aux

 

To check if a process named “apache” is running:

# ps aux | grep apache

 

Networking

 

ssh: opens an ssh connection to a server

scp: copy files between server using an ssh connection

wget: download the content of a web page/link

lynx: text-based browser

 

References

Unix tutorial for beginners: http://www.ee.surrey.ac.uk/Teaching/Unix/

Leave a Reply

  • (will not be published)