Debug School: Pavani

BASH SCRIPT AND ITS COMMANDS

Pavani — Tue, 13 Dec 2022 17:07:40 +0000

BASH SCRIPT:
The Linux Bash is also known as 'Bourne-again Shell. 'A Bash script is a plain text file. This file contains different commands for step-by-step execution. These commands can be written directly into the command line but from a reusability perceptive it is useful to store all of the inter-related commands for a specific task in a single file. We can use that file for executing the set of commands one or more times as per our requirements.
Example:
#!/bin/bash/
# it is a function
my function () {
hello world
}

function call

my function

Applications of Bash scripts:

Manipulating files
Executing routine tasks like Backup operation
Automation Advantages of Bash Scripts:

It is simple.
It helps to avoid doing repetitive tasks
Easy to use
Frequently performed tasks can be automated
A sequence of commands can be run as a single command.
Disadvantages of Bash Scripts:
Any mistake while writing can be costly.
A new process launched for almost every shell command executed.
Slow execution speed
Compatibility problems between different platforms.
Some commands of bash:
1) Basename: This command is used to strip directory and suffix from the file names.
syntax: $ basename filename.txt .txt
2) cal: displays the calender
syntax: $ cal -y year
3) df: disk file system. It will show the disk space usage in a tabular form. The df command is useful for discovering the available free space on a system or file system.
syntax: $ df
-> df -h: It displays the disk space in human readable form
-> df -T: It displays the file system type.
4) diff: diff stands for difference. This command is used to display the differences in the files by comparing the files line by line. This command tells us which lines in one file have is to be changed to make the two files identical.
-> diff uses certain special symbols and instructions that are required to make two files identical.
a- add
c- change
d- delete
syntax: $ diff file1.txt file2.txt
-> -c : To view the differences in the context mode use -c option.
-> -u: To view the differences in the unified mode use -u option.
5)dir: This command is used to list the contents of a directory.
syntax: $dir
-> dir -a: displays all the hidden files(starting with .) along with two files denoted by . and .. which signals for current and previous directory respectively.
-> dir -A: It is similar to -a option except that it does not display files that signals the current directory and previous directory.
-> dir -l --author: displays the author of all the files.
6) dmesg: driver message or display message used to examine the kernel ring buffer and print the message buffer of kernel. The output of this command contains the messages produced by the device drivers.
syntax: dmesg
-> dmesg | grep word-
-> dmesg -t- -t specifies with timestamps.
7) du- du command, short for disk usage, is used to estimate file space usage. The du command can be used to track the files and directories which are consuming excessive amount of space on hard disk drive.
-> du -h: display the disk size in human readable form.
-> du -a: printing all files including directories.
-> du -c: print total size
-> du -s: get the summary of file system
8) egrep: egrep is a pattern searching command which belongs to the family of grep functions.
syntax: egrep pattern file name
-> -c: Used to counts and prints the number of lines that matched the pattern and not the lines.
-> -v: It prints the lines that does not match with the pattern.
-> -o: prints the matched part of the line but not the entire line.
9) eval- eval is a built-in Linux command which is used to execute arguments as a shell command. It combines arguments into a single string and uses it as an input to the shell and execute the commands.
syntax: eval arg
example: eval c="clear"
10) expand: expand which allows you to convert tabs into spaces in a file and when no file is specified it reads from standard input.
syntax: expand filename
11) expr: It evaluates the given expression and displays the output
syntax: expr expression
12) fold: fold command in Linux wraps each line in an input file to fit a specified width and prints it to the standard output. Default width is 80
syntax: fold filename
-> fold -w[n] filename: we can limit width of the width by number of the columns. we can change the default width of 80.
-> fold -b[n] filename: this option of fold command is used to limit the width of the output by the number of bytes rather than the number of columns.
-> -s: This option is used to break the lines on spaces so that words are not broken. If a segment of the line contains a blank character within the first width column positions, break the line after the last such blank character meeting the width constraints.
syntax: fold -w[n] -s filename
13) free: free command which displays the total amount of free space available along with the amount of memory used and swap memory in the system, and also the buffers used by the kernel.
syntax: $free
14) gawk: gawk command in Linux is used for pattern scanning and processing language.
$gawk -F'{print $1} filename
15) id: this command used to find the user identity, group identity.
syntax: $ id
16) join: join command is used to join the two files based on a key field present in both the files. The input file can be separated by white space or any delimiter.
syntax: join file1.txt file2.txt
17) look: The look command in Linux shows the lines beginning with a given string. This command also uses binary search if the file is sorted. If file is not specified, the file /usr/share/dict/words is used.
syntax: $look string filename
18) ls: this command lists contents in a directory.
syntax: ls -l
19) more: more command is used to view the text files in the command prompt, displaying one screen at a time in case the file is large. The more command also allows the user do scroll up and down through the page.
syntax: more filename
20) nl: This command is used to count the numbering the lines syntax: $ nl filename
21) paste: It is used to join files horizontally by outputting lines consisting of lines from each file specified, separated by tab as delimiter, to the standard output.
syntax: paste file1 file2
22) ps- process status ps command is used to list the currently running processes and their PIDs along with some other information depends on different options
syntax: ps -a
23) rm: remove files
syntax: $ rm filename
-> $ rm -f filename: remove files forcefully without prompting for confirmation.
24) rmdir: remove empty directories from the file system
syntax: $rmdir directory
25) SCP (secure copy): The SCP command or secure copy allows secure transferring of files in between the local host and the remote host or between two remote hosts. It uses the same authentication and security as it is used in the Secure Shell (SSH) protocol. SCP is known for its simplicity, security and pre-installed availability.
syntax: $ scp
26) seq: is used to generate numbers from FIRST to LAST in steps of INCREMENT.
syntax: seq first increment last
seq 0 3 20
27) set: It is used to set or unset specific flags and settings inside the shell environment. It can be used to change or display the shell attributes and parameters.
-> -a- use to mark variables that are created or modified or created for export.
-> -b- use to notify the termination of the job.
-> -e- use to exit when the command exits with a non-zero status.
-> -f- it disables the file name generation known as
globbing
-> -h- It saves the location of the command where it got looked.
->-k- It places all assignment arguments in the environment variable of a command.
example: set -x
echo apple orange mango
27) sleep: This command helps delaying the execution
syntax: sleep 10s
s- second
m- minutes
h- hours
28) sort: SORT command is used to sort a file, arranging the records in a particular order. SORT command sorts the contents of a text file, line by line.
syntax: sort filename.txt
-> sort -r filename.txt: sorts the file in reverse order.
29) sudo: super user do. It will run the command with elevated privileges. This is equal to run as administrator command in windows.
syntax: sudo -l
30) sum: sum command in Linux is used to find checksum and count the blocks in a file. Basically, this command is used to show the checksum and block count for each specified file
syntax: sum -r filename
31) time: this command is used to print a summary of real-time, user CPU time and system CPU time spent by executing a command when it terminates. ‘real-time is the time elapsed wall clock time taken by a command to get executed, while ‘user ‘and ‘systems are the number of CPU seconds that command uses in user and kernel mode respectively.
syntax: $time filename
32) touch: This command will update the timestamps of input file if it exists and will create an empty file if the input file does not exist.
syntax: touch -c filename
33) top: this command shows the summary information of the system and the list of processes which are currently processed
syntax: $ top
34) tr: This command is used to translate or delete the characters. It translates upper case to lower case, deleting some characters.
syntax: cat filename| tr [a-z] [A-Z]
-> -s option is used to squeeze the repetitive occurrences.
-> -d option is used to delete the characters.
35) tty: It displays the information about the terminal. It prints the file name of the terminal connected to the standard input
syntax: $tty
36) type: This command is used to describe how arguments are converted when they are used as commands. It displays whether the command is builtin or binary file
syntax: $ type command name
37) ulimit: It is used to see, set, limit the usage of the resource usage of the current user.
-> ulimit -a: to check the ulimit value
-> ulimit -u: to display the maximum users process limit for logged in user.
-> ulimit -F: to show maximum size of a user can have.
38) uname: print system information.
syntax: $uname
39) wc- This word count (wc) is used to print word, line, byte counts.
syntax: $ wc filename
40) .- run a command script in the current shell.
41) !!- run the last command again
42) whoami- Print the current username and id
syntax: $ whoami
43) who- print all users currently logged in
syntax: $ who
44) vmstat: report virtual memory stastics.

SHELL SCRIPT

Pavani — Tue, 13 Dec 2022 06:07:45 +0000

POWER SHELL: A shell is special user program which provide an interface to user to use operating system services. Shell accept human readable commands from user and convert them into something which kernel can understand. It is a command language interpreter that execute commands read from input devices such as keyboards or from files. The shell gets started when the user logs in or start the terminal.
It is classified into 2 types
1)command line shell: Shell can be accessed by user using a command line interface. A special program called Terminal in linux/macOS or Command Prompt in Windows OS is provided to type in the human readable commands such as “cat”, “ls” etc. and then it is being execute.
2) graphical shell: Graphical shells provide means for manipulating programs based on graphical user interface (GUI), by allowing for operations such as opening, closing, moving and resizing windows, as well as switching focus between windows.

There are several shells are available for Linux systems like –

BASH (Bourne Again Shell)– It is most widely used shell in Linux systems. It is used as default login shell in Linux systems and in macOS. It can also be installed on Windows OS.
CSH (C Shell)– The C shell syntax and usage are very similar to the C programming language.
KSH (Korn Shell)– The Korn Shell also was the base for the
POSIX (portable operating system interface) Shell standard specifications etc.

Each shell does the same job but understand different commands and provide different built in functions.
Why do we need shell scripts?

1)To avoid repetitive work and automation
2)System admins use shell scripting for routine backups
3)System monitoring
4)Adding new functionality to the shell.

Advantages of shell scripts

1)The command and syntax are exactly the same as those
directly entered in command line, so programmer do not need
to switch to entirely different syntax
2)Writing shell scripts are much quicker
3)Quick start
4)Interactive debugging etc.

-Disadvantages of shell scripts

1)Prone to costly errors, a single mistake can change the
command which might be harmful
2)Slow execution speed
3)Design flaws within the language syntax or implementation
Not well suited for large and complex task
Provide minimal data structure unlike other scripting
languages. etc
Shell commands:

1)cat: It is generally used to concatenate the files. It gives the output on the standard input. It creates the file with content.
syntax: $ cat employee name

2)more: It is a filter for paging through text one screenful at a time.
syntax: $ more employee name

3) less: It is used to viewing the files instead of opening the file. Similar to more command but it allows backward as well as forward movement.
syntax: $ less employee.txt

4) head: It is used to print the first N lines of a file. It accepts N as input and the default value of N is 10.
syntax: $ head employee. txt
We can give N value so that it prints the number of lines given.

5) tail: Used to print the last N-1 lines of a file. It accepts N as input and the default value of N is 10.
syntax: $ tail N-1 employee.txt

File and directory manipulation commands:

6) mkdir: mkdir function used to create directory in the current directory or using mkdir-p, it creates directory at the specified path.

7) cp: This command will copy the files and directories from the source path to the destination path. It can copy a file/directory with the new name to the destination path. It accepts the source file/directory and destination file/directory.

8) mv : Used to move the files or directories. This command’s working is almost similar to cp command but it deletes a copy of the file or directory from the source path.

9) rm : Used to remove files or directories.

10) touch: It is used to create a file without any content. The file created using touch command is empty. This command can be used when the user doesn’t have data to store at the time of file creation.
$ touch employeename
touch -a: This command is used to change access time only. To change or update the last access or modification times of a file touch -a command is used.

touch -c : This command is used to check whether a file is created or not. If not created then don’t create it. This command avoids creating files

              touch -c employee.txt

touch -c-d: This is used to update access and modification time.

        touch -c-d employee.txt

touch -m : This is used to change the modification time only. It only updates last modification time.

             touch -m employeename

11) grep: The grep filter searches a file for a particular pattern of characters, and displays all lines that contain that pattern.
-c: This prints only a count of the lines that match a
pattern
-h: Display the matched lines, but do not display the
filenames.
-i: Ignores, case for matching
-l: Displays list of a filenames only.
-n: Display the matched lines and their line numbers.
-v: This prints out all the lines that do not matches the
pattern
-e exp: Specifies expression with this option. Can use
multiple times.
-f file: Takes patterns from file, one per line.
-E: Treats pattern as an extended regular expression (ERE)
-w: Match whole word
-o: Print only the matched parts of a matching line,
with each such part on a separate output line.

12) sort: SORT command is used to sort a file, arranging the records in a particular order. By default, the sort command sorts file assuming the contents are ASCII. Using options in the sort command can also be used to sort numerically.

  -r Option: Sorting In Reverse Order: You can perform a reverse-order sort using the -r flag. the -r flag is an option of the sort command which sorts the input file in reverse order i.e. descending order by default. 

 -n Option: To sort a file numerically used –n option. -n option is also predefined in Unix as the above options are. This option is used to sort the file with numeric data present inside. 

  -nr option: To sort a file with numeric data in reverse order we can use the combination of two options as stated below. 

  -k Option: Unix provides the feature of sorting a table on the basis of any column number by using -k option. 

  -c option: This option is used to check if the file given is already sorted or not & checks if a file is already sorted pass the -c option to sort. This will write to standard output if there are lines that are out of order. The sort of tool can be used to understand if this file is sorted, and which lines are out of order

13) Wc: It is used to find out number of lines, word count, byte and characters count in the files specified in the file arguments.
By default, it displays four-columnar output. First column shows number of lines present in a file specified, second column shows number of words present in the file, third column shows number of characters present in file and fourth column itself is the file name which are given as argument.
syntax: $ wc file name
-c: This option displays count of bytes present in a file. With this option it displays two-columnar output, 1st column shows number of bytes present in a file and 2nd is the file name.
-m: Using -m option ‘wc’ command displays count of characters from a file.
-L: The ‘wc’ command allow an argument -L, it can be used to print out the length of longest (number of characters) line in a file.

14) cut: It can be used to cut parts of a line by byte position, character and field. Basically, the cut command slices a line and extracts the text. It is necessary to specify option with command otherwise it gives error. If more than one file name is provided, then data from each file is not precedes by its file name.
syntax: cut filename

   -b(byte): To extract the specific bytes, you need to follow -b option with the list of byte numbers separated by comma.
   -c (column): To cut by character use the -c option. This selects the characters given to the -c option. 
   -f (field): -c option is useful for fixed-length lines. Most unix files doesn’t have fixed-length lines

15) echo: Echo is a command in shell scripting which is a built-in command used to display a variable, result of an expression, string, number, text, any useful information in the programming, etc. can be displayed using echo command in shell scripting. It is available in most of the programming languages and most commonly used in shell scripting where bash and c shells are mostly used. This command can also be used to display the commands or arguments sent to a shell program also.
syntax echo string name
a) To display a text or string on the console
echo hello world
b) To display a variable value on the console
x=5 echo $x
c) To remove spaces from a given string and display on the
console
echo -e " this/ is/ my/ world"
d) To print all files or folders in the current directory
echo *
16) Variables: The variables which are a type of parameter are generally managed by the users or the system. We can take an example of $var which is a variable parameter. The system sets $var, but this variable parameter can be written by the user.

17) special parameters: The special parameters are read-only which are maintained by the shell. The special parameters are with a predefined meaning. Below are the various special parameters:
a) $#: Its parameter represents the total number of
arguments passed to the script.
b) $0: This parameter represents the script name.
c) $n: This parameter represents the arguments
corresponding to a script when a script is invoked
such $1 $2…etc. $1, $2…etc are called positional
parameters.
d) $: This parameter describes the positional parameters
to be distinct by space. For example, if there are
two arguments passed to the script, this parameter
will describe them as $1 $2.
e) $$: This parameter represents the process ID of a
shell in which the execution is taking place.
f) $!: This parameter represents the process number of
the background that was executed last.
g) $@: This parameter is similar to the parameter $.
h) $?: This parameter represents exit status of the last
command that was executed. Here 0 represents
success and 1 represents failure.
i) $_: This parameter represents the command which is
being executed previously.
j) $-: This parameter will print the current options
flags where the set command can be used to modify
the options flag.
syntax: $ cat program.sh
echo "The File Name is: $0"
echo "The First argument is: $1"
echo "The Second argument is: $2"

          $ sh program.sh ab cd
          The File Name: program.sh
          The First argument is: ab
          The Second argument is: cd

18) pwd: pwd‘ stands for ‘Print Working Directory ‘. As the name states, command ‘pwd‘ prints the current working directory or simply the directory user is, at present. It prints the current directory name with the complete path starting from root (/). This command is built in shell command and is available on most of the shell – bash, Bourne shell, ksh, zsh, etc.
syntax: pwd filename

19) Ifconfig: It is used to know the kernel-based interface for networks. This command is mainly used at the boot time to know and set up interfaces as and when necessary. Otherwise, ifconfig command only comes into a role when some system tuning, or some debugging is needed.

20) Netstat: This is one major command which tops the list of shell scripting commands. Netstat is used to display the network-related information like those of routing tables, network connections, masquerade connections, interface statistics, multicast memberships, etc. the suffix –a is used to list all network ports.

21) nslookup: his shell scripting command is mainly used by infra management and techOps/DevOps team as they are required to deal with a deep level of networking. It is a network utility-based command which displays the information of internet servers. It queries Domain Name Server and thereby fetches the result related to server name information.

22) Uptime: This is a command which is used to keep a track of any malicious or any unusual activity that might be affecting your system. Uptime is used to know what actually happened when the server was left unattended.

23) Wall: This is one of the most essential shell scripting commands, especially for an administrator as this can be used to broadcast a message to n number of people, to all those who have their mesg permission set to yes. The message is then provided as the argument to a wall or it is also sent as standard input for a wall.

24) Mesg: This command lets you take the control that whether the people can make use of the “write” by providing an option of y|n.

25) w: This command is though just a one-letter command can make wonders possible as it is a combination of who and uptime commands which are given in a sequence immediately after the other.

26) top: It is used to display all the processes of a CPU. This command is best known as it refreshes itself and continuously displays all the CPU processes which are up and running at one point of time until and unless an interrupt command is given.

27) Arithmetic operator
At the very first type of operator, we have an arithmetic operator. This is assumed to be an extension of the operators we use in mathematics as well. We are already well aware of these operators, but just for the sake of listing the different operators we have:
a) Addition operator (+): This operator is for the addition
of
2 operands.
Subtraction operator (-): This operator is for the
subtraction of 2 operands
Multiplication operator (*): This operator is for
multiplication of 2 operands
The division operator (/): This operator is for the division
of 2 operands. It gives only the
quotient.
The modulus operator (%): This operator is for estimating
the remainder when one operand is
divided by the other.
Increment Operator (++): This operator is for incrementing the
operand’s value by 1.
Decrement Operator (–): This operator is for decrementing the operand’s value by 1.

28) Relational operator
As the name suggests these operators work on determining the relation between 2 operands. In the case of the relational operator, the output is either true or false, irrespective of the type of operator we would be using. Some of these relational operators are:

a) ‘==’ Operator: This operator evaluates the two operands by equating them and process the output as true if they are equal and false if they are not. These operators may be used for integers as well as strings. For strings, one would use it with [[ for pattern matching.
b) ‘!=’ Operator: This operator evaluates the two operands by checking the inequality and process the output as true if they are equal and false if they are not. Again, for this operator, one can use it for integer as well as on strings. Essentially for string, this operator would return true if the strings don’t match.
c) ‘< ‘Operator: This operator checks for the value of the first operand’s to be less than the second one and return true if that’s the case and for vice versa, it would return false. This is not so widely used in strings!
d) ‘<=’ Operator: This operator checks for the value of the first operand’s to be less than or equal to the second one and return true if that’s the case and for vice versa, it would return false. This is not present for strings.
e) ‘>’ Operator: This operator checks for the value of the first operand’s to be greater than the second one and returns true if that’s the case and false for vice versa. Not quite widely used for strings.
f) ‘>=’ Operator: This operator checks for the value of the first operand’s to be greater than or equal to the second one and return true if that’s the case and false for vice versa. This option is not available for strings.

29) Logical Operator: This set of operators are for analyzing 2 or more set of conditions and return true based on some conditions for each of them.

a) Logical AND operator: Only in case of both the set of conditions be true, this operator will process true, else false.
Logical OR operator: In any case of either of the conditions in any one of the sides be true, this operator will output out TRUE as well.
b) NOT operator: This operator reverses the output coming out of the condition and shows that as the output.

30) Bitwise Operators
These operators are pretty much similar to the logical operators except the difference that these operators work on bit patterns. One essential thing to keep in mind is the << or >> are not for less than or greater than, but something explained below.

AND (&)
OR (|)
XOR (^)
Compliment
Shift left (<<)
Shift right (>>) 31) File operators: The final piece of the puzzle is the file operator, essentially used for testing the file properties:

-b: Checks if the file is a block special file or not.
-c: Checks if the file is a character special file or not.
-d: Checks if the name of the directory exists or not.
-e: Checks if the file exists or not
-r: Checks if the file has “read” access or not.
-w: Checks if the file has “write” access or not.
-x: Checks if the file has “execute” access or not.
-s: Checks for the file size and returns if the size is greater than 0.

32) if else: "If else” is a conditional or control statement in the programming language. This plays a different action or computation depending on whether the condition is true or false.

“if-else” is dealing with two-part of the executions. If the “if-else “condition is “true” then the first group of statements will execute. If the condition is “false” then the second group of statements will execute.
syntax:
if [ condition ]
then
Statement 1 (first Group)
else
Statement 2 (second Group)
fi

33) while loop: While loop provides a way to execute the same program by checking a condition when the condition satisfies the program will execute otherwise it won’t execute and the process repeats until the condition fails to meet.
syntax:
while [condition]
do
command1
command2
done

34) Local variables: These types of variables are present only within the running instance of the shell. In case there are child processes that are started by the shell script, the local variables might not be accessible to the child processes.

35) Environment variables: These types of variables are the ones that are accessible to any child process the shell script has run, unlike the local variables.

36) Shell variables: In certain cases, the shell might be required to set some variables in order to smooth the execution of the script and these variables are known as shell variables.
a) allexport: This variable is used for marking those variables and functions which are to be exported to the environment. Until and unless we specify this variable, all variables are local and when this is turned on all variables and functions will be transported to the subshell.
b) braceexpand: This variable performs brace expansion. This is a method used for generating strings at the command line. Mainly used for reusing a file path which is quite long to be written again in bash.

c) emacs: Using this variable one can use emacs styled editing interface in the command line.
d) errext: This variable is to allow shell scripts to immediately exit if there is a non-zero status out of the script.
e) errtrace: Traps are cool techniques for implementing error handling when using bash. In this any errors which can be trapped is inherited by shell functions or substitutions in command.
f) functrace: Like the previous variable, this variable helps traps on DEBUG and RETURN to be inherited by shell functions or substitutions in command.
g) hashall: This variable helps in remembering the location of commands as and when they are looked up for execution.
h) histexpand: This enables shell scripts to use! style history substitution. One would have stumbled across the usage of ! is a sentence and get an error. This is because of the! style history substitution. If this annoys one, one can surely set off this variable.
i) ignoreeof: If you want to use Ctrl + D in windows to keep your session or script running, and not leave the shell, we would need to use this variable. In short, when the shell reads EOF, it will not exit. For example, if you set IGNOREEOF=18, one would have to press Ctrl+D 18 times to leave the shell.
j) history: This is to enable command history. You would have noticed these while pressing up button you can see the previous commands you have used.
k) monitor: This variable enables the shell script to have job control during execution.
l) noclobber: This variable enables bash to not overwrite an existing file using >, >& operators.
m) noexec: Using this variable one would just read the commands and not execute them. This is widely used for doing syntax checks in the code.
n) noglob: This variable option is used for disabling a file name generation or in other words, pathname expansion.
o) nounset: In the case of parameter expansion, any unset parameters are treated as an error when this variable is used in the set command.
p) onecmd: This variable as the name suggests executes one command and then exits.
q) physical: When this variable is used, symbolic links don’t work. For example, if one needs to change the directory, they can’t use the cd as a physical directory structure would be used.
r) pipefail: When this variable is used, the pipeline is returned with a value which is the last command to exit with a non-zero status. This is to understand the last point of error in the code.
s) posix: When used, the bash behavior is changed making default operation different from POSIX standard.
t) privileged: This allows security by running the shell script without inheriting it from the environment and hence the environment variables are not accessible.
u)verbose:: This allows the input lines of the shell to be printed as they are read.
v) vi: This is to start a vi styled editing interface.
w) xtrace: This allows all the commands and their arguments to be printed as they get executed. Widely used for tracing back to an error in case many shell processes are run.

In the output you would see that all the command post point 1 gets printed as they are executed.

37) ls: ls is the command which is responsible for listing the folders and files present in a particular directory. This shell scripting command is often appended with other commands such as –ltr or –lrt, etc. depending upon the need.
syntax: ls

38) Piping (|): This is another very basic command that is used to fetch the output received from one command straight away into another. This symbol called pipe can most often be seen along with the grepping command. At some places, this piping can also be said to be chaining.
syntax: cat filename |grep string

39) Dig: This is another Intermediate command which is used to query the Domain name servers and provide information about the host addresses, nameservers, mail exchanges, etc. related information. It is mostly used to query a single given host.

40) Rename: this command is used to rename a file name

41) chown: Different users in the operating system have ownership and permission to ensure that the files are secure and put restrictions on who can modify the contents of the files.
Each user has some properties associated with them, such as a user ID and a home directory. We can add users into a group to make the process of managing users easier.
A group can have zero or more users. A specified user can be associated with a “default group”. It can also be a member of other groups on the system as well.
Ownership and Permissions: To protect and secure files and directory we use permissions to control what a user can do with a file or directory. It uses three types of permissions:

Read: This permission allows the user to read files and in directories, it lets the user read directories and subdirectories stores in it.
Write: This permission allows a user to modify and delete a file. Also it allows a user to modify its contents (create, delete and rename files in it) for the directories. Unless the execute permission is not given to directories changes does do affect them.
Execute: This permission on a file allows it to get executed. For example, if we have a file named php.sh so unless we don’t give it execute permission it won’t run.
Types of file Permissions:

User: This type of file permission affects the owner of the file.
Group: This type of file permission affect the group which owns the file. Instead of the group permissions, the user permissions will apply if the owner user is in this group.
Other: This type of file permission affects all other users on the system.
syntax:
chown [OPTION]… [OWNER] [: [GROUP]] FILE…
chown [OPTION]… –reference=RFILE FILE…
Example: To change owner of the file:
chown owner_name file_name

42) chgrp: chgrp command is used to change the group ownership of a file or directory. All files belong to an owner and a group. You can set the owner by using “chown” command, and the group by the “chgrp” command.

Syntax:

chgrp [OPTION]… GROUP FILE…
chgrp [OPTION]… –reference=RFILE FILE…

43) chmod: the chmod command is used to change the access mode of a file.

Syntax :chmod [reference][operator][mode] file...

The references are used to distinguish the users to whom the permissions apply i.e. they are list of letters that specifies whom to give permissions.

Reference Class Description
u owner file's owner

g group users who are members of
the file's group

o others users who are neither the
file's owner nor members of
the file's group

a all All three of the above, same as ugo

The operator is used to specify how the modes of a file should be adjusted. The following operators are accepted:

Operator Description

Adds the specified modes to the specified classes
Removes the specified modes from the specified classes

= The modes specified are to be made the exact modes for the specified classes

The modes indicate which permissions are to be granted or removed from the specified classes. There are three basic modes which correspond to the basic permissions:

r Permission to read the file.
w Permission to write (or delete) the file.
x Permission to execute the file, or, in
the case of a directory, search it.

44) IFS: IFS stands for Internal Field Separator. It is an environment variable that defines a field separator. By default, space, tab, and newline are considered as field separators

45) env: env is used to either print environment variables. It is also used to run a utility or command in a custom environment.
syntax: env [OPTION]... [-][NAME=VALUE]... [COMMAND [ARG]...]
a) Without any argument: print out a list of all environment variables.
b) u or –unset: remove variable from the environment
$ env -u variable_name

46) kill: This command is a built-in command which is used to terminate processes manually. kill command sends a signal to a process which terminates the process. If the user doesn’t specify any signal which is to be sent along with kill command then default TERM signal is sent that terminates the process.
a) kill -l: To display all the available signals
Syntax: $kill -l
b) ill pid : To show how to use a PID with the kill command.
Syntax: $kill pid
c) kill -s: To show how to send signal to pro
Syntax: kill {-signal | -s signal} pid
d) kill -L:This command is used to list available signals
in a table format.
Syntax: kill {-l | --list[=signal] | -L | --table}

EBS vs EFS vs S3

Pavani — Fri, 02 Dec 2022 17:51:49 +0000

AMAZON STORAGE SERVICES:
AWS provides wide range storage services that can be used accordingly to your project and use case. These are virtual storage services on cloud that allow to store data in a scalable and highly available manner and removes the hassle of buying and
maintaining physical equipment for data storage. These storage services are divided basing on the storge types i.e file storage, block storage and object storage.

Amazon Elastic Block Store (EBS):

EBS (Amazon Elastic Block Store) is a block type storage provided by AWS. It is similar to hard drives on our physical machines. EBS is primarily used by the AWS EC2 instances for persistent data storage which means that even if EC2 instances are shut down, the data on EBS volume is not lost.
EBS volumes can be dynamically attached, detached and scaled with EC2 instances.
Types of EBS volumes:
There are different types of EBS :
1)SSD backed volumes
2)HDD backed volumes

Features of EBS:

Scalability: EBS volume sizes and features can be scaled as per the needs of the system. This can be done in two ways:
a) Take a snapshot of the volume and create a new volume using
the Snapshot with new updated features.
b) Updating the existing EBS volume from the console.
Backup: Users can create snapshots of EBS volumes that act as backups. Snapshot can be created manually at any point in time or can be scheduled. Snapshots are stored on AWS S3 and are charged according to the S3 storage charges. Snapshots are incremental in nature. New volumes across regions can be created from snapshots.
Encryption: Encryption can be a basic requirement when it comes to storage. This can be due to the government of regulatory compliance. EBS offers an AWS managed encryption feature.
Users can enable encryption when creating EBS volumes bu clicking on a checkbox.
-> Encryption Keys are managed by the Key Management Service (KMS) provided by AWS.
-> Encrypted volumes can only be attached to selected instance types.
-> Encryption uses the AES-256 algorithm.
-> Snapshots from encrypted volumes are encrypted and similarly, volumes created from snapshots are encrypted.
Charges: AWS charges users for the storage you hold or used. For example, if you use 1 GB storage in a 5 GB volume, you’d still be charged for a 5 GB EBS volume. EBS charges vary from region to region. EBS Volumes are independent of the EC2 they are attached to. The data in an EBS volume will remain unchanged even if the instance is rebooted or terminated.

Relational and NoSQL databases can also be deployed and scaled with EBS. Various databases like MySQL, Oracle, MSSQL, Cassandra, MongoDB, etc. are supported.

Drawbacks:

EBS is not recommended as temporary storage.
They cannot be used as a multi-instance accessed storage as they cannot be shared between instances.
The durability offered by services like AWS S3 and AWS EFS is greater.

Single EBS volume can only be attached to one EC2 instance at a time. However, one EC2 can have more than one EBS volumes attached to it.

Amazon Simple Storage Service (s3):

S3 (Amazon Simple Storage Service) is an object type store where users can upload and store objects into it. The objects are stored in a container which is referred to as a bucket. We can put objects into a bucket or get the objects from a bucket. An object can be a file of any type like a text file, a video, an image, etc.

Features of AWS S3:

Durability: AWS claims Amazon S3 to have a 99.999999999% of durability (11 9’s). This means the possibility of losing your data stored on S3 is one in a billion.
Availability: AWS ensures that the up-time of AWS S3 is 99.99% for standard access.Note that availability is related to being able to access data and durability is related to losing data altogether.
Server-Side-Encryption (SSE): AWS S3 supports three types of SSE models:
SSE-S3: AWS S3 manages encryption keys.
SSE-C: The customer manages encryption keys.
SSE-KMS: The AWS Key Management Service (KMS) manages the
encryption keys.
File Size support: AWS S3 can hold files of size ranging from 0 bytes to 5 terabytes. A 5TB limit on file size should not be a blocker for most of the applications in the world.
Infinite storage space: Theoretically AWS S3 is supposed to have infinite storage space. This makes S3 infinitely scalable for all kinds of use cases.
Pay as you use: The users are charged according to the S3 storage they hold.
AWS-S3 is region-specific.

Use cases for S3: AWS S3 can be used by people with all kinds of use cases like mobile/web applications, big data, machine learning and many more.

Amazon Elastic File System (EFS):

EFS (Amazon Elastic File System) is a file-based storage service which is somewhat similar to the NAS (Network Attached Storage). EFS is a file-level, fully managed, storage provided by AWS that can be accessed by multiple EC2 instances concurrently. Just like the AWS EBS, EFS is specially designed for high throughput and low latency applications.
Features of AWS EFS:

Storage capacity: EFS provides an infinite amount of storage capacity. This capacity grows and shrinks as required by the user.
Fully Managed: EFS takes the overhead of creating, managing, and maintaining file servers and storage.
Multi EC-2 Connectivity: EFS can be shared between any number of EC-2 instances by using mount targets.
Note-: A mount target is an Access point for AWS EFS that is further attached to EC2 instances, allowing then access to the EFS.
Availability: AWS EFS is region specific., however can be present in multiple availability zones in a single region.
-> EC-2 instances across different availability zones can
connect to EFS in that zone for a quicker access
EFS Life Cycle Management: Lifecycle management moved files between storage classes. Users can select a retention period parameter (in number of days). Any file in standard storage which is not accessed for this time period is moved to Infrequently accessed class for cost-saving.
->Note that the retention period of the file in standard storage resets each time the file is accessed
->Files once accessed in the IA EFS class are them moved to Standard storage.
-> Note that file metadata and files under 128KB cannot be transferred to the IA storage class.
-> Life Cycle management can be turned on and off as deemed fit by the users.
Durability: Multi availability zone presence accounts for the high durability of the Elastic File System.
Transfer: Data can be transferred from on- premises to the EFS in the cloud using AWS Data Sync Service. Data Sync can also be used to transfer data between multiple EFS across regions.\

Use cases for EFS:

Multiple server architectures: In AWS only EFS provides a shared file system. So all the applications that require multiple servers to share one single file system have to use EFS.
Big Data Analytics: Virtually infinite capacity and extremely high throughput makes EFS highly suitable for storing files for Big data analysis.
Reliable data file storage: EBS data is stored redundantly in a single Availability Zone however EFS data is stored redundantly in multiple Availability Zones. Making it more robust and reliable than EBS.
Media Processing: High capacity and high throughput make EFS highly favorable for processing big media files.

EBS vs EFS vs S3:
The difference between EBS, EFS AND S3 are as follows:

Network firewall

Pavani — Fri, 02 Dec 2022 16:44:02 +0000

FIREWALL:
A firewall is a network security device, either hardware or software-based, which monitors all incoming and outgoing traffic and based on a defined set of security rules it accepts, rejects or drops that specific traffic.
-> Accept: allow the traffic
-> Reject: block the traffic but reply with an “unreachable
error”
-> Drop: block the traffic with no reply

A firewall establishes a barrier between secured internal networks and outside untrusted network, such as the Internet.

Network Firewall:

: It is a managed service that helps deploy network protections for Amazon VPCs. It Provides fine-grained network traffic control that allows you to restrict outbound requests to prevent malicious activity from spreading. Import previously created rules in common opensource rule formats and enable integrations with managed intelligence feeds from AWS partners. With AWS Firewall Manager, you can create policies based on AWS Network Firewall rules and then apply those policies centrally across your VPCs and accounts.
Features:
-> Automatically scales firewall capacity up or down based on the traffic load.
-> Supports inbound and outbound web filtering for unencrypted web traffic. The intrusion prevention system matches network traffic patterns to known threat signatures based on attributes.
->Centrally deploy and manage security policies across AWS Organizations apps, VPCs, and accounts.
-> AWS Network Firewall has a highly flexible rules engine.
-> AWS Network Firewall supports thousands of rules, and the rules can be based on domain, port, protocol, IP addresses, and pattern matching.

Concepts:
1) Firewall: A traffic filtering logic for VPC subnets. The firewall configuration provides the parameters for the Availability Zones and subnets in which the firewall endpoints are located.
2) Rule groups: A set of rules to match against VPC traffic and actions to do when a match is discovered. can create a custom rule group or use the one that is managed by AWS. The categories of rule groups are stateless and stateful. A designated subnet for a firewall endpoint is called a firewall subnet. A stateless rule examines a single network traffic packet without taking into account the context of other packets. While the inspection of network traffic packets in the context of their traffic flow is referred to as stateful rules.
3) Monitoring: You can use the following monitoring tools with Network Firewall:
. Amazon CloudWatch
. Amazon CloudWatch Logs
. AWS CloudTrail
. AWS Config

Firewall logging is only available for traffic that you route to the stateful rules engine. Traffic is forwarded to the stateful engine via stateless rule actions and default actions.
->Using a stateful engine, you can record flow logs and alert logs.
. Flow logs – standard network traffic flow logs.
. Alert logs – report traffic that matches your
stateful rules.
Logs contain the following information:
- firewall-name
- availability-zone
- event-timestamp
- Event

You can configure the destinations of your logs to various AWS services:
- Amazon S3
- CloudWatch Logs
- Kinesis Data Firehose
Pricing:
You are charged at an hourly rate for each firewall endpoint. You are charged for the amount of traffic, billed by the gigabyte, processed by the firewall endpoint. Data transferred across the AWS Network Firewall incur standard AWS data transfer fees

Amazon Web Services

Pavani — Fri, 02 Dec 2022 09:26:52 +0000

Amazon Web Services:

AWS is the largest cloud computing platform, offering 200+ universally featured resources, from infrastructure to machine learning. The benefits of cloud computing is the opportunity to replace up-front capital infrastructure expenses with low variable costs that scale with business. With the Cloud, businesses no longer need to plan for and procure servers and other IT infrastructure weeks or months in advance. Instead, they can instantly spin up hundreds or thousands of servers in minutes and deliver results faster.

Amazon Web Services provides a highly reliable, scalable, low-cost infrastructure platform in the cloud that powers hundreds of thousands of businesses in 190 countries around the world. With data center locations in the U.S., Europe, Brazil, Singapore, Japan, and Australia, customers across all industries are taking advantage of the following benefits:

1) low cost: AWS offers low, pay-as-you-go pricing with no up-front expenses or long-term commitments. AWS is able to build and manage a global infrastructure at scale and pass the cost saving benefits in the form of lower prices.

2) Agility and Instant Elasticity: AWS provides a massive global cloud infrastructure that allows one to quickly innovate, experiment and iterate. Instead of waiting weeks or months for hardware, one can instantly deploy new applications, instantly scale up as workload grows, and instantly scale down based on demand.

3) Open and Flexible: AWS is a language and operating system agnostic platform. one choose the development platform or programming model that makes the most sense for business. one can choose which services to use, one or several, and choose how to use them. This flexibility allows you to focus on innovation, not infrastructure.

4) Secure: AWS is a secure, durable technology platform with industry-recognized certifications and audits: PCI DSS Level 1, ISO 27001, FISMA Moderate, FedRAMP, HIPAA, and SOC 1 (formerly referred to as SAS 70 and/or SSAE 16) and SOC 2 audit reports. Our services and data centers have multiple layers of operational and physical security to ensure the integrity and safety of your data.

Important services provided by AWS:

1) Amazon Elastic Compute Cloud:

EC2 stands for Amazon Elastic Compute Cloud.
Amazon EC2 is a web service that provides resizable compute capacity in the cloud.
Amazon EC2 delivers secure, reliable, high-performance, and cost-effective compute infrastructure to meet demanding business needs. Access the on-demand infrastructure and capacity you need to run HPC applications faster and cost-effectively.
Access environments in minutes, dynamically scale capacity as needed, and benefit from AWS’s pay-as-you-go pricing. Amazon EC2 delivers the broadest choice of compute, networking (up to 400 Gbps), and storage services purpose-built to optimize price performance for ML projects.
Use cases:

Customers: Finra and VOlkswagen group

2) Amazon Relational Database Services:

Amazon Relational Database Service (RDS) is a managed SQL database service provided by Amazon Web Services (AWS). Amazon RDS supports an array of database engines to store and organize data. It also helps in relational database management tasks like data migration, backup, recovery and patching.

Amazon RDS facilitates the deployment and maintenance of relational databases in the cloud. Cloud administrators use Amazon RDS to set up, operate, manage, and scale relational instances of cloud databases. Amazon RDS itself is not a database; It is a service used to manage relational databases.

Amazon Relational Database Service RDS is a managed relational database service that provides you six familiar database engines to choose from, including Amazon Aurora, MySQL, MariaDB, PostgreSQL, Oracle, and Microsoft SQL Server. This means that the code, applications, and tools you already use today with your existing databases can be used with Amazon RDS. Amazon RDS handles routine database tasks, such as provisioning, patching, backup, recovery, failure detection, and repair.
CUstomers: Mint and Samsung

3) Amazon Simple storage Service:

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Amazon S3 provides a simple web service interface that you can use to store and retrieve any amount of data, at any time, from anywhere. Using this service, you can easily build applications that make use of cloud native storage
->Amazon S3 is a simple key-based object store. When you store data, you assign a unique object key that can later be used to retrieve the data. Keys can be any string, and they can be constructed to mimic hierarchical attributes. Alternatively, you can use S3 Object Tagging to organize your data across all of your S3 buckets and/or prefixes.

   Customers: Georgia- Pacific, Zalando

4) Amazon Lambda: AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. Automatically respond to code execution requests at any scale, from a dozen events per day to hundreds of thousands per second. Save costs by paying only for the compute time you use—by per-millisecond—instead of provisioning infrastructure upfront for peak capacity. Optimize code execution time and performance with the right function memory size. Respond to high demand in double-digit milliseconds with Provisioned Concurrency.
-> Each AWS Lambda function runs in its own isolated environment, with its own resources and file system view. AWS Lambda uses the same techniques as Amazon EC2 to provide security and separation at the infrastructure and execution levels.
-> AWS Lambda stores code in Amazon S3 and encrypts it at rest. AWS Lambda performs additional integrity checks while your code is in use.
Customers: Nielsen, coco cola company

5) Amazon Glacier: AWS Glacier, is the backup and archival storage provided by AWS. It is an extremely low cost, long term, durable, secure storage service that is ideal for backups and archival needs. In a lot of its operation AWS Glacier is similar to S3, and, it interacts directly with S3, using S3-lifecycle policies.
-> The main difference between AWS S3 and Glacier is the cost structure. The cost of storing the same amount of data in AWS Glacier is significantly less as compared to S3. Storage costs in Glacier can be as little as $1 for one petabyte of data per month
Features of AWS Glacier
-> Given the extremely cheap storage, provided by AWS Glacier, it doesn’t provide as many features as AWS S3. Access to data in AWS Glacier is an extremely slow process.
Just like S3, AWS Glacier can essentially store all kinds of data types and objects.
-> Durability: AWS Glacier, just like Amazon S3, claims to have a 99.9999999% of durability (11 9’s). This means the possibility of losing your data stored in one of these services one in a billion. AWS Glacier replicates data across multiple Availability Zones for providing high durability.
-> Data Retrieval Time: Data retrieval from AWS Glacier can be as fast as 1-5 minutes (high-cost retrieval) to 5-12 hours (cheap data retrieval).
-> AWS Glacier Console: The AWS Glacier dashboard is not as intuitive and friendly as AWS S3. The Glacier console can only be used to create vaults. Data transfer to and from AWS Glacier can only be done via some kind of code. This functionality is provided via:
AWS Glacier API
AWS SDKs
-> Region-specific costs: The cost of storing data in AWS Glacier varies from region to region.
-> Security: AWS Glacier automatically encrypts your data using the AES-256 algorithm and manages its keys for you.
Apart from normal IAM controls AWS Glacier also has resource policies (vault access policies and vault lock policies) that can be used to manage access to your Glacier vaults.
-> Infinite Storage Capacity: Virtually AWS Glacier is supposed to have infinite storage capacity.
Customers: QUBE, BANDLAB

6) Amazon Simple Notification Service:
Amazon Simple Notification Service (Amazon SNS) is a web service that makes it easy to set up, operate, and send notifications from the cloud.
It provides developers with a highly scalable, flexible, and cost-effective capability to publish messages from an application and immediately deliver them to subscribers or other applications.
It is designed to make web-scale computing easier for developers. It is designed to make web-scale computing easier for developers.
Benefits of amazon SNS:
. Instantaneous, push-based delivery (no polling)
. Simple APIs and easy integration with applications
. Flexible message delivery over multiple transport protocols
. Inexpensive, pay-as-you-go model with no up-front costs
. Web-based AWS Management Console offers the simplicity of a point-and-click interface

Amazon Simple Notification Service (SNS) sends notifications two ways, A2A and A2P. A2A provides high-throughput, push-based, many-to-many messaging between distributed systems, microservices, and event-driven server less applications. These applications include Amazon Simple Queue Service (SQS), Amazon Kinesis Data Firehose, AWS Lambda, and other HTTPS endpoints. A2P functionality lets you send messages to your customers with SMS texts, push notifications, and email. Simplify your architecture an reduce costs with message filtering, batching, ordering and deduplication. Increase message duration with archiving, delivery retires and dead letter queues.
Customers: Change Health Care, NASA

7) Amazon Virtual Private Cloud (VPC):

Amazon VPC or Amazon Virtual Private Cloud is nothing but a service that allows its users to launch their virtual machines in a protected as well as isolated virtual environment defined by them. Amazon Virtual Private Cloud (Amazon VPC) gives you full control over your virtual networking environment, including resource placement, connectivity, and security. Spend less time setting up, managing, and validating your virtual network. Customize your virtual network by choosing your own IP address range, creating subnets, and configuring route tables.
It’s applicable to organizations where the data is scattered and needs to be managed well.

Amazon VPC comprises a variety of objects that will be familiar to customers with existing networks:
Components of Amazon VPC:

A Virtual Private Cloud: A logically isolated virtual network in the AWS cloud. You define a VPC’s IP address space from ranges you select.
Subnet: A segment of a VPC’s IP address range where you can place groups of isolated resources.
Internet Gateway: The Amazon VPC side of a connection to the public Internet.
NAT Gateway: A highly available, managed Network Address Translation (NAT) service for your resources in a private subnet to access the Internet.
Virtual private gateway: The Amazon VPC side of a VPN connection.
Peering Connection: A peering connection enables you to route traffic via private IP addresses between two peered VPCs.
VPC Endpoints: Enables private connectivity to services hosted in AWS, from within your VPC without using an Internet Gateway, VPN, Network Address Translation (NAT) devices, or firewall proxies.
Egress-only Internet Gateway: A stateful gateway to provide egress only access for IPv6 traffic from the VPC to the Internet.

Customers: TABLEAU, ATLASSIAN

8) Amazon Kinesis:

Amazon Kinesis is a service provided by Amazon Web Service which allows users to process a large amount of data (which can be audio, video, application logs, website clickstreams, and IoT telemetr)per second in real-time.
Amazon Kinesis makes it easy to collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information.
Amazon Kinesis offers key capabilities to cost-effectively process streaming data at any scale, along with the flexibility to choose the tools that best suit the requirements of your application.
Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.
Features of Amazon Kinesis

Cost-efficient: All the services provided by the amazon are cost-efficient as it follows the pay as you go model which means you have to pay for the service according to the usage, not a flat price. So it becomes advantageous for the user s that they have to pay only what they use.
Integrate with other AWS services: Amazon Kinesis allows users to use the other AWS services and integrate with it. Services that can be integrated are Amazon DynamoDB, Amazon Redshift, and all the other services that deal with the large amount of data.
Availability: You can access it from anywhere and anytime. Just need a good connectivity of net.
Real-time processing- It allows you to work upon the data which is needed to be updated every time with changes instantaneously. Most advantageous feature of Kinesis because real-time processing becomes important when you are dealing with such a huge amount of data. Limits of Amazon Kinesis: . The limitation that Amazon kinesis has that it only access the stream of records log for 24 hours by default but it can extend but up to only 7 days not longer than that. . There is no upper limit in the number of streams that can users have in their accounts. . One shard supports up to 1000 PUT records per second. Customers: Zillow, Netflix

9) Amazon Auto- Scaling:
Amazon EC2 Auto Scaling helps you maintain application availability. Improve fault tolerance through automatic detection and replacement of unhealthy instances. Increase availability with predictive or dynamic scaling policies with the right amount of compute capacity. Optimize workload performance and cost by combining purchase options and instance types. Reduce the complexity of configuration changes and application deployments with instance refresh.

10) Amazon Identity Access Management (IAM): With AWS Identity and Access Management (IAM), you can specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS.
-> With AWS Identity and Access Management (IAM), you can specify who or what can access services and resources in AWS, centrally manage fine-grained permissions, and analyze access to refine permissions across AWS. Continually analyze access to right-size permissions on the journey to least privilege.
customers: DOW JONES, JMT

11) Amazon simple Queue Service: Amazon Simple Queue Service (SQS) lets you send, store, and receive messages between software components at any volume, without losing messages or requiring other services to be available.
-> Eliminate overhead with no upfront costs and without needing to manage software or maintain infrastructure. Reliably deliver large volumes of data, at any level of throughput, without losing messages or needing other services to be available.
-> Securely send sensitive data between applications and centrally manage your keys using AWS Key Management.
-> Scale elastically and cost-effectively based on usage, so you don’t have to worry about capacity planning and reprovisioning.
Features of SQS:
-> Increase application reliability and scale.
-> Decouple microservices and process event-driven applications.
-> Ensure work is completed cost-effectively and on time.
-> Maintain message ordering with deduplication.
Customers: BMW, CAPITAL ONE

12) Amazon ElastiCache: Amazon Elasti Cache is a fully managed, in-memory caching service supporting flexible, real-time use cases. You can use Elasti Cache for caching, which accelerates application and database performance, or as a primary data store for use cases that don't require durability like session stores, gaming leaderboards, streaming, and analytics. Elasti Cache is compatible with Redis and Memcached. Scale with just a few clicks to meet the needs of your most demanding, internet-scale applications. Reduce costs and eliminate the operational overhead of self-managed caching.
Features of Elasticache:
-> Access data with microsecond latency and high throughput for lightning-fast application performance.
-> Cache your data to reduce pressure on your backend database, enabling higher application scalability and reducing operational burden.
-> Use ElastiCache to store non-durable datasets in memory and support real-time applications with microsecond latency.
Customers: The Pokemon Company, Tinder

13) Amazon Sagemaker: SageMaker is a full-fledged management service providing developers and data scientists with the resources to build, train, and deploy machine learning models rapidly. Use it to create highly scalable machine learning models that deploy products faster and deliver to market quickly. Access, label, and process large amounts of structured data (tabular data) and unstructured data (photo, video, geospatial, and audio) for ML. Reduce training time from hours to minutes with optimized infrastructure. Boost team productivity up to 10 times with purpose-built tools.
Customers: Pytorch, Tensorflow

14) Amazon Lightsail: Amazon Lightsail offers easy-to-use virtual private server (VPS) instances, containers, storage, databases, and more at a cost-effective monthly price. Automatically configure networking, access, and security environments. Easily scale as you grow—or migrate your resources to the broader AWS ecosystem, such as Amazon EC2. Leverage the security and reliability of the world’s leading cloud platform.
-> A Lightsail instance is a virtual private server (VPS) that lives in the AWS Cloud. Using Lightsail instances one can store data, run code, and build web-based applications or websites. Instances can connect to each other and to other AWS resources through both public (Internet) and private (VPC) networking. One can create, manage, and connect easily to instances right from the Lightsail console.

Customer: Accentric, Gourmeal

15) Amazon Elastic File System (EFS):
Amazon EFS file systems can automatically scale from gigabytes to petabytes of data without needing to provision storage. Tens, hundreds, or even thousands of compute instances can access an Amazon EFS file system at the same time, and Amazon EFS provides consistent performance to each compute instance. Amazon EFS is designed to be highly durable and highly available. With Amazon EFS, there is no minimum fee or setup costs, and one can pay only for what they use.
-> Amazon EFS provides performance for a broad spectrum of workloads and applications: big data and analytics, media processing workflows, content management, web serving, and home directories.
-> Amazon EFS Standard storage classes are ideal for workloads that require the highest levels of durability and availability.
Use cases:
-> Share code and other files in a secure, organized way to increase DevOps agility and respond faster to customer feedback.
-> Persist and share data from your AWS containers and serverless applications with zero
management required.
-> Simplify persistent storage for modern content management system (CMS) workloads.
-> Easier to use and scale, Amazon EFS offers the performance and consistency needed for machine learning (ML) and big data analytics workloads.
Customers: Johnson and johnson, Discover

16) Amazon Cloud Watch:
Amazon CloudWatch collects and visualizes real-time logs, metrics, and event data in automated dashboards to streamline your infrastructure and application maintenance. Improve operational performance using alarms and automated actions set to activate at predetermined thresholds. Troubleshoot operational problems with actionable insights derived from logs and metrics in your CloudWatch dashboards.

          Customers: Pushpay, Mapbox

17) Amazon Cloud Directory: Amazon Cloud Directory enables you to build flexible cloud-native directories for organizing hierarchies of data along multiple dimensions. With Cloud Directory, you can create directories for a variety of use cases, such as organizational charts, course catalogs, and device registries. While traditional directory solutions, such as Active Directory Lightweight Directory Services (AD LDS) and other LDAP-based directories, limit you to a single hierarchy, Cloud Directory offers you the flexibility to create directories with hierarchies that span multiple dimensions. For example, you can create an organizational chart that can be navigated through separate hierarchies for reporting structure, location, and cost center. Cloud Directory eliminates time-consuming and expensive administrative tasks, such as scaling infrastructure and managing servers.

18) Amazon Cognito:
With Amazon Cognito, you can add user sign-up and sign-in features and control access to your web and mobile applications. Amazon Cognito provides an identity store that scales to millions of users, supports social and enterprise identity federation, and offers advanced security features to protect your consumers and business. Built on open identity standards, Amazon Cognito supports various compliance regulations and integrates with frontend and backend development resources. Deliver frictionless customer identity and access management (CIAM) with a cost-effective and customizable platform.
Customers: NHS Digital,Trade Micro
19) Amazon Elastic Beanstalk:
Elastic Beanstalk is a service for deploying and scaling web applications and services. Upload your code and Elastic Beanstalk automatically handles the deployment—from capacity provisioning, load balancing, and auto scaling to application health monitoring. Use adjustable settings to scale your application for handling peaks in traffic, while minimizing costs.
features:
-> Deploy scalable web applications in minutes without the complexity of provisioning and managing underlying infrastructure.
-> Use your favorite programming language to build mobile API backends, and Elastic Beanstalk will manage patches and updates.
-> Migrate stateful applications off legacy infrastructure to Elastic Beanstalk and connect securely to your private network.

20) Amazon DynamoDB:
Amazon DynamoDB is a fully managed, serverless, key-value NoSQL database designed to run high-performance applications at any scale.
-> Build internet-scale applications supporting user-content metadata and caches that require high concurrency and connections for millions of users and millions of requests per second.
-> Scale throughput and concurrency for media and entertainment workloads such as real-time video streaming and interactive content and deliver lower latency with multi-Region replication across AWS Regions.
-> Use design patterns for deploying shopping carts, workflow engines, inventory tracking, and customer profiles. DynamoDB supports high-traffic, extreme-scaled events and can handle millions of queries per second.
-> Focus on driving innovation with no operational overhead. Build out your game platform with player data, session history, and leaderboards for millions of concurrent users.
Customers: Disney, Dropbox, zoom

Attaching EBS volume in EC2 instance

Pavani — Fri, 02 Dec 2022 09:03:39 +0000

EBS: ELASTIC BLOCK STORAGE
AWS Elastic Block Store (EBS) is Amazon’s block-level storage solution used with the EC2 cloud service to store persistent data. This means that the data is kept on the AWS EBS servers even when the EC2 instances are shut down. EBS offers the same high availability and low-latency performance within the selected availability zone, allowing users to scale storage capacity at low subscription-based pricing model. The data volumes can be dynamically attached, detached and scaled with any EC2 instance, just like a physical block storage drive. As a highly dependable cloud service, the EBS offering guarantees 99.999% availability.

AWS EBS is different from the standard EC2 Instance Store, which merely provides temporary storage available on the physical EC2 host servers. The EC2 Instance Store is useful for temporary data content such as caches, buffers or files that are replicated across the hosted servers. For data that needs to be available persistently, regardless of the operating life of an EC2 instance.

Steps to attach EBS 1 in EC2 instance:
step 1: Go to console home > Build a solution > launch a virtual
machine with EC2

Step 2: Click on launch instance

Step 3: Give the EC2 instance name EBS 1

Step 4: Choose the AWS image for EBS1

Step 5: Choose an instance type

Step 6: Choose the key pair and create the key pair.

Step 7: Create the security group and open the port as per the requirement

Step 8: click on the add volume for extra EBS1

Step 9: Configure EBS Size

Step 10: Select EBS size as per Requirement

the AWS EBS offers the following storage volume options:

General Purpose SSD (gp2): An optimum balance between cost and performance for a variety of IT workloads. Use cases include virtual desktops, apps, dev and test environments, among others.

Provisioned IOPS SSD (io1): The high-performance functionality serves particularly well for mission-critical IT workloads. Suitable use cases include large databases and business apps that require 16,000 IOPS or 250 MiB/s of throughput per volume.

Throughput Optimized HDD (st1): A low cost alternative for large storage volume workloads with high performance throughput requirements. Examples include streaming workloads, big data applications, log processing and data warehousing.

Cold HDD (sc1): An inexpensive alternative for use cases with a requirement to maintain minimal cost for large volume data storage. Examples include workloads that are accessed less frequently.

Step 11: Click launch instance button

On the Instances screen, we can view the status of the launch. It takes a short time for an instance to launch. When you launch an instance, its initial state is pending. After the instance starts, its state changes to running and it receives a public DNS name

Files vs Objects vs Blocks

Pavani — Thu, 01 Dec 2022 09:17:35 +0000

Files, Blocks and objects are storage formats that hold, organize, and present data in different ways- each with their own capabilities and limitations. File storage organizes and represents data as a hierarchy of files in folders, block storage chunks data into arbitrarily organized, evenly sized volumes, and object storage manages data and links it to associated metadata.
Containers are highly flexible and bring incredible scale to how apps and storage are delivered.

file storage:
File storage also called as file level or file based storage. Data is stored as a single piece of information inside a folder. when one need to access that data computer need to know the path to find it. Data stored in files is organized and retrieved using a limited amount of metadata that tells computer where the file is kept.it is like a library card catalog for data files. This is the oldest and widely used data storage system for direct and network attached storage system. file storage has broad capabilities and can store just above anything. Its great for storing an array of complex files and is fairly for user to navigate. File based storage must scale up by adding more systems than scale up by adding more capacity.
Block storage:
Block storage chops data into blocks and stores them as separate pieces. Each block of data is given a unique identifier which allows a storage system to place the smallest pieces of data wherever is convenient. Block storage is often configured to decouple the data from the users environment and spread it across multiple environments that can better serve the data. and when data is requested the underlying storage software reassembles the blocks of data from these environments and presents them back to the user. It is usually deployed in storage area network environments and must be tied to a functioning server.

Block storage does not rely on a single path like file storage. Each block lives on its own and can be partitioned so it can be partitioned so it can be accessed in different operating system which gives the user complete freedom to configure their data.it works well with big transactions and those deploy huge databases i.e more data to store it will be better with block storage.

It has disadvantages like it is more expensive, it has limited capability to handle metadata which means it needs to be dealt with in the application or data base level.

Object storage:

Object storage is a flat structure in which files are broken into pieces and spread out among hardware. In this storage the data is broken into discrete units called objects and is kept in a single repository. Object storage volumes work as modular units. Each is self contained repository that owns the data an unique identifier that allows the object to be found over a distributed system and the metadata that describes the data. The metadata is important and includes details like age, privacy and securities and access contingencies. Object storage metadata can also be extremely detailed and is capable of storing information in where a video was shot, what camera was used.
Object storage requires a simple HTTP application which is used by most clients in all languages. It is cost efficient. Its a storage system well suited for static data and its agility and flat nature means it can scale to extremely large quantities of data. They ae good at storing unstructured data.

there are disadvantages like objects cant be modified. This storage cant work on traditional databases because writing databases is a slow process and writing an app to use an object storage is not as simple as using file storage.

Public and Private ip

Pavani — Thu, 01 Dec 2022 07:43:37 +0000

IP Address: An IP (Internet Protocol) address is a numerical label assigned to the devices connected to a computer network that uses the IP for communication.

IP address act as an identifier for a specific machine on a particular network. It also helps you to develop a virtual connection between a destination and a source. The IP address is also called IP number or internet address. It helps you to specify the technical format of the addressing and packets scheme. Most networks combine TCP with IP.
IP Address is divided into two parts:

-> Prefix: The prefix part of IP address identifies the physical network to which the computer is attached. . Prefix is also known as a network address.
-> Suffix: The suffix part identifies the individual computer on the network. The suffix is also called the host address.

Private IP: the scope of private is local.it is used to communicate within the network. private IP address of the systems connected in a network differ in a uniform manner. it works only on LAN. it is used to load the network operating system. it is available free of cost. private IP can be known by entering ipconfig on the command prompt. Private IP uses numeric code that is not unique and can be used again. private IP addresses are secure. private IP addresses require NAT to communication with devices.
range : 10.0.0.0- 10.255.255
172.16.0.0 - 172.31.255.255
192.168.0.0- 192.168.255.255
Example:192.168.1.10

public IP: the scope of public IP is global. It is used to communicate outside the network. public IP may differ in uniform or non uniform manner. it is used to ger internet service. It is controlled by ISP. It is not free of cost. Public IP can be k own by searching "what is my IO" on google. besides private IP addresses the rest are public. public IP uses a numeric code that is unique and cannot be used by other. Public IP address has no security and is subjected to attack. Public IP does not require a network translation.
example:17.5.7.8

Region and availablity zone

Pavani — Thu, 01 Dec 2022 07:07:37 +0000

AWS Regions:
AWS Regions are separate geographic areas that AWS uses to house its infrastructure. These are distributed around the world so that customers can choose a region closest to them in order to host their cloud infrastructure there. The closer the region is the better, to reduce network latency as much as possible for your end-us and for fast service.

Each AWS Region is designed to be isolated from the other AWS Regions. This design achieves the greatest possible fault tolerance and stability.

In general, try to follow these best practices when you choose a region, to ensure top performance and resilience:
-> Proximity: Choose a region closest to your location and your customers’ location to optimize network latency.
-> Services: Try and think about what your most needed services are. Usually, the newest services start on a few main regions then pop up in other regions later.
-> Cost: Certain regions will cost more than others, so use built-in AWS calculators to do rough cost estimates to inform your choices.
-> Service Level Agreement (SLA): Just as with cost, your SLA details will vary by region, so be sure to be aware of what your needs are and if they’re being met.
-> Compliance: You may need to meet regulatory compliance needs such as GDPR by hosting your deployment in a specific — or multiple regions.

Availablity Zone: An AWS Availability Zone (AZ) is the logical building block that makes up an AWS Region. Each AWS Region has multiple, isolated locations known as Availability Zones. There are currently 69 AZs, which are isolated locations— data centers — within a region.
AWS Availability Zones give you the flexibility to launch production apps and resources that are highly available, resilient/fault-tolerant, and scalable as compared to using a single data center

DNS server record types

Pavani — Thu, 01 Dec 2022 05:05:58 +0000

DNS: Domain Name System. it translates domain names to IP addresses so browsers can load internet resources. Each device connected to the Internet has a unique IP address which other machines use to find the device. DNS servers eliminate the need for humans to memorize IP addresses such as 192.168.1.1 (in IPv4), or more complex newer alphanumeric IP addresses such as 2400:cb00:2048:1::c629:d7a2 (in IPv6).

DNS records: these records provide important information about a hostname or domain. these records include th current IP address for a domain. these records are stored in text files on the authoritative DNS server.
There are 5 major types of DNS records:
A type: the " A" in record stands for address. An A record shows the specific hostname or domain. The A record supports only IPV4 addresses.
Use of A record:

it is used for IP address lookup.
by using the A record web browser is able to load a website using the domain name.
Domain name system based blackhole list (DNSBL)
A record is used block mail from spam sources
AAAA record: it points to IPV6 addresses.
use of AAAA record:
these records are used to resolve a domain name to the newer IPV6 protocol address
these records are used for DNS resolution
CNAME record: Canonical name is record that points a domain name to another domain. in CNAME record alias doesnot point to an IP address and the domain name that the alias points to is the canonical name.
Use of CNAME record:
running of multiple subdomains for different purposes on the same server.
NS Record: A nameserver (NS) record specifies the authoritative DNS server for a domain. in general, the NS record helps point to where internet applications like a web browser can find the ip address of a domain name
MX Record: Mail exchange record is a DNS record typw which shows where emails for a domain should be routed to. this record makes possible to direct emails to mail server.

Use of MX record:

it is used to hand off emails to a dedicated email server. *

Network, subnet, internet gateway, route table

Pavani — Wed, 30 Nov 2022 12:14:36 +0000

NETWORK: A network consists of two or more computers that are linked in order to share resources, exchange files or allow electronic communications. The computer on a network are connected through cables, telephone lines, radio waves, satellites, or Infrared beams.
Benefits of a Network:

-> Information sharing – Authorized users can use other computers on the network to access and share information and data. This could include special group projects, databases, etc.
-> Hardware sharing – One device connected to a network, such as a printer or a scanner, can be shared by many users.
-> Software sharing – Instead of purchasing and installing a software program on each computer, it can be installed on the server. All of the users can then access the program from a single location.
-> Collaborative environment – Users can work together on group projects by combining the power and capabilities of diverse equipment.
Risks of networking:

Equipment malfunctions
System failures
Computer hackers
Virus attacks Types of network: 1) Local Area Network (LAN): It is usually privately owned and links the devices in a single office, building, or campus. Its size is limited to a few kilometers. It is designed to allow resources (h/w, s/w or data) to be shared between personal computers or workstations. In general, a given LAN will use only one type of transmission medium. The most common LAN topologies are bus, ring, and star. 2) Metropolitan Area Network (MAN): It is designed to extend over an entire city. A company can use MAN to connect the LANs in all its offices throughout a city. Maybe wholly owned and operated by a private company or it may be service provided by a public company (local telephone company). 3) Wide Area Network (WAN): It provides long-distance transmission of data over a country, a continent, or even the worldwide. Maybe wholly owned and operated by a single company is referred to as an enterprise network.

SUBNET: A subnet is a segmented piece of a larger network. Subnets are a logical partition of an IP network into multiple, smaller network segments. One goal of a subnet is to split a large network into a grouping of smaller, interconnected networks to help minimize traffic. Subnetting, the segmentation of a network address space, improves address allocation efficiency.
-> Each subnet allows its connected devices to communicate with each other, while routers are used to communicate between subnets. The size of a subnet depends on the connectivity requirements and the network technology employed. A point-to-point subnet allows two devices to connect, while a data center subnet might be designed to connect many more devices.
Uses of Subnets:
1) Reallocating IP addresses. Each class has a limited number of host allocations; for example, networks with more than 254 devices need a Class B allocation. If a network administrator is working with a Class B or C network and needs to allocate 150 hosts for three physical networks located in three different cities, they would need to either request more address blocks for each network -- or divide a network into subnets that enable administrators to use one block of addresses on multiple physical networks.
2)Relieving network congestion. If much of an organization's traffic is meant to be shared regularly between the same cluster of computers, placing them on the same subnet can reduce network traffic. Without a subnet, all computers and servers on the network would see data packets from every other computer.
3) Improving network security. Subnetting allows network administrators to reduce network-wide threats by quarantining compromised sections of the network and by making it more difficult for trespassers to move around an organization's network.

INTERNET GATEWAY: An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between your VPC and the internet. It supports IPv4 and IPv6 traffic. It does not cause availability risks or bandwidth constraints on your network traffic.
ROUTE TABLE: A route table is a data table with a set of rules used to determine where data packets travelling over an internet protocol network will be directed. All IP enabled devices including routers and switches use route tables.

Policy and Permission

Pavani — Wed, 30 Nov 2022 12:13:08 +0000

A policy is an object in AWS when associated with an identity or resource defines their permissions. AWS evaluates these policies when an IAM principal (user or role) makes a request. Permission in the policies determine the request is allowed or denied.
-> IAM policies define permissions for an action regardless of the method that you use to perform the operation.
-> AWS supports six types of policies: identity-based policies, resource-based policies, permissions boundaries, Organizations SCPs, ACLs, and session policies.
Policy types:
The following policy types, listed in order from most frequently used to less frequently used, are available for use in AWS.
.Identity-based policies – Attach managed and inline policies to IAM identities (users, groups to which users belong, or roles). Identity-based policies grant permissions to an identity.
. Resource- based policies: Attach inline policies to resources. The most common examples of resource-based policies are Amazon S3 bucket policies and IAM role trust policies. Resource-based policies grant permissions to the principal that is specified in the policy. Principals can be in the same account as the resource or in other accounts.

. Permissions boundaries – Use a managed policy as the permissions boundary for an IAM entity (user or role). That policy defines the maximum permissions that the identity-based policies can grant to an entity but does not grant permissions. Permissions boundaries do not define the maximum permissions that a resource-based policy can grant to an entity.

. Organizations SCPs – Use an AWS Organizations service control policy (SCP) to define the maximum permissions for account members of an organization or organizational unit (OU). SCPs limit permissions that identity-based policies or resource-based policies grant to entities (users or roles) within the account, but do not grant permissions.

. Access control lists (ACLs) – Use ACLs to control which principals in other accounts can access the resource to which the ACL is attached. ACLs are similar to resource-based policies, although they are the only policy type that does not use the JSON policy document structure. ACLs are cross-account permissions policies that grant permissions to the specified principal. ACLs cannot grant permissions to entities within the same account.

. Session policies – Pass advanced session policies when you use the AWS CLI or AWS API to assume a role or a federated user. Session policies limit the permissions that the role or user's identity-based policies grant to the session. Session policies limit permissions for a created session, but do not grant permissions. For more information, see Session Policies.