Fork() Bomb

Prerequisite : fork() in C
Fork Bomb is a program that harms a system by making it run out of memory. It forks processes infinitely to fill memory. The fork bomb is a form of denial-of-service (DoS) attack against a Linux based system.
Once a successful fork bomb has been activated in a system it may not be possible to resume normal operation without rebooting the system as the only solution to a fork bomb is to destroy all instances of it.

C program for Fork Bomb

C

// C program Sample  for FORK BOMB
// It is not recommended to run the program as
// it may make a system non-responsive.
#include <stdio.h>
#include <sys/types.h>

int main()
{
    while(1)
       fork();
    return 0;
}

Bash Script for Fork Bomb

Note : Please do not run this command to ‘test’ it unless you are prepared for a crash and/or force-rebooting your system. Also, it doesn’t need root to run.
If you using terminal then bash script for fork() bomb script as below.

:(){ :|: & };:

Step by Step Explanation of the script:

1. :() means you are defining a function called :
2. {:|: &} means run the function: and send its output to the : function again and run that in the background.
3. • : – load another copy of the ‘:’ function into memory
   • | – and pipe its output to
   • : – another copy of ‘:’ function, which has to be loaded into memory
   • Therefore, ‘:|:’ simply gets two copies of ‘:’ loaded whenever ‘:’ is called
   • & – disown the functions, if the first ‘:’ is killed, all of the functions that it has started should NOT be auto-killed
   • } – end of what to do when we say ‘:’
4. ; Command Separator
5. : runs the function first time

Essentially you are creating a function that calls itself twice every call and doesn’t have any way to terminate itself. It will keep doubling up until you run out of system resources.

How it Works

Fork bombs operate both by consuming CPU time in the process of forking, and by saturating the operating system’s process table. A basic implementation of a fork bomb is an infinite loop that repeatedly launches new copies of itself. 
To incapacitate a system, they rely on the assumption that the number of programs and processes which may execute simultaneously on a computer. fork() will generate new process but if you put this process in while true loop, then it will create many processes and when the limit is crossed, your system will crash.

Way to prevent the fork() Bomb

• Avoid use of fork in any statement which might end up into an infinite loop.
• You can limit the process of fork as below:-
  Just login as root, and edit this file, to add users and configure, their limit.

# vi /etc/security/limits.conf

Edit the file as:

 your_user_name hard nproc 10

• You can try Running the command in Virtualbox if you want to run it.
• Direct power off your system just in case you have run it and not finding a way out to proceed.

References:

• Wikipedia
• Ask ubuntu