It is a subset of machine learning that focuses on using neural networks with many layers to model and understand complex patterns in data.

Neural network is a type of machine learning algorithm that are designed to learn from data by adjusting the weights of connections between neurons based on the errors in their predictions.

Neuron

The fundamental unit of a neural network is the artificial neuron, often just called a neuron. An artificial neuron is inspired by the biological neurons in the human brain and is responsible for performing a small, specific computation in the network.

1) Each neuron receives one or more inputs, processes them (often by applying a mathematical function), and then produces an output.

2) The neuron typically applies a weighted sum of its inputs followed by an activation function to introduce non-linearity. The output of this function is then passed on to the next layer of the network or serves as the final output if it's in the output layer.

Perceptron (Single layer network)

Inputs: The perceptron receives several inputs, each represented by a floating-point number.

Weights: Each input is multiplied by a corresponding weight, which is also a floating-point number. The weight determines the importance of the input in the decision-making process.

Summation: The weighted inputs are then summed together to produce a single value.

Threshold (or Bias): The perceptron compares the result of the summation to a threshold value

Output:

If the summation is greater than 0 (or the threshold), the perceptron outputs +1 (or 1 in some versions).

If the summation is less than or equal to 0, the perceptron outputs -1 (or 0 in some versions).

(Note: Perceptrons are limited to solving problems that are linearly separable, meaning they can only classify data that can be separated by a straight line)

Most interesting problems, and even some very simple ones, were provably beyond the ability of a perceptron to solve.This period, which lasted roughly between the 1970s and 1990s, was called the AI winter

AI winter

This period was marked by disappointment with early AI technologies like expert systems, which struggled with scalability and real-world application. As a result, funding from governments and organizations dried up, and research in AI slowed down significantly.

Modern Neural Network after AI winter

1) Change 1:

It is the addition of an extra input called the bias. Unlike the other inputs, the bias is not tied to any external data or output from previous neurons.

The bias is a constant value that is directly added to the sum of the weighted inputs. It acts as a separate parameter that each neuron has, and it helps adjust the output independently of the input values.

2) Change 2:

Instead of just comparing the sum to a threshold and outputting -1 or 1, we can pass the sum (including the bias) through a mathematical function. This function will output a new floating-point value that can be anywhere within a certain range

Activation/Mathematical/Transfer Function

It determines how "active" the neuron will be based on the inputs it receives. Many activation functions introduce non-linearity, allowing the network to learn non-linear relationships, which is crucial for solving more complex problems.

Ex.

Sigmoid Function: Outputs values between 0 and 1. Useful for binary classification problems.

Tanh (Hyperbolic Tangent) Function: Outputs values between -1 and 1. It’s similar to the sigmoid but centered at 0.

ReLU (Rectified Linear Unit): Outputs the input if it's positive, otherwise 0.

Leaky ReLU: Similar to ReLU, but allows a small, non-zero gradient when the input is negative, helping to avoid the "dying ReLU" problem.

b. Unit Step Function:

Outputs 0 for input values less than a threshold and 1 for input values equal to or greater than the threshold.

b. Leaky ReLU

Function definition:

For x≥0:

f(x)=x

For x<0:

f(x)=αx (where α is a small constant, e.g., 0.01)

Output Shapes in Neural Networks

Zero-Dimensional Array

Ex.

If a neural network layer has only one neuron, its output is a single scalar value. Mathematically, this output can be represented as a zero-dimensional array.

One-Dimensional Array (1D Array)

Ex.

When a layer in a neural network has multiple neurons, the output can be described as a list or vector of values. For instance, if a layer contains 12 neurons, the output is a 1D array with 12 elements.

(Note: No matter how big or complicated our neural network is, if it has no activation functions and they are linear functions say addition,subtraction etc; then it will always be equivalent to a single neuron.)

Tensor

A general term used for an array of numbers arranged in a box-like shape with any number of dimensions. It encompasses one-dimensional (vector), two-dimensional (matrix), three-dimensional (volume), and higher-dimensional arrays.

High-Level Overview of Training Neural Networks

Training neural networks involves adjusting the network's weights to minimize errors in predictions. This is done through a process of iteratively updating the network's parameters to reduce a cost or loss function

Autoencoder

They are a type of neural network used for unsupervised learning. The key idea is to compress the input into a lower-dimensional code and then reconstruct the original input from this code.

Structure

Encoder:

This part compresses the input data into a compact representation.

Example: For an image, the encoder might reduce its dimensions from, say, 128x128 pixels to a smaller vector, like 32-dimensional

Decoder:

This part reconstructs the original input data from the compressed representation.

Example: The decoder would take the 32-dimensional vector and try to recreate the 128x128 pixel image.

Training Process

They are trained to minimize the difference between the input and the reconstructed output. This is usually done using a loss function, such as Mean Squared Error (MSE) for continuous data or binary cross-entropy for binary data. The goal is to adjust the weights of the network so that the reconstruction is as close as possible to the original input.

Variants:

1) Denoising Autoencoders

2) Variational Autoencoders

3) Sparse Autoencoders

Types of compression:

1) Lossless:

It is a type of data compression where the original data can be perfectly reconstructed from the compressed data. This means no information is lost during the compression process, and the decompressed data is identical to the original.

Algorithms: Use methods like entropy encoding and dictionary-based techniques. Examples include:Huffman Coding: Encodes frequently occurring symbols with shorter codes and less frequent symbols with longer codes.Lempel-Ziv-Welch (LZW): Builds a dictionary of sequences from the data and uses shorter codes for common sequences.Run-Length Encoding (RLE): Compresses sequences of repeated characters by storing the character and its count.Ex. PNG,FLAC,ZIP

2) Lossy:

It reduces file size by removing some of the data, often in a way that is less noticeable to the human senses but results in some loss of fidelity. The goal is to achieve a significant reduction in file size while maintaining acceptable quality for the intended use.

Ex. JPEG,H.264 or HEVC, MP3Transform Coding: Converts data into a different domain (like frequency domain) and quantizes it. Examples include:Discrete Cosine Transform (DCT): Used in JPEG image compression.Discrete Wavelet Transform (DWT): Used in JPEG 2000.

Application:

1) Dimensionality Reduction

2) Denoising

Difference

Optimizer: Adjusts weights to minimize the loss function.

Loss Function: Measures how well the model's predictions match the actual values.

Activation Function: Adds non-linearity to the model, enabling it to learn complex patterns.

Stay Connected!

If you enjoyed this post, don’t forget to follow me on social media for more updates and insights:

Twitter:

LinkedIn: madhavganesan