Breaking Down Cost Functions in Linear Regression: A Conceptual Overview

The cost function is a crucial concept in machine learning, helping us understand how well our models are performing. It's the tool that tells us how close our model's predictions are to the actual results and guides us in improving accuracy. In this post, we'll break down the cost function in simple terms, with a focus on linear regression.

Introduction to Cost Functions

The cost function tells us how well our model's predictions match the actual target values. Essentially, it measures the error between the predicted values and the true values. By minimizing this error, we can improve our model's accuracy.

Consider you have a training set with input features


$x$

and output targets

$y$

.

The model you use to fit this training set can be represented by a linear function:

fw,b(x)=w⋅x+bf_{w,b}(x) = w \cdot x + bfw,b​(x)=w⋅x+b

For a training example

(x(i),y(i))(x^{(i)},y^{(i)} )(x(i),y(i))

, the function

$f$

predicts

y(i)y^{(i)}y(i)

as

y^(i)\hat{y}^{(i)}y^​(i)

. Thus:

y^(i)=fw,b(x(i))=w⋅x(i)+b\hat{y}^{(i)} = f_{w,b}(x^{(i)}) = w \cdot x^{(i)} + by^​(i)=fw,b​(x(i))=w⋅x(i)+b

The challange is to find

$w$

and

$b$

that make

y^(i)\hat{y}^{(i)}y^​(i)

close to

y(i)y^{(i)}y(i)

for all training examples.

Here,

$w$

and

$b$

are the parameters of the model.These parameters are adjusted during training to enhance the model's performance.

• When
  
  $w=0.5$
  
  and
  
  $b=0$
  
  

$$f(x)=0.5\cdot x$$

The slope is 0.5, creating a line that increases steadily.

Understanding Errors

The cost function calculates the error between the predicted prices

y^\hat{y}y^​

and the actual prices

$y$

. This error is given by:

Error=y^−y\text{Error} = \hat{y} - yError=y^​−y

Then, we square this error to avoid negative values. This squaring operation ensures that all errors are positive and emphasizes larger errors more than smaller ones. It will be:

Error=(y^−y)2\text{Error} = (\hat{y} - y)^2Error=(y^​−y)2

Here, if we have more training examples, the sum of the errors will naturally be larger. To normalize this, we use the average squared error instead of the total squared error to get a sense of the overall performance. This way, the cost function doesn't automatically get bigger just because we have more training examples. It makes it a fair comparison, no matter how big our dataset is. Dividing by the number of examples

$m$

:

Mean Squared Error

=1m∑i=1m(y^(i)−y(i))2= \frac{1}{m} \sum_{i=1}^{m} (\hat{y}^{(i)} - y^{(i)})^2=m1​∑i=1m​(y^​(i)−y(i))2

To simplifies the derivative calculations during optimization (like gradient descent), we add factor of

12\frac{1}{2}21​

in the cost function. Thus, the final of cost function formula will be:

J(w,b)=12m∑i=1m(fw,b(x(i))−y(i))2J(w, b) = \frac{1}{2m} \sum_{i=1}^{m} (f_{w,b}(x^{(i)}) - y^{(i)})^2J(w,b)=2m1​i=1∑m​(fw,b​(x(i))−y(i))2

The extra division by 2 is a bit of a mathematical trick to make later calculations easier, especially when we use calculus to minimize the cost function.

Why Do We Square the Error?

Imagine we're trying to predict something—like the price of a house. Our model makes a prediction

(y^)(\hat{y})(y^​)

and we compare it to the actual price

$(y)$

. The error is the difference between these two:

Error=y^−y\text{Error} = \hat{y} - yError=y^​−y

But here's the thing: this error can be positive or negative.

• If your prediction is higher than the actual value, the error is positive.


• If your prediction is lower than the actual value, the error is negative.

Example:

• Predicted price
  
  (y^)(\hat{y})(y^​)
  
  : $300,000


• Actual price
  
  $(y)$
  
  : $280,000

The error is 300,000 − 280,000 = 20,000 (positive error)

But:

• Predicted price
  
  (y^)(\hat{y})(y^​)
  
  : $250,000


• Actual price
  
  $(y)$
  
  : $280,000

The error is 250,000 − 280,000 = −30,000 (negative error)

If we simply add up these errors, positive and negative values can cancel each other out, which wouldn't give us the real picture of how well our model is performing.

This helps us understand the real performance of our model and work on making better predictions.

Why Not Use Absolute Value to Avoid Negative Value?

When we square the errors, larger errors have a bigger impact. For example, an error of 10 becomes 100 when squared, while an error of 1 becomes 1. This helps the model focus on reducing larger mistakes more aggressively. The squared error function also smooth and differentiable everywhere. This smoothness is important for optimization algorithms like gradient descent because it allows for more efficient and predictable convergence to the minimum error.

Meanwhile, when we use absolute error, each error contributes linearly. An error of 10 remains 10, and an error of 1 remains 1. Both are treated equally without any extra emphasis on the larger ones. The absolute value function has a kink at zero, meaning it's not differentiable at that point. This can complicate the optimization process, making it harder to find the minimum error.

In summary, squaring the error puts more emphasis on larger mistakes, which helps in creating a better model overall by addressing those big errors more effectively. This is why squared errors are often preferred in many machine learning applications.

Why Divide by

$2m$

and Not

$100m$

?

You may wonder, if our goal is to avoid larger numbers in the cost function as our data set gets bigger, why should we divide it by 2m? Why 2? Why not 100 or some other number? As I mentioned before, the extra division by 2 is a bit of a mathematical trick to make later calculations easier. Specifically, The factor of 2 cancels out when we take the derivative, simplifying our calculations.

When training the model, we often use optimization algorithms like gradient descent to minimize the cost function. Gradient descent involves taking the derivative (gradient) of the cost function with respect to the parameters

$w$

and

$b$

. The gradient tells us how to change the parameters to reduce the cost.

Consider a simple function:

When we take the derivative of this function with respect to

$w$

:

The derivative produces a 2 from the squared term. This 2 can make the gradient calculations a bit cumbersome. By including a factor of 1/2 in the cost function, we simplify the gradient calculations. This adjustment doesn't change the ultimate goal (minimizing the cost), but it makes the math cleaner:

Now, when we take the derivative, the factor of 2 cancels out:

Why Not Another Number?

If we used a different number, like 100, in the cost function, the math wouldn't simplify as neatly:

Taking the derivative:

Here, the factor 1/50 doesn't simplify as nicely, and we end up with more complex expressions. This also means your model will learn very slowly. You would have to compensate by increasing the learning rate 𝛼, but this requires careful tuning to avoid making the model unstable.

Using

12m\frac{1}{2m}2m1​

is a balanced choice. It simplifies the gradient calculations without making the steps too small or too large. It’s also a widely-accepted convention, which makes it easier to follow standard practices and compare results across different studies and implementations.

Visualizing the Cost Function

In linear regression, the objective is to find the optimal values for the parameters

$w$

and

$b$

that minimize the cost function

$J(w,b)$

. This is typically achieved through an optimization algorithm, such as gradient descent, which iteratively adjusts

$w$

and

$b$

to reduce the difference between the predicted outputs and the actual target values.

To illustrate this concept, let's work with a simplified version of the linear regression model:

fw(x)=w⋅xf_w(x) = w \cdot xfw​(x)=w⋅x

In this model, we've eliminated the parameter

$b$

. Now, the cost function looks like this:

J(w)=12m∑i=1m(wx(i)−y(i))2J(w) = \frac{1}{2m} \sum_{i=1}^{m} (wx^{(i)} - y^{(i)})^2J(w)=2m1​i=1∑m​(wx(i)−y(i))2

The goal is to find the value of

$w$

that minimizes

$J(w)$

. Let's visualize how the cost function changes with different values of

$w$

.

The graphs below show both the function

fw(x)f_w(x)fw​(x)

(left) and the corresponding cost function

$J(w)$

(right) for four different values of

$w$

:

1. 
   

   When
   
   $w=1$
   
   : The function
   
   fw(x)f_w(x)fw​(x)
   
   is a line with a slope of 1, and the cost
   
   $J(w)$
   
   is 0 for perfectly fitting data points and resulting in a lower cost.

   
   
   
   
   
   • 
     Function:
     
     $f(x)=1\cdot x$
     
     
   
   
   • 
     Graph:
     
     
   
   
   
   


2. 
   

   When
   
   $w=0.5$
   
   : The function
   
   fw(x)f_w(x)fw​(x)
   
   has a slope of 0.5, leading to a higher cost due to the error between predicted and actual values.The line does not fit the data well.

   
   
   
   
   
   • 
     Function:
     
     $f(x)=0.5\cdot x$
     
     
   
   
   • 
     Graph:
     
     
   
   
   
   


3. 
   

   When
   
   $w=0$
   
   : The function
   
   fw(x)f_w(x)fw​(x)
   
   is a horizontal line, resulting in a significant error and a higher cost.

   
   
   
   
   
   • 
     Function:
     
     $f(x)=0\cdot x$
     
     
   
   
   • 
     Graph:
     
     The line is a flat line, which does not fit the data points at all, which also results in a high cost.
   
   
   
   


4. 
   

   When
   
   $w=-0.5$
   
   : The function
   
   fw(x)f_w(x)fw​(x)
   
   is a line with slopes downwards, showing an inverse relationship with the data points, resulting in the highest cost.

   
   
   
   
   
   • 
     Function:
     
     $f(x)=-0.5\cdot x$
     
     
   
   
   • 
     Graph:
     
     
   
   
   
   

Here's how the plots would look:

These graphs help visualize how different values of

$w$

affect the line that fits the data points and the corresponding value of the cost function

$J(w)$

. The goal is to find the value of

$w$

that results in the lowest cost, indicating the best fit for the data.

Choosing the Optimal Parameters

The goal is to choose the value of

$w$

that minimizes

$J(w)$

. This is achieved by selecting the value of

$w$

that results in the smallest possible value of the cost function. For instance, in our example, if choosing

$w=1$

results in the smallest

$J(w)$

, then

$w=1$

is the optimal parameter for our model.

Different Cost Functions for Different Applications

While the mean squared error cost function is the most commonly used for linear regression, different applications may require different cost functions. The mean squared error is popular because it generally provides good results for many regression problems.

Conclusion

The cost function is a fundamental concept in machine learning that helps us measure how well our model's predictions align with the actual target values. It guides the optimization process to improve the model's accuracy. Understanding and utilizing the cost function effectively can significantly enhance the performance of your machine learning models.