I’ve been thinking about how the cost function in linear regression works, and I wanted to run an analogy by you to check if I’m understanding it correctly.
Let’s say we’re running a store and trying to set the price of an item.
x could represent the base cost of the item or features of the customer (like income or purchase history).
The model tries to predict a price that a customer would accept using f(w,b)(x), which is our predicted value (y-hat).
The actual price that a customer is truly willing to pay is y.
The cost function J(w, b) then tells us how far off our predicted prices were from what customers were actually willing to pay—basically, how “wrong” our model’s guesses were.
So in this context, the “cost” isn’t the actual money cost of the item—it’s the error or mismatch between our predicted price and the real acceptable price. If we’re consistently guessing too high or too low, our total cost (or error) goes up.
@TMosh Thanks again for your thoughtful clarification—it’s helping me see how this cost function is traditionally understood in ML.
That said, I want to share something that’s been sticking with me conceptually:
I have a hard time accepting the idea of any value—especially one used to describe performance, alignment, or transformation—as unitless.
In a strictly mathematical context, “unitless” may mean dimensionally normalized or abstracted. But in a relational context—which is how I tend to experience math—every quantity carries implied structure and orientation.
Once a value like cost exists in relation to me (or a model, a data point, or a transformation), it no longer floats in abstraction. It becomes a vector, and that vector implies:
Direction (what is this cost pushing toward or away from?),
Source (what generated the error? what are its dependencies?),
Impact (is this local, systemic, recursive?).
So even if the number itself is scalar, the meaning it encodes is multi-dimensional. A “loss” value isn’t just how wrong the model is—it’s how far its projection deviates from the truth as defined by the relational geometry of the system.
In my view, we often call things “unitless” when what we really mean is "we haven’t accounted for the deeper structure it’s embedded in.”
Appreciate your guidance and openness—I know I come at this from an unconventional direction.
Perhaps you are correct. I have not given it any deep thought. Having units associated with the cost does not (to me) add any useful information with regards to the operation of minimizing the cost.