# \-(Multi)variate intuition (?) **URL:** \ **Category:** Improving Deep Neural Networks: Hyperparameter tun\ **Tags:** week-module-1, coursera-platform\ **Created:** [March 10, 2024, 4:29pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846 "2024-03-10T16:29:23Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 4:29pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/1 "2024-03-10T16:29:23Z")
So, granted, I realize I am still early in the game on all this-- Thus perhaps this topic arises later, but one question has been kind of bothering me: The neural nets thus far, to put it in simple terms, seem like serial linear regression with ‘enhancements’. In many other cases though you would not have just (and I am not sure how to label this correctly and not confuse with what Prof. Ng as taught, so I will use {} as my signifer) W.TX + b, but W{1}.TX1 + W{2}.TX2 + W{3}.TX3… + b. Or like this: ![image](https://global.discourse-cdn.com/dlai/original/3X/c/5/c5f974e99061a36f5cf8579db2136f3b38e6ec72.png) Of course this would make your model _way_ more complicated-- though certainly someone has already tried this (and also, otherwise I’m not quite sure how you would, upfront, instantiate feature engineering)– But, I guess my question is has anyone demonstrated that it is _better_ to toss all your data into X, with just one weight set, and then just let the network ‘figure it out’ ? Again, I just might have not reached that point in my studies yet where this is covered, but trying to think forward. ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [March 10, 2024, 5:54pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/2 "2024-03-10T17:54:04Z")
What you’re missing is that every layer includes a non-linear activation function. So it isn’t just a chain of linear regressions. ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 5:59pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/3 "2024-03-10T17:59:05Z")
@TMosh Yes, of course I understand that. Perhaps I do not always express things in the best way. I guess what I am asking is why don’t we do a multivariate version of the same thing (?) And if not, why not (or what makes the network itself somehow ‘better’) ? ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [March 10, 2024, 6:15pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/4 "2024-03-10T18:15:47Z")
Aren’t your B1, B2, etc just weight values? ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 6:31pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/5 "2024-03-10T18:31:19Z")
@TMosh, I mean yes, but at least in a ‘classical’ situation they are seperable, or to say seperable indepent variables, so you are increasing your degrees of freedom as you work through the equation. Perhaps the hidden nodes in the network are, somehow, effectively performing the same thing (?). Again this is an ‘I don’t know’ kind of question, which is why I’m asking. But to be explicit, yes, your B1, B2, etc are weight values, but the node at every layer is now dealing with multiple independent variables, and thus seperate weight sets. If you just ‘lump’ the data all on a single weight set you will lose that effect. Though, as stated, I understand your network calculations would get _really_ complicated. Perhaps this is just a topic I haven’t gotten to yet, or someone _must_ have already tried this and decided it doesn’t work that great-- So I’m curious as to why. ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [March 10, 2024, 6:36pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/6 "2024-03-10T18:36:29Z")
I’m struggling to understand the details of your idea. If you are proposing using a different set of weights for every example in the data set, 1) how are you going to train it, and 2) how are you going to avoid overfitting? ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 7:21pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/7 "2024-03-10T19:21:34Z")
@TMosh so ‘no, no, no’-- I mean I am relying heavily on my knowledge of traditional regression here. I think I need some time to come up with a better example, but just of the cuff let’s assume we are trying to predict house prices. For this lets assume you decide you have two datapoints you consider are important-- the price of the house, and then a classifer (let’s say the color of the house, white = 0, black = 1, blue = 2)-- Essentially these are each ‘independent’ variables, but now you have two datasets heading into the network: X1 - House Price, X2 - House Color. I am not suggesting a different set of weights for _every_ example-- Rather what I am saying is you have two sets of weights for all the data going in W1 (at the first layer for the price) and W2 (for the color). Now these are kind of ‘competing’ with one another as the network develops and of course adjust your loss and back-prop functions accordingly to account for the fact you are dealing with two independent input variables. In contrast, with what we’ve done so far maybe you could like ‘sneak’ this info in-- Say the house is $100,000 and so you do adjust the data for $100,000 for a white house, $100,001 for a black house, $100,002 for a blue house. I mean _maybe_ it would still figure it out… Yet it doesn’t strike me as the best way to do it-- Plus you are over-biasing your data in the process. ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 7:44pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/8 "2024-03-10T19:44:45Z")
Or you could also alternatively say your only X variable (for each example) is [house\_price, color]. I guess I am just not sure why can give up on the independece of variables this way, and it still works (well). ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [March 10, 2024, 8:38pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/9 "2024-03-10T20:38:17Z")
Well, apparently the model can still learn regardless of whether the inputs are all independent or they’re not. Why is that surprising? Maybe it’s suboptimal in some sense, but the question is which is more practical: trying to figure out how to preprocess your data into genuinely independent inputs and then training the model or just letting the model figure it out. ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 9:13pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/10 "2024-03-10T21:13:05Z")
Dear Paul, Yeah no I agree and the more I play I might just try this idea out for myself. Though, in a very distant way, this also reflects a concept I’ve long been considering-- I mean I think we all know the human brain, especially in terms of say visual representation, or fast learning is way more efficent than AI. In my mind that is not because we are ‘born’ with innate knowledge, yet our brains do come with a certain evolutionary structure. Yet if you consider the way most of AI is trained today, the problem is structured as going right to the solution (i.e. ‘Cat ?’ ‘Not Cat ?’). And though my thought might be more oriented towards ‘older’ thoughts of AI, that it would be a symbolic rather than empirical enterprise, in the end I don’t think that is not totally off. I know Prof. Ng himself strays away from this concept, and I agree, current structures of Neural Networks do not, at all, work like the brain. I mean, as the famous Oliver Sacks brought up, in the case of the condition of prosopagnosia, we don’t ‘learn’ faces the way networks do. We are born and just ‘kind of know them’. Unfortunately people with such a condition just ‘don’t’. Visually, everything is there, they just can’t put it together. In the same way, eventually I actually don’t think the solution is just more and more data-- But rather providing the proper shape of the inputs (and perhaps this itself could be the product of yet another neural net ?). Yet, what that training set would be, specific to each problem, honestly I have no idea yet. Nor is this really the question here I first asked-- I’m just trying to make sure I understand the ground theory at this point so I can trace its edges. ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [March 10, 2024, 9:27pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/11 "2024-03-10T21:27:52Z")
We’re into pretty deep (pun fully intended) waters here. Of course there are deep divisions among the practitioners in the field as to whether the current “massive data” based Deep Learning can create something that actually qualifies as AGI. And exactly as you say, it’s apparent that the way our actual brains learn things and encode knowledge is pretty clearly not equivalent to how artificial Deep Networks learn things. I’m not a neuroscientist, but my understanding from what I’ve read and heard is that even neuroscientists don’t yet have a model they can all agree on for how the brain actually works, even at the level of memory or recognizing pictures of cats. Let alone something like “consciousness”. What actually is it and what biological mechanisms give rise to it? How can you tell if a bat is conscious or not? These are fascinating things to think about, but my personal reaction is that you’re getting a bit ahead of yourself here. My suggestion is just to relax a bit and listen and learn what Prof Ng has to show us here. Once you’ve absorbed all of that, you’ll be ready to figure out how you want to take it further. But each of us has our own preferred learning style, so you can do it your way. ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 9:36pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/12 "2024-03-10T21:36:03Z")
No-- I agree, and honestly I am not all that impressed by this ‘hype’ around AGI, or even generative AI. I am kind of ‘traditionalist’. Thus please note my original question was why are we running these serial regressions with, basically only one β. I mean if you did that with standard regression on an advanced data set, you’d probably end up with a pretty lousy model… ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [March 10, 2024, 9:48pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/13 "2024-03-10T21:48:40Z")
> [@Nevermnd](#): > > Thus please note my original question was why are we running these serial regressions with, basically only one β. I don’t understand your point. As Tom said earlier in the conversation, each layer in the network is a non-linear function. We chain those together. “Composition of functions” is the mathematical term. When you compose non-linear functions, the non-linearity “compounds”. The nerd joke way to say it is that it gets “more non-linear”, but of course that’s absurd. But you can quantify it if you consider composing polynomial functions: the degree of the composite function is higher, right? I’m not saying we are using polynomial functions here. Note that I’m pretty sure we already had this conversation, right? It’s got a definite _deja vu_ feel to it, but maybe that was a thread with someone else. So the “magic” happens by adding more layers. In this way, the model can learn a decision boundary which is essentially arbitrarily complicated. There is the Universal Approximation Theorem, right? Of course that’s the classic mathematical “existence proof”: it gives you absolutely no help actually finding the function or putting any bounds on how expensive it is to compute. So what are you talking about with being limited to “only one \beta”? If you are talking about the bias term, there is one in each layer, right? ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 10, 2024, 10:02pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/14 "2024-03-10T22:02:03Z")
I may be missing something here, but this is traditional multiple term regression right ? [https://online.stat.psu.edu/stat462/node/132/](https://online.stat.psu.edu/stat462/node/132/) Yet here we substitue β for our weight vector, no ? So I am not at all talking about our bias term. And I think the conversation you had… Was with someone else (?) But I will look up the ‘Universal Approximation Theorem’ as that is not something I am familar with… ---
**Author:** ![paulinpaloalto](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/paulinpaloalto/32/187_2.png) [@paulinpaloalto](https://community.deeplearning.ai/u/paulinpaloalto)\ **Post date:** [March 10, 2024, 11:05pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/15 "2024-03-10T23:05:57Z")
> [@Nevermnd](#): > > And I think the conversation you had… Was with someone else (?) But I will look up the ‘Universal Approximation Theorem’ as that is not something I am familar with… The deja vu part that I was referring to is not the UAT paragraph, but the previous bit about composing non-linear functions and how to consider the polynomial case as a way to see how composing non-linear functions gives you the ability to express more and more complex functions. My day is about to get fully booked with an event we need to go to tonight, so I will not be able to read up on multiple term regression and how that applies here or doesn’t. I hope to have something useful to say by tomorrow midday (UTC -7 as of this morning). ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [March 11, 2024, 2:33am UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/16 "2024-03-11T02:33:28Z")
Hello @Nevermnd, > [@Nevermnd](#): > > Rather what I am saying is you have two sets of weights for all the data Is the following what you have in mind? That there are two sets of weights \beta\_1 and \beta\_2 which accept the same set of inputs X? ![image](https://global.discourse-cdn.com/dlai/original/3X/a/a/aab68a961d71b1333e8887d5af22486270c44044.png) Please feel free to draw a different flow chart if you have something different in mind 😉 Cheers, Raymond ---
**Author:** ![Nevermnd](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/nevermnd/32/126021_2.png) [@Nevermnd](https://community.deeplearning.ai/u/Nevermnd)\ **Post date:** [March 11, 2024, 10:05am UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/17 "2024-03-11T10:05:30Z")
@rmwkwok – So your idea is interesting; But again my whole point here is that of a question, not neccesarily the proposition of a novel method. For simplicities sake I’ve included only a single node here to be clear, but I am basically wondering why we do this: ![image](https://global.discourse-cdn.com/dlai/original/3X/b/a/ba87980b259e4ba5e9fd2ec91b38ce7e5c363fb4.png) And not this: ![image](https://global.discourse-cdn.com/dlai/original/3X/1/d/1da8f205a61746870d81836e8b5e25cbc65b50c5.png) I mean I am not sure (and I had to alter your numbers a bit for understanding). Perhaps researchers have just found the second method works ‘well enough’. That is what I don’t know… ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [March 11, 2024, 11:57am UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/18 "2024-03-11T11:57:05Z")
Hello @Nevermnd, Thanks for the diagrams!! > [@Nevermnd](#): > > But again my whole point here is that of a question, not neccesarily the proposition of a novel method. No problem with that! At least we have to be on the same page on what you are questioning about. If the approach of your concern is indeed novel, then to prove or not is driven by curiosity. If the approach is not novel, then we need to know it is not novel. That’s it! It seems to me that your second diagram (the one you wonder why not doing) is the same as my original diagram without the second node. I erased the second node, put yours next to mine, and get the following: ![image](https://global.discourse-cdn.com/dlai/original/3X/8/e/8e68a3a08157ed0e94c5e360ef65600667d26c2a.png) I think the first thing is to clarify the use of symbols. Let’s examine them. My X composes of two x’s. In our convention, we use the big X for a set of features, and use the small x’s for each of our features. I have two features: house size and house color, so I call them x\_1 and x\_2 respectively. Each of them takes one number as its value. You have two features too, but you used the big X’s for them. Such use may be confusing, becuase generally we use a small x for a feature and the big X for a set of features. If you refer to other online material, the big X there should be for a set of features too. For example, below comes from [the link](https://online.stat.psu.edu/stat462/node/132/) you shared: ![image](https://global.discourse-cdn.com/dlai/original/3X/b/8/b8c778b6f9c0ac8660812c5d0dca0bc95c9ee3f7.png) It has only one feature, so it is simpler than our diagrams, but the spirit is the same - small x for a feature, and the big X for the whole set of features. They use \beta\_0 and \beta\_1 as the symbols for weights, but in our diagrams, we call them b and w\_1 instead. If we follow the convention, I would change your symbols into: ![image](https://global.discourse-cdn.com/dlai/original/3X/d/7/d751ea1703f4c73ff1dd23cb8c691d8dcdc698a4.png) and this makes the two diagrams the same. We have one weight (your W\_1 or your \beta\_1 or my w\_1) that is multiplied to the first feature (your X\_1 or my x\_1), plus another weight (your W\_2 or your \beta\_2 or my w\_2) that is mulltipled to the second feature (your X\_2 or my x\_2), plus a bias (you and I both call it b). Essentially, your W\_1 or your \beta\_1 or my w\_1 has one number as its value. Similarly, your W\_2 or your \beta\_2 or my w\_2 also has one number as its value. So, @Nevermnd, after my careful examination, I think we have a different way of using symbols, but your diagram is not very different from mine. What do you think? If you think my interpretation of your symbols is not quite your way, can you point them out? If you think my interpretation is correct but some changes are needed to your diagram to show other difference, would you mind share a new diagram with the difference but using the symbols according to the convention? Following the convention helps all readers understand your ideas quickly! Cheers, Raymond ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [March 11, 2024, 12:07pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/19 "2024-03-11T12:07:59Z")
And by saying our two diagrams are indeed the same, my answer to your question of “why doing _this_ but not **that** ” is that, we ARE indeed doing **that** , but not _this_. We do NOT combine two features into ![image](https://global.discourse-cdn.com/dlai/original/3X/f/9/f9d3df5103261e8bead5c312f0269527b00d926e.png). We keep them separate, and call them small x\_1 and small x\_2 almost like your “ **that** ” diagram: ![image](https://global.discourse-cdn.com/dlai/original/3X/c/c/cc106b489dd459ba2c6b2c40e78fe977d6474049.png) Cheers, Raymond ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [March 11, 2024, 3:46pm UTC](https://community.deeplearning.ai/t/multi-variate-intuition/587846/20 "2024-03-11T15:46:00Z")
@Nevermnd, this just comes up in my mind and in case it is the centre of your question, please let me know. One thing that my diagram did not show is the training process. My training process is that my w\_1 and w\_2 and b are trained at the same time as one model. I suppose your diagram was doing the same thing. I suppose you were _not_ training a model with one of the two features as y=w\_1x\_1, and then training another model with the other feature as y=w\_2x\_2, then combining the two models into y=w\_1x\_1 + w\_2x\_2 +b with an extra b to be trained. Such three-step training process is not something I am supposing you thinking. If you think there is some difference in the training process that you need to point out, then please share your sequence of the training process. Cheers! Raymond [Next page](https://community.deeplearning.ai/t/multi-variate-intuition/587846.md?page=2)