# Tips on applying regularization **URL:** \ **Category:** AI Discussions\ **Created:** [November 15, 2023, 8:24am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126 "2023-11-15T08:24:57Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 15, 2023, 8:24am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/1 "2023-11-15T08:24:57Z")
Hello Everybody! I’m trying to implement a regularization technique for my neural network because I suspect it is overfitting my dataset. I am using a simple one hidden layer DNN with 600 input units and 6 output units. By adding L2 regularization in the hidden layer with setting lambda to 0.001, I have some findings that are a bit confusing. 1. My accuracy in my testing set has dropped from 97% to 90% (which indeed decreased overfitting issue that I suspected). It decreased the the bias though. 2. The accuracy in my testing set has not increased. (still at around 65%) 3. I applied L2 regularization on the hidden layer and the softmax layer, it didn’t result in much difference. 4. Is the whole point of regularization about decreasing the bias so that I can train my network for longer without overfitting, and have a better result on my validation set? Thank you for reading my message. If I have any conceptual misunderstandings about regularization, please let me know. It will be much appreciated. Best, John ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 15, 2023, 9:10am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/2 "2023-11-15T09:10:01Z")
Hi John @Chiang_Yuhan, 1. In your point number 1, “my testing set” should have been “my training set”? 2. In your point number 1, I think it should be “It _increased_ the bias though”, not “decreased” 3. For your question 4, can you make a plot like the following for our discussion? 4. If you still have time, can you make one more plot like the above, but with an additional hidden layer of 60 units in between the current two Dense layers? Both plots use the same set of \lambda's please. Certainly the goal of regularization would be for your model to perform better with the testing set, but it is not a guarantee 😉 For example, I can set \lambda to a unreasonably large number to destroy everything. Let’s see what you will get. Raymond ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 15, 2023, 9:19am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/3 "2023-11-15T09:19:55Z")
Please also share how many training samples and features you have in each of your training and testing sets. ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 16, 2023, 6:25am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/4 "2023-11-16T06:25:36Z")
Hello! @rmwkwok! Thank you for your reply! > [@rmwkwok](#): > > In your point number 1, “my testing set” should have been “my training set”? You were right! I would be editing the post shortly! > [@rmwkwok](#): > > In your point number 1, I think it should be “It _increased_ the bias though”, not “decreased” And you’re also right, what I meant is it decreased the variance … I was a bit confused yesterday haha. I will be making the plot you mentioned shortly. ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 16, 2023, 6:29am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/5 "2023-11-16T06:29:45Z")
Shape: ( examples, features ) ![截圖 2023-11-16 14.27.18](https://global.discourse-cdn.com/dlai/original/3X/d/3/d30a25f9c1113d4ef945d29acefa4d93aa6947a2.png) Network (1 Hidden Layer DNN) 600 units → 100 units → 6 units ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 16, 2023, 7:13am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/6 "2023-11-16T07:13:45Z")
Hello @rmwkwok : I have finished my plot: ![截圖 2023-11-16 15.07.20](https://global.discourse-cdn.com/dlai/original/3X/5/e/5e4146a72574ebcb4ce7c21dc1561711c6b52a1f.jpeg) Here is the github repository: [GitHub - JakeDaDogg/DNN: Here is the DNN that I designed for my 1D data identification task](https://github.com/JakeDaDogg/DNN) ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 16, 2023, 7:40am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/7 "2023-11-16T07:40:43Z")
I also compared different number units in my hidden layer: I have set the number of units in my hidden layer as my variable and created 5 models, evaluated their accuracy after training for 300 epochs. I used lambda = 0.01 for this task, applied in the hidden unit and the softmax layer. ![截圖 2023-11-16 15.38.55](https://global.discourse-cdn.com/dlai/original/3X/5/a/5abc492178e9db766718b2cf49b42aed6e717292.png) ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 17, 2023, 12:25am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/8 "2023-11-17T00:25:14Z")
> [@Chiang\_Yuhan](#): > > ![截圖 2023-11-16 15.07.20](https://global.discourse-cdn.com/dlai/original/3X/5/e/5e4146a72574ebcb4ce7c21dc1561711c6b52a1f.jpeg) > > 截圖 2023-11-16 15.07.201264×1022 76.6 KB Hey @Chiang_Yuhan, This one’s behavior is very easily understandable. However, before I can really comment on it, I notice in your code that you didn’t use [EarlyStopping](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping), so I suppose this graph recorded the training and validation accuracies at the end of the 200th epoch. This is not good, because the it didn’t reflect the best validation accuracies. Please allow them to early stop and remake the graph with the stopped training and validation accuracies. With earlystop, your first model training run should finish within 40 epochs (saving 80% of your time). Raymond ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 17, 2023, 12:28am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/9 "2023-11-17T00:28:25Z")
> [@Chiang\_Yuhan](#): > > ![截圖 2023-11-16 15.38.55](https://global.discourse-cdn.com/dlai/original/3X/5/a/5abc492178e9db766718b2cf49b42aed6e717292.png) > > 截圖 2023-11-16 15.38.551212×928 62.4 KB This is a very interesting graph, especially how you picked the experimented units, only it would be more usable if EarlyStopping had been applied to them. ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 17, 2023, 12:35am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/10 "2023-11-17T00:35:22Z")
It might be helpful if you re-code in this way… 1. In the 3rd code cell, change your `DNN1_model(input_shape, r)` to `DNN1_model(hidden_units, r)`, since `input_shape` is actually fixed and you are experimenting the number of hidden units. 2. In the 4th code cell, use nested loops. The outer loop goes over different values of `hidden_units`, and the inner goes over the lambda. Then change a bit the way of how you store the final accuracies such that in your next graph, there will be as many training (and validation) curves as the number of `hidden_units`. 3. In the 5th code cell, in those `plt.plot` lines, add the `label` argument to give each curve a name. Then before `plt.show`, do `plt.legend` to show the legend that contains the names of the curves. Raymond ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 19, 2023, 3:51am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/11 "2023-11-19T03:51:21Z")
Thank you sir I will be coding this for a while 😉 I will get back to you as soon as I finish! Always appreciate your insights. John ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 20, 2023, 3:41am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/12 "2023-11-20T03:41:11Z")
Hello @rmwkwok I have finished recoding as you have taught me. Here are the plots with accuracy with respect to different lambda values. There are 5 plots which means different input value. I used early stopping at 40 epochs for all cases. ![截圖 2023-11-20 11.35.56](https://global.discourse-cdn.com/dlai/original/3X/8/b/8bdcc56b3677ccf297a81f5f7fa16a2e9a0c5153.png) ![截圖 2023-11-20 11.37.09](https://global.discourse-cdn.com/dlai/original/3X/2/7/274405f1fc8a708072a29d4c37942948e5af9e99.png) ![截圖 2023-11-20 11.36.36](https://global.discourse-cdn.com/dlai/original/3X/f/d/fd05668881a321561f02f8f3be7d6f59baead631.png) ![截圖 2023-11-20 11.37.26](https://global.discourse-cdn.com/dlai/original/3X/2/8/28d91ac19c05f5a951d8a882ed906cff8c2c5f6b.png) ![截圖 2023-11-20 11.37.41](https://global.discourse-cdn.com/dlai/original/3X/6/f/6f55814bd345f00e9e9d49db4b0eb8356e640683.png) To my surprise, the 30 hidden units performed better, it might stem from simpler training for the simpler network. (Thus my implementation of early stopping is wrong). I’ve also observed that regularization has more impact when there are more are more hidden units. I’d love to hear your thoughts! Best, Yuhan ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 20, 2023, 10:29am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/13 "2023-11-20T10:29:19Z")
Hi @Chiang_Yuhan, Would you mind to also update your repo with the code that generated these 5 plots? I want to make sure we are on the same page, because something doesn’t seem to make sense to me. Also, you said 30 units performed better, but based on validation accuracy, I would say 6 units is the one I would name. Some trends are understandable: - as lambda increased, increasing validation accuracy followed by decreasing - with more hidden units, validation accuracy dropped But some are hard to understand: - increasing training accuracies with higher lambda (plot 1, 2) - larger accuracy gaps with higher lambda (plot 1, 2) - with more hidden units, training accuracy dropped (across all plots) I need to inspect your code before I can make any further comment. In case you also want to do some inspection, before that, would you first update your repo and notify me? We might exchange our findings later. Raymond ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 21, 2023, 3:09am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/14 "2023-11-21T03:09:12Z")
Hello @rmwkwok I have updated the code uploaded into the repo. There were some problems like > [@rmwkwok](#): > > with more hidden units, training accuracy dropped (across all plots) is resolved. (I don’t know how I resolved this issue though, I only recoded because it didn’t save my file) Please refer to the file ‘1 hidden layer V3’. I would love to hear your insights soon! Best, Yuhan Chiang ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 21, 2023, 11:58am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/15 "2023-11-21T11:58:18Z")
Hey @Chiang_Yuhan, > [@rmwkwok](#): > > But some are hard to understand: > > - increasing training accuracies with higher lambda (plot 1, 2) > - larger accuracy gaps with higher lambda (plot 1, 2) > - with more hidden units, training accuracy dropped (across all plots) 1. Actually, all of the three problems are gone! Now their opposites are true, see if the opposites make sense to you. 2. You are still not using EarlyStopping. It does not mean fixing the number of epochs to a smaller value. You need to google for how to use [this class](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping) to add EarlyStopping to your training process. 3. I hope that, besides what I have suggested, in your next post, I will see your analysis of the situation, your furher work, and the latest results. For example, validation accuracy does increase in some cases when Layer 1’s units increases. However, the effect is not clear because you have to go up and down to compare some plots. Having said that, if you did not realize any improvement, you might need to think about what you can do to assist yourself better. For example, if you don’t scroll up and down to compare two plots, why not make them in one single plot? Another example, accuracies gaps close as lambda increases which is reasonable. However, dropping validation accuracy isn’t something we want. Therefore, there were both improvement and unfavourable ends, and they should inform you your next actions. @Chiang_Yuhan, I believe your goal with this work is to train yourself to be a good model trainer. So, the key is your work, not mine. Below is an experiment loop you can jump into: Experiment → observation → make sense of the observation → hypothesis → Experiment your hypothesis → observation-\> … If you still have no idea what to do, here are my suggestions: 1. Scan through your code character-by-character and make a list of all tunable parameters: 2. For each, ask yourself, would increasing it help overfitting or underfitting. 3. Observing overfitting/underfitting is easy because you can compare train/valid losses and accuracies 4. Based on your observation, and your list of parameters, what you are going to do next? 5. You may add a step in the loop to log down the latest best validation accuracy you have got and how you have achieved it. I won’t be suprised if you go through that experimentation loop 100 times yourself within a day, and if you do so, there are two consequences: 1. You are accumulating experience through ups and downs, which would be very resuable in your future. We can’t only know the ups, but if I keep pushing you to the downs, I can’t imagaine how you would feel. It’s best for you to follow your own instinct, and train it as you go through the loop. 2. You can have done way more in one day than just we exchanging messages here for two weeks. Cheers, Raymond PS1: I would love to read and discuss your list of tunable parameters with how they deal with over-/under-fitting. However, it’s up to you for whether you want to share it 😉 PS2: Don’t give up 😉 😉 😉 ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 21, 2023, 12:09pm UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/16 "2023-11-21T12:09:20Z")
> [@Chiang\_Yuhan](#): > > (I don’t know how I resolved this issue though, I only recoded because it didn’t save my file) We can put this behind. It’s good if you had known how you fixed it because it was totally only for your own gain, but if you hadn’t, it can’t be a smaller issue at all, because sooner or later, it will come back and you will find out. 😉 ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 21, 2023, 12:27pm UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/17 "2023-11-21T12:27:08Z")
Btw, remember the spirit of expeirmentation - one change at a time. This is the key for you to accurately credit/blame the up/down. ---
**Author:** ![Chiang\_Yuhan](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/chiang_yuhan/32/165954_2.png) [@Chiang\_Yuhan](https://community.deeplearning.ai/u/Chiang_Yuhan)\ **Post date:** [November 22, 2023, 2:42am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/18 "2023-11-22T02:42:25Z")
Got it sir! I will be trying things out in the next few days! ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 22, 2023, 11:39am UTC](https://community.deeplearning.ai/t/tips-on-applying-regularization/495126/19 "2023-11-22T11:39:00Z")
Take your time but make it count. The process is important. Raymond PS: I have been trying things too. We are no different in this. 😉