# How different initialization of centroids of K-means results in drastic different clusters ? They all share common cost function **URL:** \ **Category:** Unsupervised Learning, Recommenders, Reinforcement\ **Tags:** week-module-1\ **Created:** [October 29, 2022, 7:01am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900 "2022-10-29T07:01:22Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Tanvi](https://avatars.discourse-cdn.com/v4/letter/t/9de053/32.png) [@Tanvi](https://community.deeplearning.ai/u/Tanvi)\ **Post date:** [October 29, 2022, 7:01am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/1 "2022-10-29T07:01:22Z")
In the lecture, we studied that there is local optima and global optima. Also, one initialization will lead to local while other can lead to global optima. I do not understand how that happens. Is there a mathematical proof for that ? ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [October 29, 2022, 7:17am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/2 "2022-10-29T07:17:48Z")
Hi, @Tanvi It has to do with the data itself if your clusters have a well-defined boundary between them it will be confident that you will always find the same centroids and only will depend on the parameter k. but if your data does not have clear boundaries between each cluster so initializing the centroid randomly will give different results depending on the initialization. recall that you initialize the centroids randomly and keep updating them until they settle at a good point. my suggestion is to use the `random_state` parameter with a single value to guarantee that you get the same centers each time you use your model. ```python from sklearn.cluster import KMeans kmeans = KMeans(n_clusters=3, random_state=42) ... ``` ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [October 29, 2022, 7:28am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/3 "2022-10-29T07:28:51Z")
@Tanvi, is there any mechanism for us to drive away from a local optima? ---
**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\ **Post date:** [October 29, 2022, 9:46am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/4 "2022-10-29T09:46:02Z")
Hi there, please bear in mind that a training process is in general not deterministic but rather stochastic, e.g. due to: - sampling from your training data (batches), resp. stochastic gradient descent - dropout (where randomly Neurons are „shut off“) for regularization - initialisation (if you sample from a distribution) - etc. So in general in your training process there is some randomness. Dependent on your actual problem and the mechanics behind your loss function, you might encounter several local minima. Here your solver can get „stuck“ due to the above described randomness [1]. If you do it several times you might end up in different optima. Note: you can prevent this with choosing a pseudo-random seed to ensure repeatability [2]. (At least you will get the identical result from your optimization routine. But it’s not clear whether or not this is the global optimum. Frankly speaking, in reality this might not even be required as long as your model is sufficient in fulfilling your business needs and is robust enough [does not overfit].) Useful links: - [1] [Local vs. Global Optima - MATLAB & Simulink](https://www.mathworks.com/help/optim/ug/local-vs-global-optima.html) - [2] [random — Generate pseudo-random numbers — Python 3.11.0 documentation](https://docs.python.org/3/library/random.html) Best Christian ---
**Author:** ![Christian\_Simonis](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/christian_simonis/32/142583_2.png) [@Christian\_Simonis](https://community.deeplearning.ai/u/Christian_Simonis)\ **Post date:** [October 29, 2022, 9:58am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/5 "2022-10-29T09:58:49Z")
In addition, solvers like ADAM [Adaptive Moment Estimation] utilize „Momentum“ of the optimization trajectory to overcome local minima in order to get to the **global optimum** eventually. ![](https://global.discourse-cdn.com/dlai/original/3X/9/6/965bf974f378c9eee2ddc326a26c3ca6bb469567.gif) Best Christian ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [October 29, 2022, 10:20am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/6 "2022-10-29T10:20:57Z")
Hi, Raymond Recently I started taking a course titled **Convex Optimization for Machine Learning** where the convex here describes functions that only has a single minimum which implies that it has a **Global Minima**. in the lecture I learned that not all function is convex in nature but there is a way that you can convert a function which is not convex in nature from on set to another and in this new set this function can be considered a convex function. one of the methods is called **Convex Hull** which is defined as the smallest convex set that contains this function in it. if we could find such a **Hull** we then can convert our function to be convex and be able to find its global minimum easily. ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [October 29, 2022, 1:25pm UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/7 "2022-10-29T13:25:11Z")
Hello @Moaz_Elesawey, Convex relaxation? Is the course able to give a simple / practical example that talks about , before and after applying the technique, what difference does it make? Can you share the example? Raymond ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [October 29, 2022, 1:42pm UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/8 "2022-10-29T13:42:01Z")
Hi Raymond For now, we only just scratch the surface of **Convex Optimization** all of what we take is just theoretical and very mathematical (which I am not very good at 😅). If you want the materials and Note I have them in this [GitHub Repo](https://github.com/moaz-elesawey/convex-optimization) I really don’t have any practical example in mind now, but the ultimate goal of this technique is will enable us to find the minima easier because as I said **Convex means that it only has one minimum which means it’s the Global**. ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [October 30, 2022, 1:07am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/9 "2022-10-30T01:07:57Z")
Hello Moaz, Thank you for sharing the repo with me. I read the notes and seems the best is yet to come. So far those chapters seem only for setting the stage. Lagrange multiplier is a very interesting stuff 😉 and I had learnt it when I studied Lagrangian Mechanics. Raymond ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [October 30, 2022, 6:18am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/10 "2022-10-30T06:18:37Z")
Hi, Raymond You’re welcome,. You’re totally right we have not yet go very deep into the subject we barely touched it in the last lecture. I hope it will be easy to study as mathematical foundation is not that great. Best regards, Moaz ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [October 30, 2022, 7:02am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/11 "2022-10-30T07:02:06Z")
Moaz, let me know if there is any maths stuff in those chapters that you want to discuss. 😉 Raymond ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [October 30, 2022, 7:22am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/12 "2022-10-30T07:22:01Z")
Thank you very much Raymond, I really appreciate it. I will let you know if I need any help. Best regards, Moaz ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 11, 2022, 12:40pm UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/13 "2022-11-11T12:40:32Z")
Hello @Moaz_Elesawey, [I come across this video explaining Lagrange multiplier visually.](https://youtu.be/5A39Ht9Wcu0) Raymond ---
**Author:** ![Moaz\_Elesawey](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/moaz_elesawey/32/30848_2.png) [@Moaz\_Elesawey](https://community.deeplearning.ai/u/Moaz_Elesawey)\ **Post date:** [November 11, 2022, 2:19pm UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/14 "2022-11-11T14:19:31Z")
Hi, @rmwkwok I am very thankful to you and I really appreciate your help. I will certainly watch it. Thank you again. Sincerely, Moaz ---
**Author:** ![rmwkwok](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/rmwkwok/32/126977_2.png) [@rmwkwok](https://community.deeplearning.ai/u/rmwkwok)\ **Post date:** [November 28, 2022, 9:18am UTC](https://community.deeplearning.ai/t/how-different-initialization-of-centroids-of-k-means-results-in-drastic-different-clusters-they-all-share-common-cost-function/223900/15 "2022-11-28T09:18:49Z")
Hello @Moaz_Elesawey, Me again. I am recently reading about lagrange multiplier again and suddenly realized that regularization terms can just be an example of using lagrange multiplier.[I found this discussion on google](https://stats.stackexchange.com/questions/134763/what-is-the-connection-between-regularization-and-the-method-of-lagrange-multipl). I think it’s nice to connect together 2 topics that we have learnt. Cheers, Raymond