Can i do this idea?

I have a idea but i dont know if i can execution it
And how get data for it …
The idea is … built model can predict and choise best hyperparameters to any model
For examble if i built new model by n samples and m features and…
I input n and m and other data to my model (idea ) and then the model (idea) say u should set alpha =6 to ur new model and 1280 hidden layers and so on …
How can i do it and how can i get data to do it

Hey @Mohamed_khaled_badr,
Let’s try to break this down into whether it’s possible to execute it or not, and if yes, then how.

Disclaimer - I believe this is an open research area, so whatever mentioned below is my opinion, which may or may not be correct. I strongly advise you to go through the existing research work focused on this.

Let me first present what I understood from your idea. You want to develop a system say f, which takes in a model and a dataset and outputs the best hyper-parameter configuration for the model.

In order to train this f in a supervised setting, you need to have meta-data about the dataset and about the model, which will form your X, and the best hyper-parameter configuration which will form your Y. Now, here comes the interesting part. There can be infinite datasets and each person can use a dataset in a large number of ways (for instance, different train/val/test splits); similarly, there is a large number of models. So, you see that there are infinite X-Y pairs for training f in a supervised setting, and since, you can’t account for all such pairs practically, hence, your f won’t be able to account for all the possible scenarios.

Another interesting aspect of this is the fact that for a particular X, there can be a large number of possible Y(s), since some users of your system might need to keep some hyper-parameters fixed, due to various reasons such as computational requirements, memory requirements, etc

But this is not all! Consider just a single X-Y pair. For a particular X (i.e., particular model and dataset configuration), how can you figure out Y? If you say that I will use the test-set accuracy to decide the best hyper-parameters configuration, what if some user wants to use the test-set F1-score to decide the best hyper-parameters configuration? So, you would even have to account for all the methods (basically the metrics) on the basic of which you can select the best hyper-parameters configuration.

I believe, at this point, we can rule out the supervised framework to train f, or at least, state that it would be practically very difficult. Now, comes the unsupervised framework, i.e., we eliminate the labels completely. Using approaches from unsupervised learning, we can find patterns between X, but how to translate these patterns into hyper-parameter configurations is another task of great feat. You may look if there is any existing research ongoing in this area.

Now, comes the most promising frameworks, at least as per my opinion. First is Reinforcement Learning, in which it is practical to train models which can deal with infinite input space. Second is Semi-Supervised Learning, in which the algorithm is trained upon a combination of labelled and unlabelled data. Third is Weakly Supervised Learning, in which the algorithm learns from limited and noisy data.

Each of these approaches, I believe could be possibly used, or at least tried. Whether they are practical or not, once again, you may explore the research which are using these approaches to solve the stated problem.

I hope this helps.

Cheers,
Elemento

1 Like