Hi everyone. Generally, if someone wants to add constraints to a neural network so that the outputs satisfy the constraints, how would it be possible? Is the only way to include the constraints in the loss function?
Which course are you attending?
You posted in the “General Discussions” forum, which isn’t very closely monitored.
2 Likes
It is not a course specific question, just one thing that I am dealing with in the problem I am trying to solve.
Thanks. Not sure who will pick this up to give an answer.
In my experience, there’s no need to constrain the outputs for a simple NN, if you have a suitable training set.
Exactly, my problem is with my training set, so I thought maybe adding constraints would help.
I am not sure if it is relevant to you or not.
I read a research paper a few months ago in which authors stated “small adversarial perturbations are added to the original input such that model produces incorrect outputs. By using this method, the fully connected neural network is trained to solve physical problems in various dimensions, which leads to better robustness, and generalization of the neural network on the test data”
Paper title: PIAT: Physics Informed Adversarial Training for Solving PDEs.
Best,
Saif.
1 Like
What exactly is the problem with your training set?
1 Like
Thanks @saifkhanengr. Yeah, I was thinking about adding perturbation to the training set. I will check the paper.
Actually, the problem I am working on is the inverse of a common problem in transportation. The original problem has a unique solution, but the inverse problem does not. I developed the training set using the original problem. I trained the neural network with the outputs from the original data as the input of the neural network and the inputs from the original problem as the outputs of the neural network. For this reason, I don’t expect the model to be perfect, but there are some equations that govern the input and outputs. Therefore, I wanted to include them in the model so it can improve the performance. I already tried to improve them by incorporating the equations o the loss function, but it was not successful.
And generally, how can we deal with problems that do have not unique solutions using neural networks?
For example, if we have “n” number of linear equations and “m” number of variables, but the number of variables is more than the number of equations, how can we model that with a neural network?
That has nothing to do with neural networks. You can’t get a unique solution to that that system using any method.
The first advice is “get more examples”.
Second, you might try a method like PCA to remove some of the features that don’t contribute very much variance.
If you have equations which govern the mapping from inputs to outputs, then perhaps Neural Networks are not the appropriate solution. Have you investigated a purely mathematical approach? It’s like trying to predict at what speed and angle of attack a given airfoil will stall. If you try to solve that with a Neural Net then it only can solve cases that it’s actually seen. What happens if you present it with a shape for which there is no training data? And if you have the training data for all possible shapes, then it means you already know the answer. In that type of case, it’s a physics problem, so you need to solve it with physics methods, e.g. numerical aerodynamic simulation, which is a well developed field.
Thanks @paulinpaloalto and @TMosh. Probably I should rethink how to solve the problem with other approaches or how to simplify the problem so we can have unique solutions.
Best,
Nasim
The high level point is that Neural Networks are not the solution for all types of problems. They are very good at pattern recognition problems (e.g. image recognition or language translation) or pattern prediction problems (e.g. ChatGPT), but they are not necessarily good at solving equations. The field of numerical analysis has a deep history and lots of methods for approaching the task of computing approximate solutions to equations of various types, including differential equations.
1 Like
Yes, you are right. However, I am working on a transportation problem, which can be represented as a graph. So, I think some neural networks such as Graph Neural Networks might help me to represent the input features in a different way.
I believe if you properly label the data, then the neural network will learn to pick the right solution, not the other ones. So there should be no need for constraints.
1 Like
Actually, the problem I am working on is a multi-output regression problem and not a classification problem or a single-output problem. For example, consider a graph with 5 nodes and 8 links. The input is the vector of weights associated with the links, and the output is the vector of weights associated with each node pair. Therefore, the input for each training or testing sample would be an 8*1 vector and the output would be a 25*1 vector.
It might help to know more about what you are actually solving for here, but I still have this nagging feeling that you’re barking up the wrong tree. If the solution is governed by a system of equations, it just feels like Deep Neural Nets are not the place to look. There are other ways to approach finite combinatorial optimization problems than networks. I took the Discrete Optimization course from University of Melbourne quite a while ago. It’s long enough ago that I don’t really remember that much about it, but it might be worth a look because it may trigger some ideas about other possible approaches.
Great, thank you for your recommendation @paulinpaloalto.
Best,
Nasim