I’m Lee Ling, a deep learning enthusiast who loves breaking down the fundamentals and tweaking them for the better. Recently, I had an idea for a modified neural network—Uni-Neuron Network (Uni-NN)—which adds skip layer connections. I turned it into reality, tested its performance, and so far, it’s showing some potential in preventing gradient descent issues:
Learning Rate: 0.005 Comparison Criteria: Model performance with similar parameter sizes
That said, it’s slow and struggles when the layers get long—so there’s still work to do.
I believe AI is shaping the future, and I’m all about finding better methods, not chasing perfection. If you’re down to collaborate or just want to check it out, it’d be an honor! Also, if you spot any flaws, I’d love to hear your thoughts—learning never stops.
It’s hard to say whether this would be a better architecture than one without skip connections, and for what tasks and/or what kinds of gradient descent, or why.
I found this survey paper on networks with skip connections in computer vision:
And here is one which examines effects on gradients:
The above contain a lot of concepts I’m not readily familiar with,
I have found that tweaking a net just hoovers up weeks of work and one still hasn’t gotten anywhere satisfying (except better at Python). For now I will try to finish the course and checking what ideas people in the decades before
Blue – Normal Neural Network
Green – Advanced Neural Network (Dropout, BN)
Orange – Uni-Neuron Network
Purple – Advanced Uni-Neuron Network (Dropout, BN)
Test Data:
GitHub Repo: Uni-Neuron Network
Personal Email: leeinfinitee00@gmail.com
