Thanks.
Are there any alternatives to linear computation of z from serially sequenced image pixels from top to bottom and left to right when computing z from a unitâs weight and bias parameters? Perhaps if one were to use a square path of discrete radius steps centred on a point in each image with highest correlation with another similar input training image?
Hi @anon88576143 , to answer your first question: yes, alternatives do exist, though the âserial top-to-bottomâ approach (raster scanning) remains the standard because it maps perfectly to how hardware (CPUs/GPUs) fetches data from memory. If you want to move away from the standard row-major z calculation, few frameworks like Quadtrees / Octrees and Vision Transformers do exist, you may want to look them up to get more details. While mathematically elegant, calculating z via a radial or square-step path centered on a dynamic point has a hurdle in the shape of modern hardware which is optimized for sequential memory access. Jumping around a âsquare pathâ requires non-contiguous memory fetches, which can be significantly slower unless using custom CUDA kernels.
If my algorithm is more efficient and accurate at delivering predictions then that would fuel efforts to develop new hardware architectures to implement them as is already the case with current architectures.
Algorithm design comes first, then hardware and software implementations - correct?
Hmm ya, it is like a typical âchicken and eggâ situation, but you are largely correct: in a pure research environment, the mathematical breakthrough or algorithmic shift usually precedes the specialized hardware.
Not chicken and egg.
The software and/or hardware architecture is the means by which the algorithm is implemented in reality.
We go from intellectual concept and validation in the form of an algorithm then implement that algorithm efficiently in hardware and software - correct?
Non-contiguity of memory access can be overcome with some preprocessing of the input image - correct?
Hypothetically everything is possible ! I was only stating the reality of the hardware of today !
I was talking about a novel algorithm and how it can be efficiently implemented in current hardware easily.
Is the convolutional layer a layer of units in a neural network model?
No, a convolutional layer is not a typical layer of units. Unlike the layers in a dense neural network which is a collection of units (neurons), a convolution layer is made of filters (also called kernels) that slides across the input images. This enables the model to detect features like an edge or a curve - regardless of its location inside the image. There are different kind of filters to detect different kind of features within an image. Unlike a layer of units where everything is connected to everything else, a convolutional layer only looks at a small neighborhood of pixels at a time. This makes CNNs more effective for images datasets compared to DNNs.
2 Likes