-By Riya R. Ganiga, Shaguftha Zuveria Kottur, Tallapalli Surabhi, A. Parkavi, Sini Anna Alex
Ramaiah Institute of Technology, Bangalore, India
Abstract
Compilers are used to translate between languages or
representations. In neural networks, the input is usually in the
form of a compute graph with tensor computations associated with
nodes which needs to be translated into executable. The compiler
plays an important role in this conversion, performing
optimizations and lowering. It can be used as a bridge to target
multiple hardware architectures from multiple front ends and
hence is a major component for scalability of neural network
frameworks. Also, the optimizations and conversions done by a
compiler lead to reduction in time taken to train a particular
network and its implementation.
courtesy: https://www.cs.utexas.edu/~roshan
Code readability, ease of construction, time and space
complexity and size of the code are some of the important
features of a program code. The simpler a code is for humans
to read, the more difficult it is for the computer to know what it
is supposed to do. Compilers translate a high-level
programming code to low-level machine understandable
language.
This translation is time consuming and hence needs to be
optimized. Code optimization is one technique in which a code
is enhanced to decrease the time and space consumption,
resulting in efficiency and lower resource utilization. It is an
important phase in engineering design and improves the quality
of the designed product elevating the value of the product.
Optimization of code can be done in the coder or by the
compiler (compiler optimization). Optimization by compiler
might involve several methods and techniques like machine independent optimization, machine-dependent optimization,
basic-block, loop optimization, dead-code elimination and
partial redundancy.
Neural networks are computing systems made up of layers of
simple, interconnected processing elements having the
capability to process information dynamical through response
to external inputs. The layers of a neural network can be broadly
classified into input layer, output layer and central hidden layer
which in turn can have many layers. All the computational
complexity comes from the advancements in the hidden layer which is the heart of the network. The hidden layers perform
computations on the weighted inputs and produce net input
which is then applied with activation functions to produce the
actual output. Compiler optimization techniques are applied to
enhance the working of the hidden layer to make the network
time and resource efficient. The faster a machine is, the better,
hence one would want to speeden the process a neural network
would process data and optimization is exactly what is to be
done. For example, development of new layers and architecture
is time consuming and optimizing these developments would
result in a huge cut down of processing time. Neural networks
also termed as artificial neural networks (ANN) are of many
types - convolution neural networks(CNN), feed-forward
neural networks(FNN), recurrent neural networks(RNN),
multi-layered perceptron, etc.
Deep learning is a machine learning technique that teaches
the machine what to do and the machine is trained to devise a
method on how to do it. Deep learning is a Neural Network
consisting of a hierarchy of layers, whereby each layer
transforms the input data into more abstract representations.
Deep learning software demands reliability and performance.
Deep neural networks are a class of models that use a system of
interconnected neurons to estimate or approximate functions.
The performance and integration with existing systems of a
deep learning network can be boosted up by the application of
compiler designer techniques on it.
A key difference between compilers for languages like C++
and compilers for deep learning frameworks is that with deep
learning, the data being operated on is large and variable-sized,
but highly amenable to parallelization.
Deep tune is an end to end machine learning approach for
optimising the heuristics with deep neural networks.
It has
following phases,
Source Rewriter: Source normalizing transformations are
applied on the raw code to generate a consistent code structure.
Sequence Encoder: Source code is encoded with integers
which is an index of predetermined hybrid vocabulary
(character and token)
Embedding: From sequence encoding the relation between
tokens cannot be interpreted so we translate the semantically
related tokens into a lower dimensional vector space.
Sequence Characterization: This phase is equivalent to
manual feature extraction. Long Short Term Memory (LSTM)
is a two-layer network, inputs the embedded vectors and
outputs a single output vector characterizing the given
sequence.
Heuristic model: The former layer has 32 neurons and the
latter one has a single neuron for each heuristic decision.
Finally, whichever neuron has the more activation is considered
to the optimal heuristic.
Conclusion
Deep learning implementations require a high level of
abstraction to separate developers from low level design. This
abstraction is provided by highly efficient compilers which
either work with specific DSLs or can be customized for any
DSL. Since the structure of neural networks can be represented
by graphs, compilers need to translate these graphs representing
tensor computations into executable which is usually done by
generating multiple IR, where on each IR optimizations and
lowering can be done. This paper explored various ways in
which compilers are used in the deep learning community from
implementing specific DSLs to use in frameworks to
simplifying the implementation by mapping onto FPGA. For
the former it resolves scalability issues by acting as an
intermediate between frontends and backends and for the latter,
it aids in faster training and implementation of neural networks
to make use of the efficiency and reprogrammability of field
programmable gate arrays. Certain compiler optimizations and
neural network frameworks were also reviewed to understand
the current approaches used.
Comments