AI/ML industry starting with Google’s TensorFlow has introduced numerous open source libraries and frameworks

Rajesh Uppal June 20, 2020 AI & IT, Industry Comments Off on AI/ML industry starting with Google’s TensorFlow has introduced numerous open source libraries and frameworks 274 Views

In mathematics, a tensor is an algebraic object that describes a linear mapping from one set of algebraic objects to another. According to Ars Technica, “Basically, a tensor is a matrix of equations, instead of a matrix of pure numbers. Tensor mathematics is the manipulation of these equation matrices as a method of solving ALL of the involved equations.”

Industry starting with google have introduced open source libraries and frameworks that support development of machine learning applications. Deep learning relies on neural networks—systems that approximate the web of neurons in the human brain.

A framework is a toolbox for creating, training, and validating deep-learning neural networks. Using a high-level programming API, it hides the complexities of the underlying algorithms to greatly simplify and speed up development. Like deep learning, frameworks are evolving rapidly.

Currently, the most popular framework is TensorFlow, an open-source software library created and supported by Google. TensorFlow is a way of building and running these neural networks—both at the training stage and the execution stage. It’s a set of software libraries—a bunch of code—that you can slip into any application so that it too can learn tasks like image recognition, speech recognition, and language translation. The name TensorFlow derives from the operations that such neural networks perform on multidimensional data arrays, which are referred to as tensors.

Google’s TensorFlow

Based on the Python programming language, TensorFlow supports C++ and R. TensorFlow supports the visualization of computation graphs. For example, a matrix multiplication would be represented by a node, while the two incoming edges would correspond to the incoming edge, and the result would be the outgoing edge. Because TensorFlow is a low-level library, creating models can be challenging and complex.

TensorFlow is a free and open-source software library for dataflow and differentiable programming across a range of tasks. TensorFlow was developed by the Google Brain team for internal Google use. It is a symbolic math library, and is also used for machine learning applications such as neural networks. Google also uses this AI engine to recognize spoken words, translate from one language to another, improve Internet search results, and more. It was released under the Apache License 2.0 on November 9, 2015.

Its flexible architecture allows for the easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices without rewriting code.

TensorFlow computations are expressed as stateful dataflow graphs. TF operates with a static computation graph. That is, we first define the graph, then we run the calculations and, if we need to make changes to the architecture, we re-train the model. It’s handy for creating and experimenting with deep learning architectures, and its formulation is convenient for data integration such as inputting graphs, SQL tables, and images together.

Pytorch

Unlike TensorFlow, the PyTorch library operates with a dynamically updated graph. This means that it allows you to make changes to the architecture in the process. PyTorch is based on Torch, a Lua-based open-source machine library. Developed by Facebook’s artificial-intelligence research group, PyTorch features complex tensor computation and strong GPU acceleration support. In PyTorch, you can use standard debuggers, for example, pdb or PyCharm.

With basic Python knowledge, users can build deep-learning models without a steep learning curve. Its PyTorch architecture simplifies the deep-modeling process and offers more transparency than Torch. It also supports both data parallelism and distributed learning. With its many pretrained models, PyTorch makes a good choice for prototyping and small projects.

Sonnet

Sonnet deep learning framework built on top of TensorFlow. It is designed to create neural networks with a complex architecture by the world famous company DeepMind. High-level object-oriented libraries that bring about abstraction when developing neural networks (NN) or other machine learning (ML) algorithms.

The main advantage of Sonnet, is you can use it to reproduce the research demonstrated in DeepMind’s papers with greater ease than Keras, since DeepMind will be using Sonnet themselves. So all-in-all, it’s a flexible functional abstractions tool that is absolutely a worthy opponent for TF and PyTorch.

Keras

Keras, also written in Python, is a simplified interface that enables efficient neural nets to be built with minimum code. Keras is on the higher Level. At this Level, the lower level primitives are used to implement Neural Network abstraction like Layers and models. Since it runs on top of TensorFlow, MXNet, and other frameworks, Keras is surging in popularity.

This user-friendly framework minimizes the numbers of APIs, and it is modular for easier model construction. Keras results in a much more readable and succinct code. Keras model Serialization/Deserialization APIs, callbacks, and data streaming using Python generators are very mature.

Less configurable than lower level frameworks, Keras works well for beginners who want to learn how to use various machine-learning models and quickly understand how they work.

MxNet

MXNet is designed for high efficiency, productivity, and flexibility. Supporting multiple popular programming languages – including Python, R, C++, and Julia – MXNet lets users train a deep-learning model without having to learn a new language. Like PyTorch, its back end is written in C++ and CUDA. Supporting recurrent neural networks (RNN), convolution neural networks (CNN), and long-short term memory (LTSM) networks, MXNet is touted for its imaging, handwriting/speech recognition, and forecasting capabilities. It scales well across multiple CPUs and GPUs, making it useful for enterprise solutions. Gluon, another simplified front end for MXNT like Keras, also supports a model zoo of predefined and pretrained models.

The original Caffe framework is best known for solving image-processing tasks, especially visual recognition, but does not perform well for non-vision network design, reduced math precision, or distributed computation. It supports MATLAB as well as C, C++, Python, and a model zoo. To address these shortcomings, Facebook created Caffe2 to support its applications; currently, Caffe2 is being merged into PyTorch. NVIDIA maintains a separate fork of Caffe (“NVIDIA Caffe” or “NVCaffe”) tuned for multiple-GPU configurations and mixed precision support. It features layer-wise adaptive rate control (LARC) with adaptive global gradient scaler for improved accuracy, especially for 16-bit floating-point training.

NVIDIA TensorRT is a high-performance inference engineering tool designed to deliver maximum throughput, low latency, and power efficiency in the deployed network. It provides APIs in C++ and Python. Trained models are optimized by first restructuring to remove layers with no output, and then fusing and aggregating the remaining layers. The model is then optimized and calibrated to use lower precision (such as INT8 or FP16). For example, a TensorFlow CNN on an NVIDIA V100 can process 305 images/second. When the CNN is optimized with TensorRT, the output is 5700 images/second. (Framework training comparison from NVIDIA is available at https://developer.nvidia.com/deep-learning-performance-training-inference)

All of these frameworks are open source, are available on GitHub, and can be deployed using NVIDIA’s TensorRT. The Caffe, TensorFlow, Pytorch, and MXNET frameworks are supported by Bright Cluster Manager in Curtiss-Wright’s OpenHPEC Accelerator Suite of development tools. Choosing the right framework depends on the type of network being developed, the programming language and tools, and the user’s skill set.

Chainer

Chainer has been the leading neural network framework for dynamic computation graphs or nets that allowed for input of varying length, a popular feature for NLP tasks. The code is written in pure Python on top of the Numpy and CuPy libraries. Chainer is the first framework to use a dynamic architecture model (as in PyTorch).

Better GPU & GPU data center performance than TensorFlow. (TensorFlow is optimized for TPU architecture) Recently, Chainer became the world champion for GPU data center performance. By its own benchmarks, Chainer is notably faster than other Python-oriented frameworks, with TensorFlow the slowest of a test group that includes MxNet and CNTK.

ONNX

The ONNX project was born from the collaboration of Microsoft and Facebook as a search for an open format for the presentation of deep learning models. ONNX simplifies the process of transferring models between different means of working with artificial intelligence. Thus, ONNX allows you to consider the benefits of various Deep Learning frameworks.

models are currently supported in Caffe2, Microsoft Cognitive Toolkit, MXNet, and PyTorch, and there are connectors for many other common frameworks and libraries.

References and Resources also include:

https://towardsdatascience.com/top-10-best-deep-learning-frameworks-in-2019-5ccb90ea6de