Satellite Image Analysis using Deep learning methods enables Automatic Target Recognition ( ATR) of military targets

Rajesh Uppal February 4, 2022 AI & IT, Space Comments Off on Satellite Image Analysis using Deep learning methods enables Automatic Target Recognition ( ATR) of military targets 1,774 Views

Despite substantial advances in global human well-being, the world continues to experience humanitarian crises and natural disasters. Long-term conflicts affect people in many parts of the world, but often, accurate maps of the affected regions either do not exist or are outdated by disaster or conflict. Satellite imagery is readily available to humanitarian organizations, but translating images into maps is an intensive effort. Today, maps are produced by specialized organizations or in volunteer events such as mapathons, where imagery is annotated with roads, buildings, farms, rivers etc.

Satellite imaging is gaining importance in many applications like remote surveillance, environmental monitoring, aerial survey etc. All these applications involve searching objects, event of interest, facilities etc., from the satellite images. In most applications, manual detection and classification of objects becomes very difficult especially with large volumes of data and the number of satellite images to process collectively.

A Satellite Image is an image of the whole or part of the earth taken using artiﬁcial satellites. It can either be visible light images, water vapor images or infrared images The different types of satellites produce (high spatial, spectral, and temporal) resolution images that cover the whole Earth in less than a day. The large-scale nature of these data sets introduces new challenges in image analysis.

The earth observation industry has now exploded with raw data as satellite constellations like MODIS, Landsat, and Sentinel have democratized access to timely satellite imagery of the entire globe. Meanwhile, cloud providers like AWS and Google Cloud have gone so far as to store satellite data for free, further accelerating global usage of these images. The challenge is interpreting the content of satellite imagery from this huge data.

Remote sensing data are geo-located, i.e., they are naturally located in the geographical space. Each pixel corresponds to a spatial coordinate, which facilitates the fusion of pixel information with other sources of data, such as GIS layers, geotagged images from social media, or simply other sensors. On one hand, this fact allows tackling of data fusion with non-traditional data modalities while, on the other hand, it opens the field to new applications, such as pictures localization, location-based services or reality augmentation.

The Copernicus program guarantees continuous data acquisition for decades. For instance, Sentinel-1 images the entire Earth every six days. This capability is triggering a shift from individual image analysis to time series processing. Novel network architectures must be developed for optimally exploiting the temporal information jointly with the spatial and spectral information of these data.

Image analysis challenge

The growing fleet of imaging satellites beam data back to Earth 24/7 — some 80 terabytes every day, according to the research, a number certain to grow in coming years. Merely storing so many terabytes of data requires a huge investment. Distilling the layers of data embedded in the images requires additional computing power and advanced human expertise to tease out strands of information that are coherent and useful to other researchers, policymakers or funding agencies.

The analysis and classiﬁcation of remote-sensing images is very important in many practical applications, such as natural hazards and geospatial object detection, precision. Remote sensing data are often multi-modal, e.g. from optical (multi- and hyperspectral) and synthetic aperture radar (SAR) sensors, where both the imaging geometries and the content are completely different.

Remote sensing also faces the big data challenge. In the Copernicus era, we are dealing with very large and ever-growing data volumes, and often on a global scale. For example, even if they were launched in 2014, Sentinel satellites have already acquired about 25 Peta Bytes of data. The Copernicus concept calls for global applications, i.e., algorithms must be fast enough and sufficiently transferrable to be applied for the whole Earth surface. On the other hand, these data are well annotated and contain plenty of metadata. Hence, in some cases, large training data sets might be generated (semi-) automatically.

Defense and Security requirements

Earth observation satellites have become an invaluable resource for defence and security missions, especially when planning operations and mission deployments remotely. The rise of asymmetric warfare and fragmentation of traditional modes of conventional war mean that the warfighter and the security industry have to be able to operate with a much greater degree of responsiveness. Whether it is for monitoring ICBM facilities in Russia, a satellite launch site in North Korea, a battle damage assessment over a Yemeni airbase or to locate a hijacked oil tanker off the coast of Somalia, today’s imagery satellites offer an exceptional level of tactical support to decision-makers and the modern soldier.

In military parlance, this is ‘sensor to shooter’. This is derived from the need for ‘dominant battlespace awareness’ to enable real-time decision-making and forms part of what the military call ‘network-centric warfare.’ Commanders need responsive intelligence that decisively supports the warfighter and, as a result, the warfighter is completely reliant on up-to-the-minute data on a 24/7 basis, potentially every day of the year.

One of the prerequisites for such flexibility while on operational deployment is now instant satellite tasking that will provide as close to real-time accuracy as possible. Furthermore, a number of critical factors such as reliability, usefulness, and speed of image acquisition and delivery, come into play. At a military tactical level, terminals are now a common piece of equipment, which allows the rapid viewing of high-resolution images.

Secondly there is a requirement of satellite imagery recognition systems that quickly identify and classify objects or targets in satellite images. Automatic detection of military targets such as oil tanks, aircraft, artillery, etc. in high-resolution satellite imagery has great significance in military applications. With the rapid development of satellite imaging and geographic information systems, a large number of high-resolution images can be acquired effortlessly from Google Earth. The non-hyperspectral image data has been used in many civil and military applications.

Satellite Image Analysis and AI Machine learning

AI is used in generally two types of applications for satellite imagery, One-level” applications and Multi-level” applications. The first applications of satellite data is making the various objects detectable to computer vision. The objects like buildings, road segments, and urban area boundaries is important for municipalities, government agencies, rescue teams, military, and other civil agencies. And such things can easily monitored from space satellites through images captured from high-resolution cameras.

The second example, AI applications is satellite images is change detection, in which numerous applications, such as crop, land use, urban infrastructure like road segments or environmental (deforestation, water reserves) and humanitarian crisis monitoring. The main motive of change detection algorithm is to create a map, in which changed areas are separated from unchanged ones. Two types of change detection can be further defined: A binary change detection and multi-class change detection, when each transition type can be explicitly identified. Apart from that there are multiple other applications of AI in satellite imagery like ship count and sizes, aircraft count in airports, urban areas, water, roads, etc. for Aerial view monitoring and data analysis.

Semantic segmentation deals with the task of assigning each pixel in a given image to one of potentially multiple classes. It deals with recognizing which objects are shown and where exactly these are presented in the image. As such, it is a challenging task that requires, on the one hand, to take into account the overall context of the image and for each pixel that of the surrounding area. On the other hand, it is required to label each pixel individually focusing on a very fine level of detail.

Towards this end, various automated techniques for detection and classification have been proposed and are in the works. From classical machine learning (ML) to current deep learning, many solutions have been proposed for object detection and classification in satellite images. These methods involve the extraction of various features from the images and classifying them using ML classifiers. Automated Object detection is still a challenge due to variations in the size of the object, orientation, and background of the target object.

Classification Algorithms

In optical remote-sensing image object detection, the common methods are basically divided into four categories. The first is based on boosting, which combines several weak classifiers to form a strong classifier for target detection. The second is based on the template method, which calculates the similarity of the image to be detected and the given template to achieve target detection.

The third is based on segmentation technology. It mainly identifies the background and target object by setting different thresholds for different objects, such as application of the multi-threshold and Otsu threshold in vehicle recognition.

The fourth is based on statistical theory. To achieve target detection, it employs machine learning, the support vector machine (SVM), and other methods, such as random forest (RF) and the backpropagation (BP) neural network. Firstly, this method is mainly to extract the feature of the target by RF, backpropagation artificial neural network (BPANN), et al. Then, using the feature, the classification model can be trained by SVM.

Yildiz et al. employed Gabor feature and used SVM classifier to detect different aircraft. Gabor filter is also employed by authors for road crack detection in aerial images and settlement zone detection is satellite images respectively. Hsieh et al. employed Zernike moments, aircraft contour and wavelets and used SVM classifier for the detection of aircraft in satellite images. Most of the methods discussed above use hand-crafted features and work effectively in their scenes only.

Machine learning

Machine Learning algorithms have proved to be a powerful tool for analyzing satellite imagery of any resolution and proving better and more nuanced insights. In its nascent stages, there are a few challenges as well in the application of Machine Learning on satellite images, including the extraordinarily large file size of satellite imagery and data format being exclusively designed for geo-referenced images that make Big Data and Machine Learning applications quite difficult.

Unsupervised learning algorithms such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), e.t.c are used for dimensionality reduction on satellite imagery. KMeans, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), e.t.c are used for clustering the data of satellite imagery.

Deep Learning

Remote sensing scientists have exploited the power of deep learning to tackle these different challenges and started a new wave of promising research.

Deep learning is a very effective method for learning optimum features directly from huge training datasets automatically. Now a day in numerous applications computer vision along with deep learning has outperformed humans. Furthermore, the use of Graphical Processing Units (GPUs) has decreased the training time of deep learning methods. Large databases of labeled data and pre-trained networks are now publicly available.

Still, even though training datasets for satellite imagery are freely available, the problem of actually wrangling that data or amending the architecture of common machine learning models to work with that data is still mostly in the research phase. The most popular deep learning architectures are not designed for imagery that is often a gigabyte or larger, may contain over a dozen channels (most of which are not in the visible spectrum), and is stored in spatially referenced file formats like GeoTIFF and JPEG2000.

The two popular models of deep learning are Deep Belief Network (DBN) and Convolutional Neural Network (CNN). Many deep learning models based on convolutional neural network (CNN) are proposed for the detection and classification of objects in satellite images. These models involve two steps. In the first step, the regions of the presence of object in the image are detected. In the second step, the objects are classified using a convolutional neural network. CNN is a modern deep learning method that is widely used for image recognition because it is invariant to small rotation and shifts.

Convolution is basically the action of sliding this 2×2 or similar kernel across the image from the top left to the other end while ensuring that the kernel is going through all different movements or positions where it does not overflow the boundary of the image, but covers the full image in its movement.

DBN is a probabilistic generative model which is pre-trained as Restricted Boltzmann Machine layer by layer, and then finally tuned by the back-propagation algorithm to become a classifier. Chen et al. employed object locating method along with DBN for aircraft detection in satellite images. Saliency has also been used for image classification by various researchers such as Li et al. applied visual symmetry detection and saliency computation for aircraft detection in satellite images. Zhang et al. and Sattar et al. employed saliency and used unsupervised learning for image classification. Identification of fixation points, detection of image regions representing the scene and detection of dominant objects in a scene are the primary goals of saliency.

Nevertheless, satellite images often contain several targets and correct localization of each target is required. Saliency cannot be directly employed for automatic target in satellite images and it needs the help of other methods such as symmetry detection.

CNN is being used in various computer vision applications for the last two decades. The sliding window approach for target detection using CNN is very slow and contrasts to mechanisms in the human visual system. Many objectiveness detection methods have been proposed to increase the computational efficiency of target detection, such as EdgeBoxes, Binarized Normed Gradients (BING), and Selective Search. Selective search greedily merges low-level super-pixels to generate object proposals.

Ross Girshick et al. proposed the use of selective search along with CNN instead of sliding-window detector and achieved outstanding results on ILSVRC2013 detection dataset. BING generates object proposals based on binarized normed gradients. EdgeBoxes uses the object boundaries in the image as a feature for proposing candidate objects. Moreover, EdgeBoxes are robust to varying size of objects.

In satellite images, scale and orientation changes are the main characteristics of targets. Moreover, the edge information of targets in satellite imagery contains very prominent and concise attributes. The major challenges in target detection in satellite imagery include presence of targets in different sizes, different orientations and at very close locations. BING generates very loosely fitting proposals and thus is only suitable at low IoU. Selective search is relatively good in general object detection, but it is considerably slower and doesn’t perform well if the size of objects is rather small. EdgeBoxes provides best tradeoff between speed and quality. An automatic target detection method based on EdgeBoxes and Convolutional Neural Networks (CNN) is proposed in this paper.

Industry solutions

The new release, named ‘Planet Analytics Feeds’, makes automatic analysis of satellite imagery now possible. With the new flagship Analytics Feeds, Planet Analytics is able to provide its users automated road, building, and vessel detections over their areas of interest. The users can leverage these feeds on top of the Planet imagery products and achieve global-scale insights, no custom builds are required.

Planet has a dedicated solution for Machine Learning called Planet Analytics, which uses Machine Learning algorithms for processing of daily satellite imagery, detecting and classifying objects, locating topographic and geographical features and consistently monitoring even the most infinitesimal change over time. The information feed is seamlessly integrated into the workflows and offers dazzling insights on almost any place on the globe.

Satellite imagery refining start-up Descartes Lab has a cloud-based platform that applies Machine Learning forecasting models to petabytes of satellite imagery that is drawn from a number of sources. Descartes Labs has an expertise applying Machine Learning to Earth Observation satellite imagery. Before machine learning can extract valuable data from imagery, the data has to be pre-processed to line up pixels and correct for varying atmospheric conditions and spectral calibrations.

DigitalGlobe GBDX team runs Machine Learning object detection on a very large scale. Every time a new model is applied to GBDX a comparison is made to ascertain the plus points over existing capabilities.

Google is also among the trailblazers tapping the potential of Machine Learning in satellite imagery. Google launched an application that can tell the exact geo-location of any photograph captured anywhere on earth. The project called PlaNet which deploys the power of machine learning is based on the combination of convolutional neural networks with mapping technology. The information that it provides is truly invaluable and unprecedented in both qualitative and quantitative aspects.

Azavea has developed an open-source python library for applying machine learning to satellite imagery called Raster Vision. Raster Vision allows users to do three messy things in an elegant way:

Transform satellite imagery into a format that plays nicely with most machine learning frameworks. You can “chip” a large image into hundreds or thousands of smaller images that can be used to train a model and then retrospectively stitched back together while maintaining all of the relevant geospatial information crucial to most mapping tasks.
Abstract the process of using common machine learning libraries, like PyTorch, so that you can easily train models, evaluate their results, and manage different experiments in parallel.
Package trained models so that you can easily deploy them in different settings (e.g. online vs. “at the edge”) and use them to predict on new data.

Lockheed Martin develops Deep learning products for automatic satellite image analysis

Lockheed Martin is marketing a new artificial intelligence product that helps analysts identify objects in satellite imagery. In a demonstration, it searched the entire state of Pennsylvania and in two hours located every fracking site in the state.

The company showed the system publicly for the first time at the GEOINT 2019 symposium that is heavily attended by intelligence analysts from the National Geospatial Intelligence Agency and the National Reconnaissance Office. The so-called global automated target recognition system could be used to find any type of objects in satellite imagery, saving analysts a lot of time and manual labor, said Mark Pritt, senior fellow at Lockheed Martin who helped develop the system.

Satellite imagery analysis is a growing and crowded industry where defense contractors compete with commercial players. Pritt said there are many companies that offer satellite imagery recognition systems that quickly identify and classify objects in areas across the world, but few provide global coverage. “With our tool, the user can draw a box anywhere in the world and hit the button,” he said. “The system will go search for objects of interest such as fracking wells, airplanes or refugee camps.” The objects of interest show up as icons on the map and the user can click on the icon to get a closer look.

“Today there’s still a lot of manual labor involved in identifying what you’re seeing in those images, they are time consuming to classify and label,” Pritt said.

Lockheed’s target recognition system uses satellite imagery from major commercial vendors like Maxar and Planet. With sharper 30cm resolution images, the system can distinguish between a cargo plane and a military transport jet, for example. It uses deep learning techniques common in the commercial sector to identify ships, airplanes, buildings and seaports.

The model, Global Automated Target Recognition (GATR), runs in the cloud, using Maxar Technologies’ Geospatial Big Data platform (GBDX) to access Maxar’s 100 petabyte satellite imagery library and millions of curated data labels across dozens of categories that expedite the training of deep learning algorithms. Fast GPUs enable GATR to scan a large area very quickly, while deep learning methods automate object recognition and reduce the need for extensive algorithm training.

The tool teaches itself what the identifying characteristics of an object area or target, for example, learning how to distinguish between a cargo plane and a military transport jet. The system then scales quickly to scan large areas, such as entire countries. GATR uses common deep learning techniques found in the commercial sector and can identify airplanes, ships,, buildings, seaports, etc.

“There’s more commercial satellite data than ever available today, and up until now, identifying objects has been a largely manual process,” says Maria Demaree, vice president and general manager of Lockheed Martin Space Mission Solutions. “Artificial intelligence models like GATR keep analysts in control while letting them focus on higher-level tasks. GATR has a high accuracy rate, well over 90% on the models we’ve tested so far. It only took two hours to search the entire state of Pennsylvania for fracking sites – that’s 120,000 square kilometers.”

“I’m not an expert on what oil production sites are, and I don’t have to be,” says Mark Pritt, senior fellow at Lockheed Martin and principle investigator for GATR. “This system teaches itself the defining characteristics of an object, saving valuable time training an algorithm and ultimately letting an image analyst focus more on their mission.” Pritt said the analysis of satellite imagery has been revolutionized by advances in artificial intelligence. “We’re trying to show the government what is possible and helping to shape their expectations,” he said.

In a presentation at GEOINT, Deputy undersecretary of defense for research and engineering Lisa Porter said DoD is still figuring out how to take advantage of artificial intelligence in military applications. Porter is a well known AI expert who served as the first director of IARPA.

“AI right now is a little bit of the shiny object,” she said. It’s important to understand what problems can be solved with AI, Porter said. The analysis of electro-optical satellite imagery has been one successful application for AI algorithms, but she called on DoD to help push the technology further to analyze other types of data such as radar imagery.