Skip to main content

A Baseline for 3D Multi-Object Tracking

-By Xinshuo Weng, Kris Kitani Carnegie Mellon University


Abstract
 3D multi-object tracking (MOT) is an essential component technology for many real-time applications such as autonomous driving or assistive robotics. However, recent works for 3D MOT tend to focus more on developing accurate systems giving less regard to computational cost and system complexity. In contrast, this work proposes a simple yet accurate real-time baseline 3D MOT system. We use an off-the-shelf 3D object detector to obtain oriented 3D bounding boxes from the LiDAR point cloud. Then, a combination of 3D Kalman filter and Hungarian algorithm is used for state estimation and data association. Although our baseline system is a straightforward combination of standard methods, we obtain the state-of-the-art results. To evaluate our baseline system, we propose a new 3D MOT extension to the official KITTI 2D MOT evaluation along with two new metrics. Our proposed baseline method for 3D MOT establishes new state-of-theart performance on 3D MOT for KITTI, improving the 3D MOTA from 72.23 of prior art to 76.47. Surprisingly, by projecting our 3D tracking results to the 2D image plane and compare against published 2D MOT methods, our system places 2nd on the official KITTI leaderboard. Also, our proposed 3D MOT method runs at a rate of 214.7 FPS, 65× faster than the state-of-the-art 2D MOT system.



Multi-object tracking (MOT) is an essential component technology for many vision applications such as autonomous driving , robot collision prediction and video face alignment. Due to the significant advance in object detection, there has been much progress on MOT. For example, for the car class on the KITTI MOT benchmark, the MOTA (multi-object tracking accuracy) has improved from 57.03 to 84.24 in two years. Although the accuracy has been significantly improved, it has come at the cost of increasing system complexity and computational cost. Complex systems make modular analysis challenging and it is not always clear which part of the system contributes the most to performance. For example, leading works have substantial different system pipelines but only minor differences in performance. Also, the adverse effect of increased computational cost is obvious. Despite having excellent accuracy, real-time tracking is out of reach. In contrast to prior work which tends to focus more on accuracy over system complexity and computational cost, this work aims to develop an accurate, simple and real-time 3D MOT system. We show that our proposed system which combines the minimal components for 3D MOT works extremely well. On the KITTI dataset, our system establishes new state-of-the-art performance on 3D MOT. Surprisingly, if we project our 3D tracking results to the 2D image plane and compare against all published 2D MOT methods, our system places 2nd on the official KITTI leaderboard . In addition, due to the simplicity of our system, it can run at a rate of 214.7 FPS on KITTI test set, 65 times faster than the state-of-the-art MOT system BeyondPixels. When comparing against other real-time MOT systems such as Complexer-YOLO, LP-SSVM, 3D-CNN/PMBM, and MCMOT-CPD, our system is at least twice as fast and achieves much higher accuracy.

Related Work



2D Multi-Object Tracking. 

Recent 2D MOT systems can be mostly split into two categories based on the data association: batch and online methods. The batch methods attempt to find the global optimal solution from the entire sequence. They often create a network flow graph and can be solved by the min-cost flow algorithms. On the other hand, the online methods consider only the detection at the current frame and are usually efficient for real-time application. These methods often formulate the data association as a bipartite graph matching problem and solve it using the Hungarian algorithm. Beyond using the Hungarian algorithm in a post-processing step, modern online methods design the deep association networks that are able to construct the association using neural networks. Our MOT system also belongs to online methods. For simplicity and real-time efficiency, we adopt the original Hungarian algorithm without using neural networks. Independent of the data association, designing a proper cost function for affinity measure is also crucial to the MOT system. Early works employ hand-crafted features such as spatial distance and colour histograms as the cost function. Instead, modern methods apply the motion model and learn the appearance features. In contrast to prior works which combine both appearance and motion models in a complicated way, we choose to employ only the simplest motion model, i.e., constant velocity, without using an appearance model.

3D Multi-Object Tracking. 

Most 3D MOT systems share the same components with the 2D MOT systems. The only distinction lies in that the detection boxes are in 3D space instead of the image plane. Therefore, it has the potential to design the motion and appearance models in 3D space without perspective distortion. proposes an image-based method which estimates the location of objects in image space and also their distance to the camera in 3D. Then a Poisson multi-Bernoulli mixture filter is used to estimate the 3D velocity of the objects. [30] applies an unscented Kalman filter in the bird’s eye view to estimate not only the 3D velocity but also the angular velocity. proposes a 2D-3D Kalman filter to jointly utilize the observation from the image and 3D world. Instead of using the hand-crafted filters, Design Siamese networks to learn the filters from data. Unlike previous works which use complicated filters, our proposed system employs only the original Kalman filter for simplicity, but extends its state space to full 3D domain, including not only 3D velocity but also the 3D size, 3D location and heading angle of the objects.

3D Object Detection. 

As an indispensable component of the 3D MOT, the quality of the detected 3D bounding box matters. Prior works mainly focus on processing the LiDAR point cloud inputs. Divide the point cloud into equally-spaced 3D voxels and apply 3D CNNs for 3D bounding box prediction. Converts the point cloud to a bird’s eye view representation for efficiency and exploit 2D convolutions. Other works such as directly process the point cloud inputs using PointNet++ for 3D detection. In addition, instead of using the point cloud, achieve the 3D detection from only a single image.


Please refer to paper for implementation and metrics.


Comments

Popular posts from this blog

DDSP: DIFFERENTIABLE DIGITAL SIGNAL PROCESSING

 -By Google Research {jesseengel,hanoih,gcj,adarob}@google.com Paper Link Code Demo Abstract Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enable...

Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network

By - Siyang Lu, Xiang Wei, Yandong Li, Liqiang Wang Department of Computer Science, University of Central Florida, Orlando, FL, USA School of Software Engineering, Beijing Jiaotong University, China Paper Link Previous week we did a scan of the survey of Anomaly detection   , that covers breath of the topic (Types of Anomalies, Types of Models and their applications). In this summary paper, we look in to the core Heavy lifting part of Detecting Anomaly in Big Data System Logs Using Convolutional Neural Network. Abstract Nowadays, big data systems are being widely adopted by many domains for offering effective data solutions, such as manufacturing, healthcare, education, and media. Big data systems produce tons of unstructured logs that contain buried valuable information. However, it is a daunting task to manually unearth the information and detect system anomalies. A few automatic methods have been developed, where the cutting-edge machine lear...

Capsule Neural Networks

-By Sara Sabour, Nicholas Frosst,  Geoffrey E. Hinton  Google Brain Toronto Paper Link Abstract   A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher...