Jan 2021 - Mar 2021, Northwestern ME-499 Individual Project
I. Introduction
Recently, I have built a prototype of an advanced drivr assistance system (ADAS) using a Jetson Nano computer. In this project, I successfully deployed a deep neural network and some computer vision algorithms for lane detection on the Jetson Nano.
Background and Motivations
Although many big companies are racing harshly in the industry of autonomous driving, I personally do not see the possibility of having the road filled with autonomous vehicles in the near future.
And there are two reasons:
- People simply do not believe in autonomous vehicles. We are so used to driving the car on our own, and letting the computer do all the control does not make most people feel safe.
- The current autonomous vehicles are basically driving on their own, without any communication with other vehicles or utilities on the road (traffic lights, stree lights, etc).
In order to solve these two issues, I believe a popularization of strong and well-functional ADAS system is necessary. The ADAS system itself is not as aggressive as the idea of autonomous driving, since it is mearly giving out suggestions, and therefore should be much more acceptable by the society. After every car on the road is equipped with some kind of ADAS device, people would think: “I guess the car is indeed very smart, and maybe it can indeed do driving!” Besides that, when every car on the road is equipped with ADAS, they can now communicate with each other. The intention of the drivers can be broadcasted wide across the road and is going to be much more reliable than the best prediction algorithms that we have right now.
Therefore, I think it is very important to have a mature ADAS device that can be used on every car. And this project is my first attempt in pursuing such final goal.
Purpose and scope
The purpose of this project is to design a prototype of an advanced driver-assistance system. The implemented product should be a system with hardware and software to provide three main functions: (i) forward collision warning with forward vehicles and pedestrians, (ii) lane analysis and lane departure warning, (iii) sign detection for maximum speed limit signs and over-speed warning. In the scope of this project, because of the limitation in time and effort, I was only able to develope and apply lane detection on the device. However, the completed design with all the functionalities is still considered.
Jetson Nano Computer
Released in March 2019 by NVIDIA, Jetson Nano is a powerful platform for deploying machine learning algorithms. Because of a small size board with a quiet strong GPU, it is suitable to be used as the center processing computer. One special feature of this computer in comparison with ones from other companies is that it can use TensorRT, an SDK (software development kit) for high-performance deep learning inference. This SDK includes a deep learning inference optimizer and runtime for low latency and high-throughput experience. In this project, this feature can be leveraged to run deep learning networks to analyze images from dash camera. The detail of system configuration of Jetson Nano is listed below:
- CPU: Quad-core ARM Cortex-A57.
- GPU: 128-core NVIDIA Maxwell architecture-based.
- RAM: 4 GB 64-bit LPDDR4; 25.6 gigabytes/second.
II. Deep Learning Neural Network Training
Lane Detection Module
In order to achieve a light-weight segmentation model to be run on the Jetson Nano, a modified U-Net model was trained. The U-Net is different from its original version as it now has a smaller number of decoder filters and is now using ResNet-18 as its backbone. The number of parameters in the U-Net is therefore decreased to give us a light-weight model.
Dataset
I used Mapillary Vista dataset for the training of the lane detection model. The dataset contains about 18000 images in training set and about 2000 images in validation set.
III. Computer Vision
Lane Detection with Hough Transform
Hough Transform is a line detection algorithm that is very effective in image processing.
The general process of finding lane line candidates is shown in the figure below. The neueral network produces the segmentation mask, and is than processed by probabilistic Hough transform to detect raw line segments. Then the lines are partitioned into groups using disjoint-set/union-find forest algorithm. As a result, the lines are finally grouped into several groups.
Since the left and right lane lines relative to the car (the camera angle) is also determined, the relative positions can then be used to determine lane departure situation.
VI. System Performance
As shown in the video at the top, the lane detection is performing quite consistently in the recorded simulated video. Another video is provided here to demonstrate its behavior on the Jetson Nano.
V. Future Work
- Implement the car detection and traffic sign detection in this project
- Test this device on real car
- Use Jetson Xavier for better performance