Advanced Lane Finding Project

The goals / steps of this project are the following:

• Compute the camera calibration matrix and distortion coefficients given a set of chessboard images.
• Apply a distortion correction to raw images.
• Use color transforms to create a thresholded binary image.
• Apply a perspective transform("birds-eye view") to rectify binary image .
• Detect lane pixels and polynominal fit to find the lane boundary.
• Determine the curvature of the lane and vehicle position with respect to center.
• Warp the detected lane boundaries back onto the original image.
• Output visual display of the lane boundaries and numerical estimation of lane curvature and vehicle position.

This is the advanced project of my lane detection project. It aims to detect the driving lane in various environment in videos and calculate lane curvature. The output looks like the following gif:

Normal Vidieo	Challenge Video	Harder Challenge Video

project.py contains source code to process the video.

• cv2
• Numpy
• collections
• moviepy.editor

The chessboard corners are the reference to generate objpoints and imgpoints.

I then used the output objpoints and imgpoints to compute the camera calibration and distortion coefficients using the cv2.calibrateCamera() function. I applied this distortion correction to the test image using the cv2.undistort() function and obtained this result:

Matrix mtx and dist from camera calibration are applied to distortion correction to one of the test images like this one:

This part is the key to the lane detection. Inspired by Udacity cource and Justin Heaton, I tried RGB, HLS, LUV and LAB color space, in addition, gradient and magnitude, and their combination. As you can see from the following comparison, RGB filter is very sensitive to the threshold and could not split lanes from lightful environment. S channel and gradient combination could split lines on the road, but too much disturbance also left.

Finally I found color space transformation method as last figure shows. B channel from LAB space did a fairly good job to identify yellow lanes while L channel from LUV space could detect white lanes.

I verified that my perspective transform was working as expected by drawing the src and dst points onto a test image.

Firstly, I calculated the histogram of non-zero x-axis in binary image. And based on the maximum sum of x position, I used sliding window search method to identify lane pixels. If it is previously detected in last frame, a quick search could be applied based on last detected x/y pixel positions with a proper margin.

Then I fitted lane lines with a 2nd order polynomial kinda like this:

To connect pixel unit with real world meter unit, I defined conversions in x and y from pixels space to meters. In order to calculate precisely, I used detected lane width dynamically.

ym_per_pix = 30 / 720  # meters per pixel in y dimension
xm_per_pix = 3.7 / abs(xleft_eval - xright_eval)  # meters per pixel in x dimension
xmean = np.mean((xleft_eval, xright_eval))
offset = (img_shape[1]/2 - xmean) * xm_per_pix  # +: car in right; -: car in left side

fit_cr = np.polyfit(ploty * ym_per_pix, fitx * xm_per_pix, 2)
curverad = ((1 + (2 * fit_cr[0] * y_eval * ym_per_pix + fit_cr[1]) ** 2) ** 1.5) / 
                np.absolute(2 * fit_cr[0])

Last step is to transform the lane identified back onto the road image. Here is an example of my result of work flow:

There are various difficult environment conditions in the video, such as shadows, brightful / dirty road surface and confusion of new and old lane marks. To combat with all these conditions and make my algorithm robust, a Line Class is created to record and interact key line info among lastest multiple frames, which includs polynominal fit, x and y pixels and, more importantly, whether line is detected in the last frame.

I spent lots of time on selecting proper color space and applicable threshold for finetuning. For now in project.mp4 and challenge.mp4 videos, the lane detection is stable and correctly. However, it is still far from robust to deal with harder_challenge video, in which the light differs extremely and part of lane is invisible for a few seconds. Recalling the behavioral cloning and transfer learning benefits with deep learning neural networks, I missed "smart brain" so much, which can be adaptive to various conditions and automatically self-fit.

1. Apply algorithm to my own camera recordings.
2. Combine deep learning thought with computer vision pipeline.