This repository contains the results of team [insert name here]'s Capstone project for the Udacity Self-Driving Car Engineer Nanodegree.

The project utilizes Ubuntu Linux 14.04 or 16.04 with Robot Operating System (ROS), the Udacity System Integration Simulator and code written in C++ and Python to provide a System Integration solution to the self-driving car problem.

ROS nodes were implemented in order to achieve a core functionality of the autonomous vehicle system, including traffic light detection, control and waypoint following.

The following is a system architecture diagram showing the ROS nodes and topics used in the project.

The purpose of this node is to publish a fixed number of waypoints ahead of the vehicle. It subscribes to the following topics:

• /base_waypoints - Publishes a list of all waypoints for the track, which includes waypoints ahead and behind the vehicle. Only published once
• /current_pose - Publishes the current position of the car
• /current_velocity - Publishes the current velocity of the car
• /traffic_waypoint - Traffic light data, position and status

The node will then publish to the following topic:

• final_waypoints - list with a fixed number of waypoints currently ahead of the vehicle with the correct target velocities, depending on traffic lights.

With the messages being published to /final_waypoints, the vehicle's waypoint follower will publishtwist commands to the twist_cmd topic and provide appropriate throttle, brake and steering commands. These commands can then be published to the following topics:

• /vehicle/throttle_cmd
• /vehicle/brake_cmd
• /vehicle/steering_cmd

This node will also subscribe to the /vehicle/dbw_enabled topic, which contains the current DBW status, i.e., Manual or Autonomous. This is in place for situations where the safety driver takes over, and Autonomous Mode is disengaged.

For the classifier a hybrid approach was followed by having two models with two separate graphs available within the final submission, one for the sim and a second one for the site run. An SSD Inception V2 Coco was trained using data captured and labeled from the Unity simulator and an RCNN Inception V2 Coco model trained on the real-world data, the ROSbag file recorded on the Udacity self-driving car. The SSD model proved to be fast enough to cope with a significantly higher rate of change for the traffic lights throughout the simulator lap, however it wasn't good enough to generalize on new data. The RCNN inference is significantly slower (up to 3s) but very precise, with good results on real data.

The ROSbag file used for training the SSD model is a collection of simulator captured images, labelled with labelImg. A TFRecord file was further generated alongside a pbtxt file with Red, Green, Yellow and Off labels, both used for completing the transfer learning of the Coco-trained models.

The training was done using an AWS EC2 instance over 10000 steps for both models.

Please note Vatsal's dataset was used as well for both evaluation and training to supplement the existing datasets.

Please use one of the two installation options, either native or docker installation.

• Be sure that your workstation is running Ubuntu 16.04 Xenial Xerus or Ubuntu 14.04 Trusty Tahir. Ubuntu downloads can be found here.

• If using a Virtual Machine to install Ubuntu, use the following configuration as minimum:

  • 2 CPU
  • 2 GB system memory
  • 25 GB of free hard drive space

  The Udacity provided virtual machine has ROS and Dataspeed DBW already installed, so you can skip the next two steps if you are using this.

• Follow these instructions to install ROS

  • ROS Kinetic if you have Ubuntu 16.04.
  • ROS Indigo if you have Ubuntu 14.04.

• Dataspeed DBW

  • Use this option to install the SDK on a workstation that already has ROS installed: One Line SDK Install (binary)

• Download the Udacity Simulator.

Install Docker

Build the docker container

docker build . -t capstone

Run the docker file

docker run -p 4567:4567 -v $PWD:/capstone -v /tmp/log:/root/.ros/ --rm -it capstone

To set up port forwarding, please refer to the instructions from term 2

1. Clone the project repository

git clone https://github.com/udacity/CarND-Capstone.git

2. Install python dependencies

cd CarND-Capstone
pip install -r requirements.txt

3. Make and run styx

cd ros
catkin_make
source devel/setup.sh
roslaunch launch/styx.launch

4. Run the simulator

1. Download training bag that was recorded on the Udacity self-driving car.
2. Unzip the file

unzip traffic_light_bag_file.zip

3. Play the bag file

rosbag play -l traffic_light_bag_file/traffic_light_training.bag

4. Launch your project in site mode

cd CarND-Capstone/ros
roslaunch launch/site.launch

5. Confirm that traffic light detection works on real life images