AWS DeepRacer is an integrated learning system for users of all levels to learn and explore reinforcement learning and to experiment and build autonomous driving applications. This blog will guide you to build your model on the new Deepracer Console. This can help you get ready for the Deepracer leagues at your nearest summits and also for Virtual circuits that take place throughout the year.
What is AWS DeepRacer?
There are three main racing formats in the AWS DeepRacer League:
Head-to-Head Racing: In this mode, racers navigate the track while avoiding other AWS bot cars. The top racers qualify for a head-to-head elimination bracket at the end of each month.
Object Avoidance: In this format, participants must complete the track while dodging obstacles. The fastest time wins and advances toward the Championship Cup.
Time Trial: Here, racers aim to complete the track within a set number of laps while staying on track. The fastest racer in this category advances to the Championship Cup.
Understanding Reinforcement Learning:
In reinforcement learning (RL), an agent (in this case, the AWS DeepRacer car) interacts with an environment (the track) and learns by receiving rewards for taking certain actions that lead to desirable outcomes. The agent’s goal is to maximize these rewards by identifying the best actions over time.
Key Reinforcement Learning (RL) Terms:
AWS DeepRacer leverages several AWS services:
- AWS DeepRacer Console: Interface for training and evaluating models.
- Amazon SageMaker: Manages and trains the RL models.
- AWS RoboMaker: Simulates the virtual track environment.
- Amazon S3: Stores the model data.
- Amazon Kinesis: Streams data for live feedback.
- Amazon CloudWatch: Monitors and logs activity.
Steps to Build Your AWS DeepRacer Model:
Let’s train and evaluate our virtual model! Here are the steps (note that you can follow along
Building an Effective Reward Function:
The AWS DeepRacer reward function takes a dictionary object as the input.
def reward_function(params) :
reward = ...
return float(reward)
The params dictionary object contains the following key-value pairs:
{
"all_wheels_on_track": Boolean, # flag to indicate if the agent is on the track
"x": float, # agent's x-coordinate in meters
"y": float, # agent's y-coordinate in meters
"closest_objects": [int, int], # zero-based indices of the two closest objects to the agent's current position of (x, y).
"closest_waypoints": [int, int], # indices of the two nearest waypoints.
"distance_from_center": float, # distance in meters from the track center
"is_crashed": Boolean, # Boolean flag to indicate whether the agent has crashed.
"is_left_of_center": Boolean, # Flag to indicate if the agent is on the left side to the track center or not.
"is_offtrack": Boolean, # Boolean flag to indicate whether the agent has gone off track.
"is_reversed": Boolean, # flag to indicate if the agent is driving clockwise (True) or counter clockwise (False).
"heading": float, # agent's yaw in degrees
"objects_distance": [float, ], # list of the objects' distances in meters between 0 and track_length in relation to the starting line.
"objects_heading": [float, ], # list of the objects' headings in degrees between -180 and 180.
"objects_left_of_center": [Boolean, ], # list of Boolean flags indicating whether elements' objects are left of the center (True) or not (False).
"objects_location": [(float, float),], # list of object locations [(x,y), ...].
"objects_speed": [float, ], # list of the objects' speeds in meters per second.
"progress": float, # percentage of track completed
"speed": float, # agent's speed in meters per second (m/s)
"steering_angle": float, # agent's steering angle in degrees
"steps": int, # number steps completed
"track_length": float, # track length in meters.
"track_width": float, # width of the track
"waypoints": [(float, float), ] # list of (x,y) as milestones along the track center
}
The reward function that worked for us is:
import math
def reward_function(params):
track_width = params['track_width']
distance_from_center = params['distance_from_center']
steering = abs(params['steering_angle'])
direction_stearing=params['steering_angle']
speed = params['speed']
steps = params['steps']
progress = params['progress']
all_wheels_on_track = params['all_wheels_on_track']
ABS_STEERING_THRESHOLD = 15
SPEED_TRESHOLD = 5
TOTAL_NUM_STEPS = 85
# Read input variables
waypoints = params['waypoints']
closest_waypoints = params['closest_waypoints']
heading = params['heading']
reward = 1.0
if progress == 100:
reward += 100
# Calculate the direction of the center line based on the closest waypoints
next_point = waypoints[closest_waypoints[1]]
prev_point = waypoints[closest_waypoints[0]]
# Calculate the direction in radius, arctan2(dy, dx), the result is (-pi, pi) in radians
track_direction = math.atan2(next_point[1] - prev_point[1], next_point[0] - prev_point[0])
# Convert to degree
track_direction = math.degrees(track_direction)
# Calculate the difference between the track direction and the heading direction of the car
direction_diff = abs(track_direction - heading)
# Penalize the reward if the difference is too large
DIRECTION_THRESHOLD = 10.0
malus=1
if direction_diff > DIRECTION_THRESHOLD:
malus=1-(direction_diff/50)
if malus<0 or malus>1:
malus = 0
reward *= malus
return reward
For more examples of reward functions, you can visit this link.
Pricing:
AWS DeepRacer provides a Free Tier, that covers the first 10 free hours to train or evaluate models and 5GB of free storage during your first month. After this, it cost around
- Training or evaluation : $3.50 per hour
- Model storage : $0.023 per GB-month
Key Takeaways:
- You should be careful while choosing the training time of your model. If you train it too much it will overfit the track and won’t be able to perform with slight changes in the environment, whereas on the other hand if your train it less, the model won’t be strong enough to make correct decisions.
- Your primary focus while building and training the model on the virtual environment should be on the accuracy and reliability of your model and not the speed or lap time of your DeepRacer. As in the Physical racing league, you will be able to accelerate your DeepRacer using their app on the phone.
SOCIAL SHARE CARD GENERATOR