starsky logo

Starsky Robotics is an autonomous trucking company working to make trucks autonomous on the highway and remote controlled for the first and last mile

safety-driver
safety-engineer

Context

The US is going to need 100k more truck drivers in the next five years. But truck driving is a demanding job with long hours of isolation far away from home, resulting in high turnover and a shortage of drivers.

Starsky Robotics is trying to solve that by producing an autonomous + teleoperation system. Instead of driving trucks directly, drivers would work in centralized control centers, supervising a dozen self-driving trucks each, and be able to go home at the end of the day.

The challenge

The founders wanted to demonstrate a working MVP in less than 100 days

I was pivotal in helping Starsky complete an end-to-end run in live traffic, controlled exclusively by remote driving and the autonomous system.

Early teleop team

My role

I was responsible for delivering the remote-driving system:

  • UX research & design through frequent iterative cycles
  • Product / UI design for the control station, command hubs, and various truck HUDs
  • Program manager including project roadmap, resource allocation & scheduling, tracking progress and metrics, and driver training & evaluation

I was responsible for delivering the teleoperation system:

  • UX research & design through frequent iterative cycles
  • Product / UI design for the control station, command hubs, and various truck HUDs
  • Program manager including project roadmap, resource allocation & scheduling, tracking progress and metrics, and driver training & evaluation
BACKGROUND

Introduction to the remote driving prototype

teleop-prototype-station@2x

When I joined Starsky Robotics, they had a working prototype that could drive 10 mph on a straight road. A camera in the truck transmitted live video to the control station, which would return user telemetry for steering, gas, and brakes.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

teleop-prototype-team

Testing took place in a busy workspace. There was no direct communication between the remote driver and safety driver — everything was relayed between engineers on cell phones.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

Remote sessions would take place in a busy work area. There was no direct communication between the remote driver and safety driver — everything was relayed between engineers on cell phones.

teleop-prototype-screen

Three 20-inch monitors displayed a 7-inch high video feed that was prone to visual artifacts and signal loss. The HUD only had gauges for speed (in kph) and steering.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

hardware update
UX RESEARCH & DESIGN

Upgrading the control station for better ergonomics

The control station was relocated in an isolated room with upgraded hardware and viewscreens to match a truck ergonomically, based on research of existing training simulators.

Microphones and speakers were installed in the control room and truck so everyone could talk to each other.

We experimented frequently with resolutions, data rates, formats, and hardware. The control system depended on ultra-low latency video — anything over 250 ms meant more than a half-second between an event and response.

HUD closeup
UX / PRODUCT DESIGN

Implementing safety systems and driver communication

There was always a safety driver in the truck ready to intervene if something went wrong. But critical problems like signal loss or a system malfunction would occur and were not always obvious.

I quickly added prominent warning alerts and status mode indicators to the control station and safety driver’s HUDs.

I also established verbal communication protocols. Drivers were required to confirm ”you have control", "I have control" during every handoff. They also practiced  “no signal!” and other safety drills at the start of each training session.

training diagrams
training-diagrams
UX RESEARCH

Developing a training program for remote drivers

We experimented with basic driving skills. Some carried over easily (steering, stopping distance) while others required retraining (turn velocity, braking).

Next, drivers would take the tractor out on small streets to practice lane-keeping, turns, intersections, and driving in minimal traffic. In time they moved on to more advanced tasks and eventually, they could safely remote-operate a tractor-trailer through city streets and on the highway.

teleop-view
PRODUCT DESIGN

Expanding to multiple teleoperation centers

Over time, I separated admin controls from driver controls, creating the building blocks for a command station to serve as a hub to multiple driving stations.

When we set up a second teleoperation center in Florida, this approach allowed us to quickly redesign and reconfigure our control system from a one-to-many to an any-to-any system.

Naturally, new safety features had to be implemented to prevent conflicts between the distant centers.

CONCLUSION

Delivering a successful demo

The entire team traveled to Florida and spent a week doing practice runs and finalizing the controls. It was long hours and little sleep, but we pulled it off.

In September 2017, Starsky Robotics completed the world's first end-to-end, autonomous trucking run.

no hands
hands-up
Starsky Teleop

Selected works

Starsky RoboticsUX research & design for autonomous vehicles

SWAPI: Android demoUX development exercise

Exercise goalsProduct design exercise

Disney Movies AnywhereSeinor UX Developer, Product Designer

DayframePrinciple Designer, Senior UX Developer

HD WidgetsPrinciple Designer, Senior UX Developer

App StatsPrinciple Designer, Senior UX Developer

CloudskipperPrinciple Designer, Senior UX Developer

Flash apps2000 - 2011

Elemental design1994 - 2003