starsky logo

Starsky Robotics is an autonomous trucking company working to make trucks autonomous on the highway and remote controlled for the first and last mile

safety-driver
safety-engineer

Synopsis

Starsky Robotics was working on an autonomous vehicle solution with a clever twist. They would use a remote-driver (aka teleoperation) for anything their AI system couldn't handle.

I was responsible for making the teleoperation program succeed. I started with a rough prototype that could drive 10 mph on a straight road. By the time I finished, we had two teleoperation centers driving trucks in live traffic.

The challenge

The challenge

The founders wanted to demo in less than 100 days

The founders wanted a demo in less than 100 days

The founders wanted a demo in less than 100 days

We needed to demonstrate a “gate-to-gate” trucking run using only the teleoperation and autonomous systems.

Early teleop team

My role

I was Senior UX Designer and the only employee working specifically on the teleoperation program.

I designed the system hardware & software, developed the training program, and coordinated with the founders and autonomous team for access to resources.

INITIAL ASSESSMENT

The prototype required a team-effort
to drive 10 mph on a straight road

The prototype required a team-effort to drive 10 mph on a straight road

The prototype required a team-effort to drive 10 mph on a straight road

teleop-prototype-station-large
teleop-prototype-screen

The HUD displayed video streamed from a 180° camera mounted in the truck. At the bottom of the screen are indicators for speed and steering-wheel rotation.

teleop-prototype-team

Driving tests took place in a busy open-office environment. The remote driver and safety driver in the truck could not talk directly. All communication was relayed by engineers on cell phones.

UX Design

Ergonomics

starsky-ergonomics-big-example

Driving figure eights in the new teleoperation station

starsky-ergonomics-small-example
starsky-ergonomics-engineer-example
The easiest way to drive adoption for an experimental system is to make it feel natural and familiar

I relocated the control station to a private room. I upgraded all the equipment to make the driving experience more true to life. Microphones and speakers were installed so the drivers could talk directly.

Poor video visibility was an ongoing challenge. Some of my solutions included:

  • apply a matrix algorithm to warp the video fit the screens
  • split the cell signal between services to reduce dropouts
  • repositioning the driver camera so percieved speed matched actual speed

UX Research & Design

Safety & confidence

starsky-teleoperation-screen-hud-bar-closeup

Teleoperation control station HUD

starsky-safety-driver-screen

Safety driver HUD

Drivers need situational awareness at all times

The system was prone to problems like disconnections and signal loss. I immediately added error alerts and system status indicators. They were prominently displayed in the teleoperation control station and safety driver HUD.

We also established verbal communication during handoffs. Drivers were required to confirm "I have control / you have control.” They practiced various safety drills at the start of every training session.

UX Research

Driver training

training diagrams
training-diagrams

Early training exercise diagrams

Start with a video game

I installed American Truck Simulator in the control station so new remote-drivers could get used to the interface before handling a real truck.

Hands-on training was incremental. We started with slow figure-eights, handoffs, and safety drills. Drivers gradually learned how to apply the brakes and make turns without tossing everyone around in the truck.

Over time, we were able to move from a closed lot to low-traffic areas. After that was driving with a trailer, driving on the highway, on-ramps, off-ramps, and eventually in street traffic.

teleop-view

Waiting to make a left turn in Florida

SYSTEM Design

Designing for scale

The initial prototype was a one-to-one system with one truck linked to the station. A developer had to update code so we could switch between trucks.

I separated the admin settings from the driver station so the station only contained video, indicators, and controls. The admin station slowly expanded into managing multiple trucks and tracking their location, speed, fuel levels, and cell strength.

A month before the demo, I opened a new teleoperation center in Florida. This required us to develop new safety checks to prevent conflicts between operation centers.

Delivery

A successful demo

On September 22nd, 2017 Starsky Robotics completed the world's first end-to-end autonomous trucking run

On September 22nd, 2017 Starsky Robotics completed the world's first end-to-end autonomous trucking run

no hands
hands-up
Starsky Teleop

Selected works

Omnicell: Figma libraryModular design library

Obvious: MVP designHigh-fidelity design specifications

Obvious: wireframesMulti-device strategy for a solo developer

OmnicellSketch design library & developing interactive kiosks

Starsky RoboticsUX research & design for a remote-driving system

Disney Movies AnywhereMaterial Design & Development

Star Wars APIAndroid design & code exercise

Fitness goals appProduct design exercise

DayframeAutomated photo frame for social media

HD WidgetsTop 10 Android app 2011-2014

App StatsPlay Store app tracking & metrics

CloudskipperFree music player

Flash / FlexBack when the web was fun

Elemental DesignOld school rave flyer designs