starsky logo

Starsky Robotics is an autonomous trucking company working to make trucks autonomous on the highway and remote controlled for the first and last mile

safety-driver
safety-engineer

Synopsis

Autonomous vehicles are hard. The hardest part is creating an AI that can react naturally to the environment. Starsky Robotics was working on an autonomous solution with a clever twist. They would use a remote-driver for anything the AI couldn't handle.

I was responsible for making the teleoperation program succeed. I started with a rough prototype that could drive 10 mph on a straight road. When I finished, we had multiple control stations that could drive trucks in live traffic.

The challenge

The founders wanted to demo an MVP in less than 100 days

The demo would be a simulation of a gate-to-gate trucking run. We could only use teleoperation and the autonomous system. The safety driver could not intervene.

My role

I was the sole Senior UX Designer. I was responsible for managing and delivering the teleoperation program.

I was the only person working full time on teleoperation. I worked directly with the lead software developer and a half-dozen drivers. I coordinated with the founders and autonomous team for access and resources.

Early teleop team

INITIAL ASSESSMENT

The prototype required a team-effort to drive 10 mph on a straight road

teleop-prototype-station-large

The system was a modified racing-game station with a truck steering wheel.

teleop-prototype-screen

The HUD displayed video streamed from a 180° camera mounted in the truck. At the bottom of the screen are indicators for speed and steering-wheel rotation.

teleop-prototype-team

Driving tests took place in a busy open-office environment. The remote driver and safety driver in the truck could not talk directly. All communication was relayed by engineers on cell phones.

UX & Product Design

Ergonomics

starsky-ergonomics-big-example
starsky-ergonomics-small-example
starsky-ergonomics-engineer-example

Teleoperation control station in San Francisco

The easiest way to drive adoption for an experimental system is to make it feel familiar and natural

I relocated the control station to a private room. I upgraded all the equipment to make the driving experience more true to life. Microphones and speakers were installed so the drivers could talk directly.

Poor video quality was an ongoing challenge that required clever solutions. We split the cell signal between services to reduce dropouts. We applied a matrix algorithm to warp the video fit the screens. I repositioned the camera in the truck to reduce angular velocity misperception. It looked like they were driving much faster than they actually were.

UX Research & Design

Safety = confidence

starsky-teleoperation-screen-hud-bar-closeup

Teleoperation control station HUD

starsky-safety-driver-screen

Safety driver HUD

The only way we could move forward was with strong safety systems and practices

The drivers needed complete situational awareness at all times. Not only what the truck was doing, but who (or what) was in control of the truck at any given time.

The system was prone to problems like disconnections and signal loss. I immediately added error alerts and system status indicators. They were prominently displayed in the teleoperation control station and safety driver HUD.

We established verbal handoff protocols. Drivers were required to confirm "I have control / you have control" during handoffs. They practiced safety drills at the start of every training session.

UX Research

Driver training

training diagrams
training-diagrams

Early training exercise diagrams

Training was iterative cycles of small steps and big leaps

New drivers started by playing a video game (American Truck Simulator). This enabled them to become comfortable with the control station.

The hands-on training was experimental. We started with baby-steps: slow figure-eights, handoffs, and safety drills. Next, we started to address “stuff you can’t feel because you’re not in the truck” like controlled braking and turns.

Everything from there was a matter of progression. We moved from a closed lot to low-traffic areas and driving with a trailer. After that was driving on the highway, on-ramps, off-ramps, and eventually in street traffic.

Product Design

Designing for scale

teleop-view

Remote driver waiting to make a left turn in a rest stop in Ft. Lauderdale

I scaled the system to multiple control stations & trucks

The initial prototype was a one-to-one system with one truck linked to the station. An engineer had to update the code to switch to another truck.

I decoupled the teleoperation system into separate master and control stations. The master station could track and assign trucks. Tracking included location, speed, fuel levels, and cell strength for all trucks. The control station focused on driving systems like video and indicators.

A month before the demo, I opened a new teleoperation center, this time in Florida. The second location required new safety checks for conflict prevention between master stations.

Delivery

A successful demo

On September 22nd, 2017 Starsky Robotics completed the world's first end-to-end autonomous trucking run

no hands
hands-up
Starsky Teleop

Selected works

Activity JournalProductivity app for iPhone, iPad, & macOS

OmnicellDesign component library, responsive web design, & interactive kiosks

Starsky RoboticsUX research & design for a remote-driving system

Star Wars APIAndroid design & code exercise

Exercise GoalsProduct design exercise

Disney Movies AnywhereSenior Android Developer, Product Designer

DayframeAutomated photo frame for social media

HD WidgetsPopular widget app

App StatsPlay Store app tracking & metrics

CloudskipperFree music player

Flash / Flex2000 - 2011

Elemental Design1994 - 2003