starsky logo

Starsky Robotics is an autonomous trucking company working to make trucks autonomous on the highway and remote controlled for the first and last mile

safety-driver
safety-engineer

Context

The US is going to need 100k more truck drivers in the next five years. But truck driving is a demanding job with long hours of isolation far away from home, resulting in high turnover and a shortage of drivers.

Starsky Robotics is trying to solve that by producing an autonomous + teleoperation system. Instead of driving trucks directly, drivers would work in centralized control centers, supervising a dozen self-driving trucks each, and be able to go home at the end of the day.

The challenge

The founders wanted to demonstrate a working MVP in less than 100 days

I was pivotal in helping Starsky complete an end-to-end run in live traffic, controlled exclusively by remote driving and the autonomous system.

Early teleop team

My role

I was responsible for delivering the remote-driving system:
  • UX research & design through frequent iterative cycles
  • Product / UI design for the control station, command hubs, and various truck HUDs
  • Program manager including project roadmap, resource allocation & scheduling, tracking progress and metrics, and driver training & evaluation

I was responsible for delivering the teleoperation system:

  • UX research & design through frequent iterative cycles
  • Product / UI design for the control station, command hubs, and various truck HUDs
  • Program manager including project roadmap, resource allocation & scheduling, tracking progress and metrics, and driver training & evaluation
BACKGROUND

Assessing the  prototype

teleop-prototype-station@2x

When I joined Starsky Robotics, they had a working prototype that could drive 10 mph on a straight road. A camera in the truck transmitted live video to the control station, which would return user telemetry for steering, gas, and brakes.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

teleop-prototype-team

Testing took place in a busy workspace. There was no direct communication between the remote driver and safety driver — everything was relayed between engineers on cell phones.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

Remote sessions would take place in a busy work area. There was no direct communication between the remote driver and safety driver — everything was relayed between engineers on cell phones.

teleop-prototype-screen

Three 20-inch monitors displayed a 7-inch high video feed that was prone to visual artifacts and signal loss. The HUD only had gauges for speed (in kph) and steering.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

hardware update
UX RESEARCH & DESIGN

First steps: better ergonomics

Based on my observations of the prototype, I determined we needed a better signal-to-noise ratio. This meant optimizing the video quality, removing distractions, and providing a clear channel of communication between the drivers.

I relocated the control station to an isolated room with upgraded hardware and viewscreens to match a truck ergonomically.

I installed a microphone and speakers in the control room and truck so everyone could talk to each other.

We experimented frequently with video resolutions, data rates, formats, and hardware. We warped the video to fit the screens and lowered the camera to provide a better view of the horizon.

HUD closeup
UX / PRODUCT DESIGN

Establishing self-confidence through safety systems and  communication

There was always a safety driver in the truck ready to intervene if something went wrong. But critical problems like signal loss or a system malfunction would occur and were not always obvious.

To establish self-confidence, I needed to make it perfectly clear to the drivers what the truck was doing and who was managing it. More importantly, they had to know right away if the state had changed unexpectedly or didn't change as expected.

I quickly added prominent warning alerts and status mode indicators to the control station and safety driver’s HUDs. This included a "NO SIGNAL!" alert when the station lost connection with the truck for more than a half-second.

I established verbal communication protocols. Drivers were required to confirm ”you have control", "I have control" during every handoff. They also practiced  “no signal!” and other safety drills at the start of each training session.

training diagrams
training-diagrams
UX RESEARCH

Driver training

I relied on the drivers' training experience and knowledge for driver training. They knew the basic steps as well as the best locations we could practice.

I created simple exercises at first, then more complex sets. Soon after, drivers would take the truck out on small streets and in time, some of them could remotely drive a tractor with a trailer through city streets and on the highway.

The UX research was very casual. I noted (but didn't measure) the driver's confidence level, communication, and responsiveness. I'd talk with drivers before, during, and after training sessions to gather feedback on the experience which allowed me to quickly iterate on improvements.

PRODUCT MANAGEMENT

Managing the teleoperations program

Although my title was Senior UX Designer, I essentially managed the teleoperations program. I set roadmaps and schedules, evaluated drivers, tracked metrics for success, and negotiated for resources.

For metrics, I tracked:

  • hands-on time for drivers, the total time to reach milestones, and the reduction in time to reach those milestones as we improved the system
  • technical issues such as video artifacts, signal loss, and where the truck was located at the time to determine if the problem was our bug or a limitation of the environment
  • downtime caused by operational issues, which lead to the engineering team maintaining a working branch for teleoperations

I negotiated for resources and set roadmaps

The team was focused on autonomous testing, so I had limited access to drivers, trucks, and the principal software engineer. Also, they were bringing in different drivers each week, so remote-driving training could only cover basic skills.

To ensure we could still achieve our goal, I pitched that we focus on a single driver and safety driver. I then evaluated drivers and helped select the final candidates. 

Next, I worked on providing equal access to autonomous and teleoperation resources. I showed them the current, irregular schedule and then proposed a more routine schedule which would overlap between California and Florida allowing us three half-day sessions per day. I continued to provide the schedule outline for each week.

Finally, there was an expectation that they would have a fully autonomous system working in six months and 10,000 trucks within a year. To temper this expectation, I produced a technology tree of automotive system integration requirements and milestones they would need to achieve their goal.


teleop-view
PRODUCT DESIGN

Designing for scale

My long-term vision for teleoperations was a traffic control center with a few command stations managing several driving stations. The center would manage a fleet of trucks and handle departures, arrivals, and special circumstances.

Working towards this vision, I started separating admin controls from driver controls. The driving station contained video feeds, telemetry gauges, and driving controls. The admin panel had tools to track multiple trucks and assign a truck to the control station.

When we set up a second teleoperation center in Florida, my approach allowed us to quickly reconfigure our control system from a one-to-many to an any-to-any system. Naturally, new safety features had to be designed and implemented to prevent conflicts between the distant centers.

CONCLUSION

Delivering a successful demo

The entire team traveled to Florida and spent a week doing practice runs and finalizing the controls. It was long hours and little sleep, but we pulled it off.

In September 2017, Starsky Robotics completed the world's first end-to-end, autonomous trucking run.
no hands
hands-up
Starsky Teleop

Selected works

Starsky RoboticsUX research & design for autonomous vehicles

SWAPI: Android demoUX development exercise (2019)

Exercise goalsProduct design exercise

Disney Movies AnywhereSeinor UX Developer, Product Designer

DayframePrinciple Designer, Senior UX Developer

HD WidgetsPrinciple Designer, Senior UX Developer

App StatsPrinciple Designer, Senior UX Developer

CloudskipperPrinciple Designer, Senior UX Developer

Flash apps2000 - 2011

Elemental design1994 - 2003