starsky-logo

I joined Starsky Robotics to design and help manage their teleoperations program. In less than 100 days we needed to transform a simple remote driving prototype into a fully operational system.

In September 2017, our remote driver completed the first ever end-to-end, unmanned run in live traffic.

Truck Interior

The Challenge

Starsky Robotics is working on an artificial intelligence system for big-rig trucks that run autonomously on highways, while driven from a remote operations center for all other conditions.

The founders had given the team a challenge:

demonstrate a real-world, unmanned end-to-end run...
in less than 100 days.

This meant that a remote driver would:

  1. Drive a loaded traktor-trailer from the gate, through traffic, and onto the highway
  2. Activate the system to operate autonomously
  3. Supervise the highway journey until the truck reached the exit
  4. Resume control of the vehicle and exit the highway
  5. Navigate through streets until arriving at the final gate.
Early teleop team

My Role

As Sr. UX Designer I was primarily responsible for managing the teleoperations program including:

  • Hardware / software research & design
  • Driver experience, training, and evaluation

Note: due to the company’s confidentiality, specifics of my work won’t be disclosed. I’ve limited this case study to solutions made availalble publicly.

My Role

As Sr. UX Designer I was primarily responsible for managing the teleoperations program including:

  • Command and control station research and design
  • Driver training & evaluation

Note: due to the company’s confidentiality, specifics of my work won’t be disclosed. I’ve limited this case study to solutions made availalble publicly.

Design Approach

Progress moves fast in a start-up environment. Meetings were done on the spot, changes came rapidly, and there was no existing model for what we were trying to achive.

I applied a lean, iterative approach to developing the program. Time was limited, so I spent most of it optimzing the user experience rather than on visual design and design processes.

I'd make small changes each day and observe how the changes affected driver abilities. Throughout development, I'd discuss these changes with the drivers and engineers for feedback and suggestions.

Early Prototype

When I first joined Starsky, I watched a couple of demo sessions of the early prototype. It had the essential ingredients - low-latency video and response time for the controls - but the capabilities were very limited.

I could tell there were serious issues that could be immediately fixed:

  • The control interface was visually and physically cramped.
  • There was no direct communication between the drivers (it was  relayed between engineers at each end).
  • The workspace / kitchen surrounding the prototype was active and loud.
  • It wasn't always clear who was in control of the vehicle.

UX Design & Research

marie and team

The remote driver experience was the fundamental challenge. We needed something that made drivers feel comfortable, in control, and  confident. We also needed to be able to monitor and measure progress live during training sessions without distracting the driver.

Improve Driving Experience

  • Isolated the control station in a closed suite to avoid environmental distractions
  • Installed truck driving simulator software to safely introduce drivers to the control interface
  • Provided wireless headsets and direct link between remote & safety driver
  • Established verbal communication protocols for handoffs and critical safety issues
  • Scouting training locations for increasing skill levels
  • Researched solutions to provide a better "feel" for driving (e.g. audible, physical)

Develop UX Research

  • Added observation desks and seating behind driver
  • Followed training sessions with minimal intervention
  • Tracked potential safety issues, driver ability, and communication
  • Debriefed remote drivers after sessions
  • Evaluated for final driver selection
  • Reported findings to founders
teleop-view

Hardware Design

Teleoperation is a two-part system: the control system in the suite and the remote system in each truck.

My first task was to upgrade the control station to full size screens and controls. I started with a modular system that allowed us to change positions, distances, and heights of the controls and interfaces.

Over time I'd continue to research and experiment with various off-the-shelf and custom-built components.

Control Station

  • Implemented modular system for height, position
  • Installed "truck window" sized HDTV screens
  • Installed a gaming wheel with control buttons for system commands
  • Custom stand for steering wheel and foot pedals
  • Ergonomic seating and side table for the driver (sorry, no cup holders)

Remote System

  • Updated video camera for optimal viewing perspective
  • Standardized status & warning lights
  • Added cell phone mount, wireless headset, and extra charging cables in each truck for safety drivers

Hardware UX Example

One interesting example of how a small UX design improvement can make a big change was when I had the engineers remount the video camera in the truck cab.

We spent a great deal of time working with drivers to develop a more natural feel for driving. They were hesitant to drive at higher speeds because low speeds already felt too fast.

From the beginning, the video camera in the truck cab was mounted above the safety driver's head. The idea was that the remote driver would need to see from the same direct angle.

teleop-screen

Unfortunately, the higher position and low-pointing angle forced the remote driver to view more of the road surrounding the truck, rather than the road out on the horizon. This caused everything to feel like it was moving faster than it actually was.

I decided to experiment. I had an engineer go out to a the truck, unmount the camera, and move it around by hand while we observed from the control station. We soon found a better position: down more alongside the safety driver's head.

the-road

The engineers built a new mount, installed the camera, and the difference was subtile but striking. The remote driver's perspective was naturally back on the horizon.

Driving speeds immediately improved and training became much easier.

HUD closeup

Software Design

I spent more time on UX design than UI design. The little UI work I did do focused on making the HUD uncluttered and easy to scan.

I worked with drivers to determine which gauges were most often needed, and which gauges would need extra details for remote control such as brake and gas comparison gauges.

It was also necessary to separate the command controls from the control station. As we expanded, we needed to be able to manage multiple stations and prevent conflicts between active systems.

Control Station

  • Separated command controls to separate window
  • Added status and warning indicators
  • Added comparison gauges for key controls
  • Gauges for other truck systems
  • Optimized video rates and resolution for extreme low-latency signal

Command Station

  • Designed to manage multiple control stations from single command station
  • Connect any control station to any truck
  • Track availability and connection status for all trucks
  • Track various telemetry for all trucks
  • Follow trucks on GPS maps
training diagrams
training-diagrams

Driver Training & Evaluation

Driver training and evaluation fell under the purview of UX research. We didn't have a training program in place, so I had to make it up as we went along. I immediately established communication protocols so drivers would always know who controlled the vehicle and the quality of the connection between the station and the truck.

We tried different drills early on (see above), eventually establishing a system for new and returning drivers. Our drivers also helped scout out new locations for training, which help expand driving from closed lots to off-streets, and eventually into highway and city traffic.

Remote Driver Training

  1. Drivers were first introduced to truck driving simulator so they could get familiar with screens and controls.
  2. Next drivers practiced communication drills while driving in a closed environment. All training sessions started with these drills.
  3. Drivers practiced essential driving skills including accelleration, deceleration, turns, lane-keeping, stopping, and stop-distance.
  4. Drivers would progress from closed lot to low traffic environments, highways, and eventually street traffic.

Driver Evaluation

All of our truck drivers had 5 years of driving expereince with zero traffic incidents on their records. Both remote drivers and safety drivers were evaluated for their ability to participate in teleoperation sessions, including:

  • Remote driver skill development
  • Remote driver confidence
  • Ability to adhere to communication protocols
  • Ability to identify and respond efficiently to issues
  • Situational awareness vs. prone to distraction

Final Result

the-road
hands-up
no hands

Eventually we settled on our finalists for remote driver and safety driver. We worked up until the last minute making incremental changes to the control hardware and software.

Once we were confident we could complete the challenge, the entire Starsky team went down to Florida and had a great time producing the final unmanned, end-to-end run with no interventions.

In 2018, Starsky released The Long Haul, documenting our success:

Starsky Truck

Bonus design: truck wrap

Starsky RoboticsProject type

DayframeProject type

HD WidgetsProject type

App StatsProject type

CloudskipperProject type

Flash Design & AppsProject type

Elemental DesignProject type