starsky logo

Starsky Robotics is an autonomous trucking company working to make trucks autonomous on the highway and remote controlled for the first and last mile

safety-driver
safety-engineer

Synopsis

Starsky Robotics was working on autonomous trucking solutions that used teleoperation (a remote-control driver) for anything the AI couldn't handle.

When I joined the company, they had a primitive prototype that could only drive safely at slow speeds with minimal turning and a lot of supervision.

In my first three months, I successfully converted the prototype into a network of stations capable of operating multiple trucks independently (with a safety driver) in live traffic.

The challenge

The founders wanted to demonstrate a working MVP in less than 100 days

Our truck had to do an end-to-end run using only teleoperation and autonomous driving, with no interventions from the safety driver.

Early teleop team

Role

Senior UX Designer,
Product Manager

Tools

QT, PyQT, Omnigraffle,
American Truck Simulator

Contribution

  • UX research & design
  • Resource allocation & scheduling
  • Driver training & evaluation
  • Progress metrics
  • HUD design for teleoperation, safety driver, & engineering systems
  • Website & truck skin designs

Initial Assessment

The prototype

teleop-prototype-station

The prototype was a modified racing-game station. It had gas and brake pedals, a modified trucking steering wheel, a racecar seat, and three 20” screens. An engineer sat next to the driver to manage technical/admin controls.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

teleop-prototype-team

Remote driving took place in a busy workspace. Communication between the remote and safety drivers was relayed between engineers on cell phones.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

teleop-prototype-screen

A panoramic conference camera in the cab, along with two reverse angle side cameras, transmitted the view around the truck to the station.

The prototype they had in place when I started was a modified video game racing harness. A camera placed over the driver seat in the truck would stream live video to the control station, and the station would return user inputs for steering, gas, and brakes.

starsky-ergonomics-big-example
starsky-ergonomics-small-example
starsky-ergonomics-engineer-example

Teleoperation control station in San Francisco

UX Design

Ergonomics

The easiest way to drive adoption for an experimental system is to make it as familiar and natural as possible.

My first step was to move the control station to an isolated room. We replaced the gaming-chair rig with adjustable seating, controls, and 55” HDTVs on wheeled stands so drivers could arrange everything how they preferred it. Software indicators were arranged in a floating window that drivers could reposition on the screen.

We used a matrix algorithm to warp the video and make it more proportional to the screens and view. We repositioned the camera in the truck to focus more on the horizon and significantly reduce jiggle and angular velocity issues.

Based on feedback from the drivers, we added indicators for brake and gas pedal alignment, engine RPMs, and transmission gear. They also asked for a better “feel” for driving, so I started experimenting with playing engine noise through a backpack subwoofer for haptic feedback.

starsky-teleoperation-screen-hud-bar-closeup

Teleoperation control station HUD

starsky-safety-driver-screen

Safety driver screen (in the truck)

Research & Design

Safety systems

Safety was paramount: drivers needed total situational awareness.

There was always a safety driver in the truck ready to intervene, but the system was not designed for quick-response issues like signal loss.

My first software update added a prominent alert and status mode indicator to the control station and safety driver HUDs. A “no signal” warning appeared at both ends if the signal dropped for more than one second. Status mode indicated which driving system was currently engaged: teleoperation (standby/active), autonomous mode, or the safety driver. I updated the autonomous box LED lights to match green/red for positive/negative states.

Later, we doubled the cellular signal so it transmitted concurrently through two service providers for redundancy and reliability in poor service areas.

Microphones, speakers, and headsets were installed so drivers could talk directly. We tested and established verbal handoff protocols. Drivers were required to confirm "I have control", ”you have control" during every handoff between the safety driver and teleoperator. This was reinforced with safety drills at the start of every training session.

training diagrams
training-diagrams

Early training exercise diagrams

UX Research

Driver training

New drivers started with a video game.

I installed American Truck Simulator on the station so drivers had a fun and easy way to get familiar with the control station.

Once they were confident, remote-driving training began with simple figure-eights and handoffs on a closed lot in Alameda. Next, we addressed “stuff you can’t feel because you’re not in the truck” like stopping, controlled turns, and staying in a lane.

The next level was operating a truck in low-traffic areas so they could get reacquainted with intersections, cross-traffic, and driving with a trailer (empty/full). After that was driving on the highway, then off-ramps, on-ramps, and eventually street traffic.

The last step was practicing the switch between teleoperation and autonomous modes and then managing the autonomous system from the control station.

teleop-view

Remote driver waiting to make a left turn in a rest stop in Ft. Lauderdale

Product Design

Designing for scale

The prototype software was a single interface with admin and driver controls all in the same layout. I moved the admin controls to a separate (fourth) screen so the driver could focus on video and telemetry.

Admin controls connected the station to a truck, engaged the teleoperation and autonomous systems, adjusted video-matrix warp settings, tracked truck locations, and measured cell signal quality.

In August, we set up a second teleoperation center in Florida. Part of this upgrade required safety mechanisms to prevent conflicts between the two stations, like taking over the wrong truck while it’s already remote driving.

Delivery

A successful demo

On September 22nd, 2017, Starsky Robotics completed the world's first end-to-end, autonomous trucking run.

no hands

The entire team traveled to Florida and spent a week doing practice runs and finalizing the system. It was long hours and little sleep, but we pulled it off.

hands-up
Starsky Teleop

The Long Haul

A film crew documented the experience to produce a promotional video.

(Teleops starts at 2:00)


Addendum

Product management

I was responsible for delivering the teleoperations program.

In addition to my design responsibilities, I negotiated for resources, planned team schedules, set roadmaps, evaluated drivers, and tracked metrics to ensure we could achieve the MVP goal in a limited time.

We were rotating in different drivers every week, so they were only getting 2-8 hours of practice per month. I pitched that we focus on a single remote driver and safety driver (with backups) and then evaluated drivers and helped select the final candidates.

The autonomous team controlled access to trucks and drivers. I had to negotiate for equal access to resources between our projects by pointing out expenses and setbacks caused by irregular planning. I proposed (and we implemented) a new schedule that overlapped between California and Florida that allowed 4x more test sessions per day.

The founders predicted we would solve “supervised autonomy” in six months and have 10k trucks running within a year. To help temper their expectations, I produced a roadmap of automotive system integration requirements and milestones we would need to achieve a fully autonomous system.

Selected works

Time journalProductivity app for iPhone, iPad, & macOS

OmnicellDesign library & interactive kiosks

Star Wars APIAndroid design & code exercise

Exercise goalsProduct design exercise

Starsky RoboticsUX research & design for autonomous trucks

Disney Movies AnywhereSenior Android Developer, Product Designer

DayframeAutomated photo frame for social media

HD WidgetsRanked #1 paid app in 2011

App StatsPlay Store tracking and updates

CloudskipperMusic player

Flash apps2000 - 2011

Elemental design1994 - 2003

lion-nav-dark

@radley  •  Twitch  •  Mixcloud  •  Github  •  LinkedIn