Autonomous vehicles are hard. The hardest part is creating an AI that can react naturally to the environment. Starsky Robotics was working on an autonomous solution with a clever twist. They would use a remote-driver for anything the AI couldn't handle.
I was responsible for making the teleoperation program succeed. I started with a rough prototype that could drive 10 mph on a straight road. When I finished, we had multiple control stations that could drive trucks in live traffic.
The demo would be a simulation of a gate-to-gate trucking run. We could only use teleoperation and the autonomous system. The safety driver could not intervene.
I was the sole Senior UX Designer. I was responsible for managing and delivering the teleoperation program.
I was the only person working full time on teleoperation. I worked directly with the lead software developer and a half-dozen drivers. I coordinated with the founders and autonomous team for access and resources.
The system was a modified racing-game station with a truck steering wheel.
The HUD displayed video streamed from a 180° camera mounted in the truck. At the bottom of the screen are indicators for speed and steering-wheel rotation.
Driving tests took place in a busy open-office environment. The remote driver and safety driver in the truck could not talk directly. All communication was relayed by engineers on cell phones.
UX & Product Design
Teleoperation control station in San Francisco
I relocated the control station to a private room. I upgraded all the equipment to make the driving experience more true to life. Microphones and speakers were installed so the drivers could talk directly.
Poor video quality was an ongoing challenge that required clever solutions. We split the cell signal between services to reduce dropouts. We applied a matrix algorithm to warp the video fit the screens. I repositioned the camera in the truck to reduce angular velocity misperception. It looked like they were driving much faster than they actually were.
UX Research & Design
Teleoperation control station HUD
Safety driver HUD
The drivers needed complete situational awareness at all times. Not only what the truck was doing, but who (or what) was in control of the truck at any given time.
The system was prone to problems like disconnections and signal loss. I immediately added error alerts and system status indicators. They were prominently displayed in the teleoperation control station and safety driver HUD.
We established verbal handoff protocols. Drivers were required to confirm "I have control / you have control" during handoffs. They practiced safety drills at the start of every training session.
Early training exercise diagrams
New drivers started by playing a video game (American Truck Simulator). This enabled them to become comfortable with the control station.
The hands-on training was experimental. We started with baby-steps: slow figure-eights, handoffs, and safety drills. Next, we started to address “stuff you can’t feel because you’re not in the truck” like controlled braking and turns.
Everything from there was a matter of progression. We moved from a closed lot to low-traffic areas and driving with a trailer. After that was driving on the highway, on-ramps, off-ramps, and eventually in street traffic.
Remote driver waiting to make a left turn in a rest stop in Ft. Lauderdale
The initial prototype was a one-to-one system with one truck linked to the station. An engineer had to update the code to switch to another truck.
I decoupled the teleoperation system into separate master and control stations. The master station could track and assign trucks. Tracking included location, speed, fuel levels, and cell strength for all trucks. The control station focused on driving systems like video and indicators.
A month before the demo, I opened a new teleoperation center, this time in Florida. The second location required new safety checks for conflict prevention between master stations.
Activity JournalProductivity app for iPhone, iPad, & macOS
OmnicellDesign component library, responsive web design, & interactive kiosks
Starsky RoboticsUX research & design for a remote-driving system
Star Wars APIAndroid design & code exercise
Exercise GoalsProduct design exercise
Disney Movies AnywhereSenior Android Developer, Product Designer
DayframeAutomated photo frame for social media
HD WidgetsPopular widget app
App StatsPlay Store app tracking & metrics
CloudskipperFree music player
Flash / Flex2000 - 2011
Elemental Design1994 - 2003