starsky-logo

I joined Starsky Robotics in June 2017 to design and manage their teleoperations program. In less than 100 days we needed to transform a simple remote driving prototype into a fully operational system.

By September 2017, our remote driver was able to achieve the world's first end-to-end, unmanned run with zero interventions.

Truck Interior

The Challenge

Starsky Robotics is working on an artificial intelligence system for big-rig trucks that run autonomously on highways, while driven from a remote operations center for all other conditions.

The founders had given the team a challenge:

Demonstrate a real-world, unmanned end-to-end run...
in less than 100 days.

Early teleop team

My Role

I was responsible for delivering the remote driving portion of the unmanned run.

I researched and designed the hardware and software interfaces, worked directly with drivers to iterate on the system, developed a training & evaluation program, and managed teleoperations.

Note: due to the company’s confidentiality, many specifics of my work won’t be disclosed.

My Role

As Sr. UX Designer I was primarily responsible for managing the teleoperations program including:

  • Command and control station research and design
  • Driver training & evaluation

Note: due to the company’s confidentiality, specifics of my work won’t be disclosed. I’ve limited this case study to solutions made availalble publicly.

Design Approach

Progress moves fast in a start-up environment. Meetings were done on the spot, changes came rapidly, and there was no pre-existing example for what we were trying to achive.

For these reasons, I applied a lean, iterative design approach to developing the system.

Design Approach

Progress moves fast in a start-up environment. Meetings were done on the spot, changes came rapidly, and there was no pre-existing example for what we were trying to achive.

For these reasons, I applied a lean, iterative design approach to developing the system.

The Prototype

 When I first joined Starsky Robotics, I observed a couple of sessions with the remote driving prototype. It was challenging and awkward to operate, so the driver couldn't drive much faster than a few mph in a straight line.

teleop-prototype-station

The control system was cramped and uncomfortable.

teleop-prototype-screen

The video was small and distorted.

teleop-prototype-team

The prototype resided in the center of a distracting and active work area.

Phase 1: The Begining

I started off by asking "who's done this before?" I eventually discovered truck driving simulators made by L3 Training Solutions. This gave me a model for screen size, distance, angles, seating, controls, and the HUD.

Truck Driving Simulator
Truck Driving Simulator Seat
Truck Driving Simulator – Dashboard

Next I started diagramming potential layouts for our control system. I wanted to go with a modular system, so we could experiment and optimize for screen and seating postions.

Starting with 50" HDTVs, I measured and mapped out the space required for the equipment, determining how a driver might fit within the available view and how much total space would be required for everything.

teleop-station-tables
teleop-station-stands

Problems and Solutions

The prototype control station sits in the middle of a busy workspace.

We moved it to an isolated room that was lage enough to host a larger control station and observers.

The prototype control station was cramped and uncomfortable.

The new station was larger, modular, and adjustable to provide for optimal ergonomics.

The live video was small and distorted.

We used large HDTV screens that matched the view size from a truck cab. To fix the distortion, we used a matrix algorithm to warp the video back to natural proportions. We also experimented with different zero-to-low latency video settings.

HUD closeup

Phase 2: Safety and Communication

The next most critical issue was addressing safety and communication issues. All of our drivers had 5 years of driving experience with zero traffic incidents on their records, however remote driving was new to everyone involved and required additional safety checks in place.

One key issue during the prototype tests was that communication between the remote station and the truck was relayed between engineers on cell phones at each end. The drivers in control of the vehicle had no direct line of communication between each other.

This often lead to confusion over who was operating the vehicle and whether or not it was in the correct mode of operation. Further, the prototype station's HUD only displayed minimal telemetry. The remote driver would need much more information to drive effectively and safely.

Problems and Solutions

The remote driver and safety driver could not talk directly.

Microphones and speakers were installed so teams at each end could communicate directly. We also provided wireless headsets (with chargers) for the drivers for direct communication.

Who's controlling the tractor?

We established communication protocols requiring the safety driver and remote driver to both verbally acknowledge who was in control during each hand-off. We ran drills at the start of each training session to reinforce these protocols.

How are the system and tractor operating right now?

System mode indicators were added to the truck and control station HUDs. We added more telemetry gauges in the control station HUD.

What if there's a technical problem?

Prominent alerts were added to the truck and control station HUDs. We ran safety drills at the start of each training session.

training diagrams
training-diagrams

Phase 3: Introduction to Driver Training & Evaluation

When I started, drivers helped us improve the control system. Now that we had the essentials in place, it was time to start improving the drivers.

Problems and Solutions

The control station is unfamiliar.

I installed a truck driving simulation game on the control station so new drivers could become familiar with the controls. Remote drivers would do virtual ride-alongs to get accustomed to the live video.

Where are the most basic skill requirements?

We started off on a closed lot with exercises for turns, acceleration, decelleration, stop distance, and slolom.

How do we transition from a closed lot to real world traffic?

Once drivers were comfortable with basic skills, we'd allow them to drive on small side streets to practice lane-keeping, intersections, turns, and driving in low traffic.

How do we advance quickly enough to meet our deadline?

I recommended that we focus on a single driver. I evaluated and selected a finalist based on their driving confidence and ability to quickly identify and communicate issues to the safety driver.

teleop-view

Phase 4: Scaling Up

Now that we had a dedicated remote driver, it was time to scale up capabilities and infrastructure.

A big part of Starsky Robotics' operation strategy is supervised autonomy. The remote driver would be fully responsible for the journey, including all of the autonomous run. The safety driver could not intervene. We needed additional remote control features.

We also needed a teleoperation room in Florida. This would offer us more, local training time and have a station in place for the big run

Problems and Solutions

How do we optimize for Florida?

We set-up a duplicate teleoperation center in Florida. Drivers received more training sessions. Alternates began training.

How do we manage multiple command centers?

We expanded the command station system to allow for connecting any vehicle to any control station. Safety features were added to prevent accidental takeovers between centers.

What additional features do we need for complete autonomy?

Specific functionality was added to the system and new controls were mapped to the gaming wheel. We iterated on these new features.

How do we scale into more complex traffic scenarios?

We scouted and mapped out more training areas that allowed us to advance to medium traffic, heavy traffic, and highways.

Small Change = Big Result

Often a seemingly insignificant design change can have a huge impact. This is what happened when we repositioned a video camera a few inches.

teleop-screen

From the beginning, Starsky assumed that the remote driver would need to be able to see from the same position as they normally drive the truck. They mounted the video camera centered and above the safety driver's head, and then pointed it down to see out the windshield.

Unfortunately, it wasn't obvious that this downward angle was causing a problem: high angular velocity perception - an optical illusion that made it look like the truck was travelling much faster than it actually was. This meant that drivers were routinely driving 30-50% slower than normal traffic flow.

I had a hunch something like this might be happening, so I asked an engineer go out to a truck, unmount the camera, and move it around while we watched from the control station. We easily found a better position next to the driver, instead of above. The engineers updated the mount and reinstalled the camera.

the-road

The result was striking. Angular velocity perception was reduced and remote driving now felt very similar to regular driving. Drivers were much more confident and average remote driving speeds soon matched normal traffic flow.

Starsky Teleop
hands-up
no hands

Final Phase: The Big Run

We worked up until the last minute making incremental changes to the control hardware and software.

Once we were confident we could complete the challenge, the entire Starsky team travelled to Florida and produced the final unmanned, end-to-end run with no interventions.

In 2018, Starsky released The Long Haul, documenting our success:

Bonus Design

Starsky Truck

I did a truck wrap!

DayframeProject type

Starsky RoboticsProject type

HD WidgetsProject type

App StatsProject type

CloudskipperProject type

Flash Design & AppsProject type

Elemental DesignProject type