My role

I was responsible for delivering the remote driving portion of the unmanned run. Although my title was Senior UX Designer, my work was primarily product management.

My approach

I used a methodology I call Just in Time (JIT) Development, which is a variantion of Lean UX cycles. UX design is prioritized and visual design is minimized so features can be built, released, and validated in quick cycles.

The challenge

When I started in June, they already had a basic remote-driving prototype. The system was difficult to operate so they could only drive a few miles per hour on a straight road.

The challenge: our founders wanted to be able to demonstrate a fully autonomous end-to-end run by end of summer.

The control station was truck steering wheel attached to a racing game harness.

The environmental view was a small and distorted live video feed.

The station was located in the middle of an busy work area. Communication was relayed through engineers on cell phones.

Phase 1: hardware upgrade

My first step was to upgrade the station. For ideas, I looked for the next best thing: truck driving simulators.

L3 Training Solutions' simulator

Hardware upgrade

Hardware solutions

The prototype control station sat in the middle of a busy workspace.

We moved it to an isolated room that was large enough to host a full size control station and allow room for a UX workstation and observers.

The prototype was cramped and uncomfortable.

We upgraded to 50" HDTV screens with rolling stands and replaced the racing harness with an ergononic seat and a steering wheel mount. Drivers could adjust all hardware as needed.

The live video was small and distorted.

The large HDTV screens increased the size of the view to match a tractor windshield. To fix the video distortion, we applied a matrix algorithm to warp the video back to natural proportions. We also experimented with various near-zero latency video solutions.

Phase 2: safety & communication

All of our drivers had five or more years of driving experience with zero traffic incidents on their records, yet remote driving was new for everyone. The most pressing issue was addressing safety and communication issues. 

For example, one problem that persisted during the prototype tests was that the remote driver and safety driver could not talk to each other directly — communication was relayed between engineers on cell phones at each end.

Another issue was poorly designed status indicators. The safety driver and engineer in the truck knew if the system was engaged, but the remote driver could only see vehicle speed and steering wheel rotation. And nobody could immediately tell when there was signal loss. This often lead to confusion over who was controlling the vehicle. Who's driving? Is it working right now?

Safety & communication solutions

The remote driver and safety driver could not talk directly.

Microphones and speakers were installed in the control room and tractor cab so teams at each end could communicate directly. Later on, we used wireless headsets so the drivers could talk direct.

Who's driving?

We established communication protocols, such as requiring the drivers to verbally confirm who was driving during each hand-off ("You have control", "I have control"). We ran drills at the start of each training session to reinforce these protocols.

Is it working?

Prominent warning alerts were added to the truck and control station HUDs. We ran safety drills at the start of each training session so drivers could be confortable during technical issues.

What mode is it in? Is the tractor doing what I expect?

System mode indicators were added to the truck and control station HUDs. We added various control gauges and telemetry to the control station HUD.

Phase 3: driver training

Now that drivers could see what they're doing and can handle problems when they occur, it was time to develop their remote driving skills. Again, this was new ground for everyone, but drivers did have their own training rituals, so we adapted what they had and made up new ones for any they didn't.

Training solutions

The control station is strange for new drivers.

I installed a truck driving simulation game on the control station and did my best to match controls between the systems.

Where are the most basic skill requirements?

We started off on a closed lot with exercises for turns, acceleration, deceleration, stop distance, hand-offs, and safety training.

How do we transition from a closed lot to real world traffic?

Once drivers were comfortable with basic skills, we'd allow them to drive on small side streets to practice lane-keeping, intersections, turns, and driving in low traffic.

How do we advance quickly enough to meet our deadline?

I recommended that we focus on a single driver. I evaluated and selected a finalist based on their driving skill and confidence.

Phase 4: expansion

Now that we had a dedicated remote driver, it was time to scale up capabilities and infrastructure.

A big part of Starsky Robotics' operation strategy is supervised autonomy. The remote driver would be fully responsible for the journey, including all of the autonomous run. The safety driver could not intervene. We needed additional remote control features.

We also needed a teleoperation room in Florida. This would offer us more, local training time and have a station in place for the big run.

Phase 4 solutions

The big run is in Florida.

We set-up a second teleoperation center in Florida.

How do we manage multiple command centers?

We redesinged and expanded the system to connect any vehicle to any control station. Safety features were added to prevent conflicts.

What do we need for autonomy?

Additional functionality was added so the remote driver could toggle between teleop and autonomous modes and handle autonomous driving requirements.

How do we scale into heavier traffic scenarios?

We scouted and utlilized more training areas that allowed drivers to advance to medium traffic, heavy traffic, and highways.

A shift in perspective

This is what happened when we eventually changed the video camera position.

During most of our development, the video camera was centered and above the safety driver's head, and then pointed it down to see out the windshield. It was assumed this was the best position because it was lined up with the driver's point of view.

Unfortunately, it wasn't obvious that this downward angle was causing a problem. High angular velocity perception is an optical illusion that made it look like the vehicle was travelling much faster than it actually was.

This meant that remote drivers were routinely driving 30-50% slower than they thought they were.

I asked an engineer go out to one of the tractors, unmount the camera, and move it around while we watched from the control station. We quickly found a better position: next to the driver (instead of above) and the result was striking:

The hardware engineers modified the mount in all the tractors. Angular velocity perception was eliminated and remote driving suddenly felt normal. The average speed for remote driving went up to match regular driving.  

Final phase: the big run

Once we were confident we could complete the challenge, the entire Starsky team travelled to Florida and made the run.

In 2018, Starsky released The Long Haul, documenting part of our success.

Bonus design

I did a truck wrap...