Human-in-the-Loop: Behavior Cloning for CarRacing

A human-in-the-loop machine-learning project that applies behavior cloning, learning a driving policy directly from human demonstrations, to OpenAI Gym's CarRacing-v2 environment. The central question: how does behavior cloning perform in CarRacing-v2 beyond the tracks it was trained on?

What it is

Behavior cloning (BC) is the simplest form of imitation learning: it reduces control to supervised learning by fitting a policy to a dataset of human state-action pairs. Here a human plays CarRacing-v2 and the recorded frames are paired with the corresponding steering / gas / brake actions, then a neural network learns to imitate that human driver, putting the human directly in the learning loop. The task is a nuanced control problem, so a single expert annotator was used rather than filtering across many drivers.

Method

The environment state is a 96x96x3 pixel top-down view. In the continuous action space the agent controls steering (-1 full left, +1 full right), gas, and braking. Raw frames are pre-processed: converted to grayscale, padded, and cropped to remove irrelevant HUD indicators (true speed, four ABS sensors, steering-wheel position, and gyroscope), then normalized to [0, 1]. A sequential CNN of alternating Conv2d and BatchNorm layers with ELU activations maps each frame to the action set. Because a single frame cannot capture dynamics like speed and rotation, the braking and acceleration behavior is tuned to avoid over-acceleration on straights that would cause understeer in corners.

The reward structure rewards progress: each frame incurs a small penalty of -0.1, and visiting a track tile yields +1000/N points, where N is the total number of tiles. Finishing a lap in fewer frames therefore scores higher, making cumulative reward a direct measure of how human-like and efficient the driving is.

Results

The first experiment measured how the volume of human demonstration data affects performance. Training on 10% / 20% / 50% / 100% of the data (5000 instances at 100%) and evaluating the maximum reward achieved in a single run on a baseline track (seed 4), performance scaled clearly with data: from scoring essentially nothing at 10% to strong driving with the full set. This underscores how heavily behavior cloning depends on comprehensive coverage of the state space.

Percentage of training set	10%	20%	50%	100%
Max reward during 1 run	12	110	357	782

Table 1 from the report: impact of training-data variation, 5000 training instances used for 100%.

Generalization across tracks

The second experiment probed domain adaptation. A model trained on 5 laps of a single track (seed 4) was tested on that track and on unseen tracks. As the figures below show, reward climbs steadily on the trained track (red) but most test laps on different tracks (blue) fail earlier and never complete the circuit, indicating the model learns corners specific to the trained track and does not generalize well. When instead trained on 15000 instances spanning many different tracks, the model performs nearly as well on tracks it has never seen before, showing that broad coverage, not more epochs, is what buys generalization.

Takeaways

Behavior cloning replicates human driving well when the demonstration data covers the state space, but it inherits BC's known weaknesses: distribution shift and compounding errors away from the demonstrated trajectories. Frame-level accuracy also turned out to be an inadequate metric: higher accuracy did not reliably translate into completing more tracks, partly because a human strategy leans on memory of previous frames that a single-frame policy cannot see. The report argues the human element is essential for reliability, safety, and trust in self-driving tasks, and points to active learning on hard track segments and multimodal models as promising next steps.

Report

• Full write-up (PDF): Human-in-the-Loop: Behavior Cloning for CarRacing

Author

Erencan Tatar