Autonomous Vehicles - How Self-Driving Cars Think

A human driver glances at the road, checks mirrors, listens for sirens, and makes dozens of micro-decisions every second - most of them unconsciously. Replicating this in a machine is one of the hardest problems in AI. Self-driving cars must perceive a chaotic world, predict what every other road user will do, and act safely - all in real time.

SAE Levels of Automation

The Society of Automotive Engineers (SAE) defines six levels of driving automation:

| Level | Name | Description | |-------|------|-------------| | 0 | No Automation | Human does everything | | 1 | Driver Assistance | One automated function (e.g., adaptive cruise control) | | 2 | Partial Automation | Steering and acceleration automated; human monitors at all times | | 3 | Conditional Automation | Car handles most driving; human must take over when requested | | 4 | High Automation | Car drives itself in defined conditions; no human intervention needed | | 5 | Full Automation | Car drives anywhere, any conditions, no steering wheel required |

Most commercially available systems today - including Tesla Autopilot and GM Super Cruise - operate at Level 2. Waymo's robotaxis operate at Level 4 within geofenced areas. No production vehicle has achieved Level 5.

Sensor Fusion - Seeing the World in Layers

Self-driving cars do not rely on a single sensor. They combine multiple types to build a rich, redundant picture of the environment:

• Cameras - Provide colour, texture, and high resolution. Excel at reading signs, detecting lane markings, and recognising traffic lights. Struggle in poor lighting and heavy rain.
• LiDAR (Light Detection and Ranging) - Fires laser pulses and measures reflections to create precise 3D point clouds. Excellent for measuring distance and detecting obstacles. Expensive and affected by heavy weather.
• Radar - Uses radio waves to detect objects and measure speed. Works in all weather conditions. Lower resolution than LiDAR but highly reliable.
• Ultrasonic sensors - Short-range sensors used for parking and low-speed manoeuvres.

Sensor fusion combines all of these inputs into a single, coherent model of the environment. If one sensor is degraded (e.g., camera blinded by sun), others compensate. The fusion algorithm must handle differences in timing, resolution, and coordinate systems across sensor types - a significant engineering challenge in itself.

The Perception Pipeline

Raw sensor data is useless until the AI converts it into actionable understanding. The perception pipeline typically includes:

1. Object detection - Identifying cars, pedestrians, cyclists, traffic signs, and lane boundaries in camera and LiDAR data.
2. Object tracking - Following each detected object across frames to understand movement trajectories.
3. Classification - Distinguishing a pedestrian from a cyclist, a parked car from a moving one.
4. Prediction - Forecasting where each object will be in the next 1–5 seconds based on its trajectory, speed, and context.

This is where deep learning shines. Convolutional neural networks process camera data; point-cloud networks (like PointNet) process LiDAR data. Transformer-based architectures are increasingly used to fuse information across sensor types. The predictions feed directly into the planning system, and any error in perception cascades through the entire stack - making this stage safety-critical.

HD Maps and Localisation

Most autonomous vehicles rely on high-definition maps - centimetre-accurate 3D maps of roads, lane markings, kerbs, traffic signals, and signs. These maps are built in advance by mapping vehicles and updated continuously.

The self-driving car uses its sensors to localise itself within the HD map - determining its precise position to within a few centimetres. This is far more accurate than GPS alone, which can drift by several metres.

HD maps give the vehicle a massive head start: it already knows the road layout before it starts perceiving the live environment. However, they are expensive to build and maintain, and they cannot account for temporary changes like roadworks. Some companies, notably Tesla, are betting they can eliminate HD map dependency entirely and rely solely on real-time perception.

Motion Planning - Deciding What to Do

Once the car knows where it is and what surrounds it, the motion planner decides how to act. This involves:

• Route planning - The high-level path from A to B.
• Behavioural planning - Deciding manoeuvres: change lanes, yield at a roundabout, stop for a pedestrian.
• Trajectory planning - Computing the exact path and speed profile, smooth and comfortable, that executes the chosen behaviour.

The planner must balance safety, efficiency, comfort, and traffic rules - often under tight time constraints. It generates plans many times per second, adjusting as the world changes. Passengers should feel the ride is smooth and predictable, even when the vehicle is making complex manoeuvres.

The Trolley Problem in Practice

Philosophers have debated the trolley problem for decades: if a runaway trolley is heading towards five people, should you divert it to a track with one person? Self-driving cars face real versions of these dilemmas.

In practice, engineers focus on collision avoidance rather than choosing whom to harm. The system is designed to minimise risk continuously - maintaining safe distances, planning escape routes, and reacting faster than any human could. Explicit "whom to sacrifice" logic is not programmed, but the underlying ethical questions remain deeply important for regulation and public trust.

Tesla Autopilot vs Waymo vs Cruise

Three approaches illustrate different philosophies:

• Tesla - Relies primarily on cameras (removed radar and ultrasonic sensors in newer models), betting that computer vision alone - trained on billions of miles of fleet data - can achieve full autonomy. Currently Level 2 (Full Self-Driving Beta requires constant human supervision).
• Waymo (Alphabet) - Uses a full sensor suite (LiDAR, cameras, radar) and operates Level 4 robotaxis in geofenced cities like Phoenix, San Francisco, and Los Angeles. No human safety driver in many vehicles.
• Cruise (GM) - Pursued a similar sensor-rich approach to Waymo but paused operations in late 2023 after a pedestrian safety incident, highlighting how fragile public trust can be.

Why Full Self-Driving Is So Hard

The gap between Level 2 and Level 5 is enormous. The core challenges:

• Edge cases - Unusual situations (a mattress on the motorway, a child chasing a ball, emergency vehicles) are rare individually but collectively endless. You cannot pre-programme every scenario.
• Weather - Snow covers lane markings, rain distorts camera images, fog limits LiDAR range.
• Human unpredictability - Pedestrians jaywalk. Cyclists weave. Drivers ignore rules. The car must anticipate irrational behaviour.
• Construction and temporary changes - Road layouts change daily. Maps become outdated within hours.
• Regulatory and liability - Who is responsible when an autonomous car causes an accident? Laws vary by country and are still evolving.

The long tail of rare events is the fundamental barrier. A system that handles 99.9% of situations perfectly can still cause serious harm in the remaining 0.1%.

📚 Further Reading

• Waymo Safety Report - Detailed overview of Waymo's approach to autonomous vehicle safety
• SAE J3016 - Levels of Driving Automation - The official SAE standard defining autonomy levels
• How Tesla's FSD Works - AI DRIVR - Community analysis of Tesla's Full Self-Driving system with real-world footage