From “find a chair” to the CAN bus: KTH students teach the Rover to navigate by language

The brief: open-world navigation on a real car

In DD2414, small teams plan and deliver a 15-credit engineering project, about 400 hours of work per student, and finish with a working system. This team chose zero-shot semantic navigation: give the car a goal in natural language and let vision-language models work out where to drive.

Their specification starts from a real gap. Today’s autonomous scale cars mostly navigate with geometric lidar SLAM. They know where the walls are, but not what anything in the room is. The team set out to add that layer of meaning and committed to three deliverables:

• a ROS 2 pipeline that connects vision-language models to the car’s control system
• a physical demonstration of the Rover driving to targets named in plain language
• an evaluation of success rate, latency and how much performance drops between simulation and the real world

Each student owns a subsystem. Kerun leads perception and the vision-language models. Dehan leads the ROS 2 architecture and the maths that turns pixels into 3D goals. Xianhan leads control and the move from simulation to the physical car. The spec also requires hardware adapters that turn abstract velocity commands into real CAN bus signals, and that’s where the Rover comes in.

What they built: a navigation stack that respects the car

The team made a deliberate call early on: make the car drive reliably first, then add the semantics on top. A language model that picks the perfect target is no use if the car can’t get there safely.

The result is a ROS 2 pipeline where every stage can be traced from sensor to wheel:

• Sensing. Wheel speeds come straight off the Rover’s CAN bus. The team added a ZED stereo camera, an Xsens IMU and an RPLIDAR.
• State estimation. An extended Kalman filter fuses wheel speed, camera odometry and IMU yaw rate at 30 Hz. slam_toolbox builds the map from lidar scans and removes long-term drift.
• Planning. Nav2 runs a Hybrid-A* planner with Reeds–Shepp curves, so paths respect the car’s 3.35 m minimum turning radius and can include reversing. Regulated Pure Pursuit tracks the path and a collision monitor looks 1.2 s ahead.
• Actuation. A custom motion controller converts velocity commands into throttle and steering, using feedforward and PI control, battery voltage compensation and a measured steering map. The commands then go to the Rover’s ROS 2 gateway, which puts them on the CAN bus.
• Supervision. Goal gating, session management, a geofence, sensor health checks and RC takeover sit over everything. If something fails, the goal is cancelled and the controller is disabled.

Safety is also layered in time. A watchdog in front of the gateway zeroes the output if commands stop for 120 ms. A 250 ms gap in velocity commands stops the controller, and 300 ms without wheel feedback disables closed-loop control.

Seeing with words

On the perception side, the ZED camera feeds two detectors. YOLOv7 quickly handles common objects such as people and office chairs. Grounding DINO takes free-text queries like “fire extinguisher” or “keyboard”. MobileSAM then cuts each object out of the background, so only its own depth pixels are used. That gives a robust position relative to the camera: so many metres ahead, so many to the left or right.

A target is only published once it clears a confidence threshold and its depth stays stable across frames. As the students put it, a false target here becomes an unsafe navigation goal.

Results so far, and what simulation doesn’t teach you

The headline result is a recorded real-world test from 8 October: four navigation goals, four arrivals. The Rover built its map online, planned with Hybrid-A* and tracked its paths with Pure Pursuit, all on its own.

In a live lab scene, the vision pipeline found an office chair (YOLOv7, confidence 0.93) and a fire extinguisher (Grounding DINO, 0.91), and gave the position of each relative to the camera. Across their runs, office chairs and shoes were the most reliable targets.

The students are open about what doesn’t work yet. Desktop monitors produced both missed detections and false positives, and backpacks were only reliable when upright and fully visible. They concluded that one global confidence threshold isn’t enough and that each target class needs its own.

The most useful part of the report may be the section on what went wrong. These are problems that only show up on real hardware:

• Low-speed control. At low throttle, friction and the speed controller’s nonlinear response made the car stop, surge forward, then stop again. The team measured the throttle-to-speed curve and built a closed-loop controller around it. Slow commands from Nav2 now become smooth motion.
• Sensor faults. Blocking the camera’s view caused a 33 cm jump in visual odometry, and the camera’s gyro drifted by about 4° per minute. The fix was to fuse only what each sensor does best: forward speed from the camera, yaw rate from the IMU, and an outlier gate in front of the filter.
• Planning for a big car. Periodic replanning made the Rover change direction 28 times on the way to a goal just 1 m to the side. A lattice planner with only 16 headings failed 5 of 12 simple real-world goals. Replanning only when needed and raising the heading resolution fixed both.

Simulators rarely force you to solve problems like these, and this is the experience that carries over to industry.

Why the Rover

This project shows what the CanEduDev Rover was built for: real vehicle engineering at lab scale.

• Real car dynamics. Ackermann steering, a motor driven by an electronic speed controller, and a 0.73 × 0.55 m footprint. Students face the same constraints as full-size autonomous vehicles: turning radius, multi-point turns and reversing.
• A real CAN bus. Every module on the Rover, from the wheel speed sensors to the steering servo, talks over CAN. Students can read wheel speeds for odometry and send drive commands over the same network the vehicle itself uses.
• ROS 2 from day one. The Rover’s ROS 2 gateway accepts standard velocity commands and publishes sensor data as ROS 2 topics. This team connected their own controller and safety supervisor to it instead of writing a CAN driver from scratch.
• Safety built in. Whenever the physical radio is active, it overrides autonomous control, so a supervisor can always take over. The students built their RC takeover on top of that.
• Room to grow. The chassis carries a stereo camera, an IMU, a lidar and an onboard computer alongside the stock electronics. Because the firmware and tools are open source, the team could dig as deep as they needed.

What’s next

The second half of the semester is about closing the loop from language to motion. The team plans to:

• integrate VLFM (Vision-Language Frontier Maps), so the Rover can search for a named object it can’t see yet
• add a goal manager that turns a detection into an approach pose the car can actually reach, at a sensible distance from the target
• use the ZED’s depth data to catch low obstacles that sit below the lidar’s scan plane
• measure success rate, path efficiency and safety events across different rooms and starting positions

The final demonstration is in December, and we’ll share the results here when it’s done.

Teaching robotics, autonomous vehicles or embedded systems? The CanEduDev Rover gives students a real vehicle with a CAN bus, open-source firmware and ROS 2 support from the first lab. Get in touch or browse the documentation at docs.canedudev.com.