In order to keep my road follower in the middle of the road, I tried switching from an incremental similarity (compare each point to the next) to absolute (compare each point to the starting point). Since the starting point is given in the middle of road, it happily follows the road center, until this happens: With incremental similarity, we were largely immune to disruptions along the side of the road, because we came upon them gradually. Now, a large shadow is sufficiently unlike our starting point that it scares the algorithm into swerving away from the shadow and running off the road. (I can sympathize with the algorithm. I did the same thing in a car once :-)

So, it just solves one problem, and exposes another.

I also tried the idea of, each step, taking a cross-section and look for symmetry to find where the “middle” of the road is. It didn’t work; the RGB is just too noisy to find a clear center of symmetry.

Perhaps next I will try, like Richard said, a flood-fill. But rather than try to flood-fill the entire road network, just a local fill to find an approximate road extent. That might work, although there are plenty of examples where it definitely won’t, like where the road is surrounded by similarly colored pixels: Here is an example of a road which my road follower has no trouble with (following, but not staying in the middle of). Attempting to flood-fill it is extremely sensitive to initial point; picking just the right point is OK but any other (or any looser tolerance) will fail in countless ways, including fill way off the road, or just getting one side of it (as in the image above), or filling everything except the middle.