My naive thought was, many roads are clear and self-similar, how hard could it be to write an algorithm which simply walks along a step at a time, moving in the direction which is most similar to the previous spot in the image?
It turns out the catch is in "similar". There are apparently countless academic papers on how to evaluate when two images are "similar". I naively went ahead and tried a dumb algorithm: the summed difference of the RGB values.
Amazingly, it actually works in a lot of cases. Behold:
The first two points are given, the rest moving downward follow the road based on naive image similarly. Now, it's not hard to find cases where it fails and drifts off the road - in particular it struggles if the road gets a few pixels wider, as many do - but this is just a first test.