I decided to remove all the relations and "outline" members, only keep the building parts, and manually change them to building=*. Seems like this is the easiest way...

should be all good now with changeset #186767285

There are still issues left. I will try to fix them later.