Previous prime ministers were reluctant to move into the building, said to be haunted by the ghosts of men who attempted ...