A bid to build four homes in Great Witchingham, near Fakenham, has been refused after villagers objected to the plans.