Plans for 180 homes next to the M20 have been approved despite fears residents will be “very isolated” at the new site.