Formation of the Guardians (The Guardian Series, #1)

A wandering elf finds a castle with an astronomer and pregnant wife and agrees to protect them. It's not until after a nearby village is raided that more people join and are allowed to live in the castle as long as they are willing to work. As the little community grows, they discover they need people with specific skills and they go and recruit them especially if they are oppressed.