Depends how formal lunch is round there house I guess. If it was my family, I'd probably just wear comfortable jeans and a sweatshirt - probably my navy blue one from The Gap because it's soooo soft and cozy.
If it was like a more traditional lunch or something, then I'd wear a
smart flowing top or blouse with fitted dark jeans or something
If it was super formal thing like someone's christening or something that my parents cared about then I'd consider a dress.