Go for it. Don't be afraid to mix and match.
Having different scale/gauge layouts is something I'm hoping to achieve. I'm 90% of the way there with my first layout (in N), but want to do some stuff in 4mm scale as I keep going to shows and being amazed at how big everything looks in 4mm.