Having set out along a similar path quite some time ago I really would recommend building an Inglenook.
You only need a couple of turnouts and maybe 6" length in P4, depending on what wagons you want to shunt around and what size of turnouts you choose.
Although an inglenook doesn't need an fiddle yard to work, there's no reason not to have one.
My own P4 inglenook is called Thorburn's and has been an on-going although currently stalled project for a couple of years. Thinking about starting something else soon.
Look forward to seeing what you come up with.
Mike