The size of your garden will be determined by how many mouths you intend to feed at any one time. It's reccomended that each person you intend to feed be allocated between 5 and 20 full grown plants each year. This of course depends on how much of the crop you intend to use, and how large their harvests are.
It's a good idea to plant a couple extra plants than you'd expect initially. This leaves room for error, and extras can always be sold, turned into other products, or saved for later.
