The potting soil bags at the garden centres have the advisory about not using the soil for edibles. Since a couple of potted daffodils (gifts) did end up being put into a flowerbed, we did use the soil from the pot to keep the plant going as we did not have other soil on hand and then those fertilizer beads in the soil were around to feed the bulb/stalks/flowers.
Best be safe and change the dirt so you can have yummy crops to look at growing and then in the kitchen for making dishes! I know I have to amend the soil around where the mint grows and where there was lavender as I'd like other herbs to join that sunny stretch to grow!