If you are a heathen on a budget, there are a lot of things that you can collect for your witches' cabinet for relatively cheaper. This just kind of takes a bit of time and effort on your part.
Now, before anything, here I am going to promote garage sales and thrift stores. There's a lot of GOOD things about them. I mean, waay back in the day when I was young and dinosaurs roamed the earth, everything was about bright, shiny, new and without anyone else's energy to contend with. Too bad it sucks on the pocketbook and is generally badly made (a lot of modern, mass-produced crap). I used to be kind of a snot when it came to things like that as a little kid, and growing older, I realize how much more valuable places like that can be (hey, I can admit my faults!).
Anyway, I digress. Thrift stores and garage sales are the best. You can find all sorts of nifty things there, starting with candle-holders, cups, bowls, etc. I have bought some amazing things at garage sales and thrift stores, including what looks like some sort of wooden finial for a staircase which is over 12 inches tall and a triple goddess (for 1.99....seriously! I'd post the picture, but it's already been boxed for when I move.). I've bought mirrors for spell work and 'witches' bottles', I've even hacked up bedsheets for an assortment of crafts, including altar cloths, robes, and drawstring bags. And when we're talking pennies on several dollars, you're doing good. Jars, buttons, containers, etc. You can find dang near anything your little witchy heart desires. Sometimes it takes longer than others, but eventually it shows up.
Seriously. I mean, I go in for the mindset of offering bowls. I found an Aurthur Court pewter tray that retailed like 40 dollars I used for quite some time on my altar. Hand-thrown bowls is what I look for mainly, and I find quite a few of those too. Small side tables can be refinished or painted for altar space (a friend of mine found a nice six-sided side table he uses because it has some sort of storage compartment under it he keeps all this small altar stuff in). Altar bells. Small boxes for spellwork. All sorts of neat things...just take some time out!
Other than the cheapness of some of the amazing, adventurous things you can find, this is also a 'green' thing. I could be greener. But at the very least, this is one of the tiny green things that I do, because you're recycling STUFF. One person's junk is another person's treasure, that sort of thing.
And you can get newer stuff if you wish being frugal. If you work on some crafts, but aren't generally a crafty person, sometimes you have to have people point you in the general direction of savings. For fabrics in the making of things like poppets or bags, you can actually hit up several of these fabric stores which sale 'remnants'. These are pieces of cloth which are left overs of a bolt of cloth (which is what they usually measure out from), and sometimes they're actually cloth that was cut someone decided at the last minute against buying. Specialty stores, like Hancock Fabrics, will sometimes have special sales where you can get an eight dollar pattern for 99 cents. Bigger places like Michaels, Hobby Lobby, and Garden Ridge Pottery will sometimes sell buttons and ribbons in a bag mixed with a sale tag on it amongst the remnants because, well, they're remnants themselves of packages that have been damaged and the 'leftovers' is what has been salvaged for sale. Goodwill actually does this kind of bagging too of assorted odds and ends.
A word of caution though on candles, especially from thrift stores. When you find those candles with the wicks that seem to have metal in them, be warned. A lot of times, that metal is actually lead-based, so when you burn the candle, you're diffusing a small concentration of lead in the area. So be careful what you pick up along those lines!
So happy hunting to you and show what you find spectacular!