I've bought cardmaking magazines when they have 3 in a pack for the same price and use them for inspiration - I keep coming back to them for ideas because they're pretty current, so they've been worth it. I've got books from the library for techniques, although I don't do anything fancy anyway. The books tend to be dated for designs, but the techniques are universal. And the internet is great for ideas.
I tend to go Spotlight, KMart and cheapie shops for bits and pieces. It's easy to buy on a budget but still look like you've spent a lot if you choose well. I don't do the proper scrapbooking shops as I can't justify the cost. I try to buy thinking of how many different uses I can have for a product, rather than for a single project.
I buy hole punches when they're on special at spotlight, but make sure I chose very versatile ones. I've bought a few stamps recently, but again, make sure I can get a lot of use out of them.
I print sentiments onto nice paper instead of using rub ons. You can download some great fonts for free.
And I often re purpose cards we've received, cutting elements and saving them for future projects.
Essentials - cardstock, papers, a paper cutter, or cutting mat, metal rule and craft knife. Double sided tape.
In my boxes I've also got different colours and shaped brads, beads, glittery threads, foam shapes, 3D embellishments, stars, rub ons, stamps, inks, glitter, ribbons, tape, 3D tape, glue spots, edgers, scissors that cut different shapes, stickers.........I'm sure there's heaps more, but I can't think right now.
I'm self taught, but I've always been creative, so it's a natural progression.
Have fun and just play around. Take photo's of every card, and you'll soon see how you're developing.