I'm working on a site that pulls product price data from Amazon.com and Walmart. I'm guessing that in the future, it will also pull data from other places.
My first idea was to pull the data directly from Amazon (using their product advertising API) and then display the data on the site for every single visitor who landed on the page. That's not a bad idea if there aren't many product prices I'll be retrieving (or if the number of site visitors is low). I think that I will run into problems once the site gets busy and if i increase the number of product whose price i want to pull.
Using the Amazon and Walmart API, i was able to make successful REST api calls and parse the XML returned to obtain the information that i needed.
Does it make sense to store that information in a local database, update it say, every 1-5 minutes, and then get the site visitors to pull the pricing information from my local database instead of making an API call to Amazon and Walmart?
If I do go this route and create a function that uses the Amazon and Walmart API to pull price data, how do I then automatically run this function every 1 to 5 minutes in the background, 24/7/365?
Does it make sense to store that information in a local database
Yes. Actually this sounds exactly like a typical caching setup. I would recommend looking into Redis instead of using a relational database for this.
how do I then automatically run this function every 1 to 5 minutes in the background
Probably a Cron job. You would have to provide more information like where your application is running (AWS EC2 or somewhere else?) and if it is running on Linux or Windows, before I could give a more detailed recommendation.