I have a script that hits my website for stock quotes and pulls prices. currently it does this one by one until all the symbols have been pulled. I've solved for bandwidth issues by putting in a random 1-10 second sleep function, but this will take too long.
I know that I can do everything at once by adding & at the end, but that will bomb out my website's resources.
Is there a way to hit my site with a random count of 5-10 symbols pulled all at once, every 1-10 second randomly?
I know that I can do everything at once by adding & at the end, but that will bomb out my website's resources.
Is there a way to hit my site with a random count of 5-10 symbols pulled all at once, every 1-10 second randomly?
Code:
while read line
do
array+=("$line")
done < "../stocks/aaa-stockuniverse.txt"
#this reads all the tickers into an array
for ((i=0; i < ${#array[*]}; i++))
do
eval $(curl -s "http://www.website.com/api?stock=${array[i]}"|sed 's/</\n</g' |sed '/data=/!d; s/ data=/=/g; s/\/>/; /g; s/</GF_/g' |tee ~/stocks/${array[i]}.txt)
echo "${array[i]},$(date +%Y-%m-%d),$GF_open,$GF_high,$GF_low,$GF_last,$GF_volume"
sleep $[ ( $RANDOM % 10 ) + 1 ]s
done