website for web crawler
Post a Reply
|Oldest Newest Rating|
· July 5, 2015
Try Ebay, Its probably illegal but it should be fine as long as you are only doing it on your computer for practice and not distributing your web crawler. Maybe search for a specific item and web crawl all the results and extract item names and number of bids? You can also try this http://feeds.bbci.co.uk/news/rss.xml?edition=uk it should be pretty easy.
Otto Von Chesterfield
· July 7, 2015
Pretty much any website that doesn't give a 403 Forbidden error should be safe.
I know that whenever Amazon detects somebody using a bot requesting for a page, they send a simpler form of the webpage than what you would get if you go there manually.
This section is all about snakes! Just kidding.
|Bucky Roberts Administrator|