website for web crawler

0 RIAS GREMORY · June 30, 2015
Can anyone suggest me a simple website for the web crawler making?

Post a Reply

Replies

Oldest  Newest  Rating
0 Tony Stark · July 5, 2015
Try Ebay, Its probably illegal but it should be fine as long as you are only doing it on your computer for practice and not distributing your web crawler. Maybe search for a specific item and web crawl all the results and extract item names and number of bids? You can also try this http://feeds.bbci.co.uk/news/rss.xml?edition=uk it should be pretty easy.
0 Otto Von Chesterfield · July 7, 2015
Pretty much any website that doesn't give a 403 Forbidden error should be safe.

I know that whenever Amazon detects somebody using a bot requesting for a page, they send a simpler form of the webpage than what you would get if you go there manually.
  • 1

Python

107,107 followers
About

This section is all about snakes! Just kidding.

Links
Moderators
Bucky Roberts Administrator