website for web crawler

0 RIAS GREMORY · June 30, 2015
Can anyone suggest me a simple website for the web crawler making?

Post a Reply


Oldest  Newest  Rating
0 Tony Stark · July 5, 2015
Try Ebay, Its probably illegal but it should be fine as long as you are only doing it on your computer for practice and not distributing your web crawler. Maybe search for a specific item and web crawl all the results and extract item names and number of bids? You can also try this it should be pretty easy.
0 Otto Von Chesterfield · July 7, 2015
Pretty much any website that doesn't give a 403 Forbidden error should be safe.

I know that whenever Amazon detects somebody using a bot requesting for a page, they send a simpler form of the webpage than what you would get if you go there manually.
  • 1



This section is all about snakes! Just kidding.

Bucky Roberts Administrator