Web Crawler question my code look just like the example except using the new sites url

0 joe c · September 10, 2015

I am attempting to complete the tutorial on creating the web-crawler and
I am getting the page variable is not used error message and page not defined error message when debugging..
Anyone else getting these and know of any ideas to resolve?

My code looks just like the example except the website name. I'm using Python 3.4

Post a Reply


Oldest  Newest  Rating
0 Halcyon Abraham Ramirez · September 11, 2015
code man. give us your code
0 joe c · September 15, 2015
import requests
from bs4 import BeautifulSoup

def trade_spider(max_pages):
page = 1

while page <= max_pages:
url = 'https://www.thenewboston.com/forum/recent_activity.php?page=' + str(page)
source_code = requests.get(url)
plain_text = source_code.text
soup = BeautifulSoup(plain_text)
for link in soup.findAll('a',{'class':item-name}):
href = link.get('href')
page += 1

0 James Kon · September 15, 2015
It's most likely because you forgot to make sure the end of your URL in python ends with page=
Nothing else in the URL should be after the page=
0 Halcyon Abraham Ramirez · September 15, 2015
can you show the exact stack trace? like the error msg that you get?

everything looks fine
  • 1



This section is all about snakes! Just kidding.

Bucky Roberts Administrator