While loop not working in Python2.7

0 Probuddha N · April 28, 2015
I am new at programming and at Python

I have coded this simple scraper with Python 2.7 to fetch all the profile links from this URL 

http://www.therapy-directory.org.uk/search.php?search=Sheffield&services[23]=1&business_type[individual]=1&distance=40&uqs=626693

..page 1 and 2 both.

The code works fine for scraping all the links from page 1, but it fails to fetch links from page 2.

What am I doing wrong here?


Many thanks!

My code is as follows:-

Post a Reply

Replies

Oldest  Newest  Rating
0 Probuddha N · April 28, 2015
Sorry the code tag was truncating most of the codes, so I had to use the Quote tag. Sorry for any inconvenience

import requests
from bs4 import BeautifulSoup

def tru_crawler(max_pages):
    p = '&page='
    page = 1
    while page <= max_pages:
        url = 'http://www.therapy-directory.org.uk/search.php?search=Sheffield&distance=40&services[23]=on&services=23&business_type[individual]=on&uqs=626693' + p + str(page)
        code = requests.get(url)
        text = code.text
        soup = BeautifulSoup(text)
        for link in soup.select('div.member-summary h2 a'):
            href = 'http://www.therapy-directory.org.uk' + link.get('href')
            print(href)
        page += 1

#Run the function
tru_crawler(3)

  • 1

Python

107,130 followers
About

This section is all about snakes! Just kidding.

Links
Moderators
Bucky Roberts Administrator