How To Scrape Rentals Websites Using Beautifulsoup And Python?

By Author: 3i Data Scraping
Total Articles: 46
Comment this article

Web scraping using BeautifulSoup and data wrangling using Pandas to discuss generated insights.

Would renting a condo or apartment in Etobicoke, North York, or Mississauga be considerably cheaper than having one in downtown Toronto?

How do suburb's rents compare to the Toronto city’s rents?
How much can you potentially save if you have rented a basement unit?
Which suburbs have the lowest rent rates?
Browsing manually using listings on rental websites can be very time-consuming. So, the better option is to scrape rental websites using web scraping Python as well as analyze that to get answers to all your questions.

Scraping Rental Website Data through Web scraping using BeautifulSoup and Python
Scraping Rental Website Data through Web scraping using BeautifulSoup and Python
We have decided to extract data from TorontoRentals.com with Python and BeautifulSoup. This website has lists for Toronto as well as many suburbs like Brampton, Scarborough, Mississauga, Vaughan, etc. This has various kinds of listings like apartments, houses, condos, as well as basements.

Initially, ...
... we imported the necessary Python libraries.

# Import Python Libraries
# For HTML parsing
from bs4 import BeautifulSoup
# For website connections
import requests
# To prevent overwhelming the server between connections
from time import sleep

# Display the progress bar
from tqdm import tqdm
# For data wrangling
import numpy as np
import pandas as pd
pd.set_option('display.max_columns', 500)
pd.set_option('display.width', 1000)
# For creating plots
import matplotlib.pyplot as plt
import plotly.graph_objects as go
Next, we have written the function named get_page to have soup objects for every page (iteration). Functions accept 4 user inputs — type, city, beds, and page. The function consists of logic for checking HTTP response status codes for finding if HTTP requests have been completed successfully. A get_page function is named from the key function named page_num.

def get_page(city, type, beds, page):

url = f'https://www.torontorentals.com/{city}/{type}?beds={beds}%20&p={page}'
# https://www.torontorentals.com/toronto/condos?beds=1%20&p=2

result = requests.get(url)

# check HTTP response status codes to find if HTTP request has been successfully completed
if result.status_code >= 100 and result.status_code = 200 and result.status_code = 300 and result.status_code = 400 and result.status_code = 500 and result.status_code

3i Data Scraping is an Experienced Web Scraping Services Company in the USA. We are Providing a Complete Range of Web Scraping, Mobile App Scraping, Data Extraction, Data Mining, and Real-Time Data Scraping (API) Services. We have 11+ Years of Experience in Providing Website Data Scraping Solutions to Hundreds of Customers Worldwide.