Using BeautifulSoup to scrape your competitors site.

Web scraping is the act of extracting data from websites. You will want to scrape a website to get information directly from the website.

For the purpose of this tutorial, we will be scraping some hotel booking websites to get the title tags and meta descriptions. We will be using BeautifulSoup, a python library that makes it easy to scrape data from a site. You can read the official documentation here .

You can install BeautifulSoup using pip

pip install BeautifulSoup4

Now that we have successfully installed BeautifulSoup, we can proceed to scraping the hotel booking websites. We will be scraping the following sites: hotels.com, tripadvisor.com, booking.com and travel.jumia.com

In the first and second line, we import the BeautifulSoup and request libraries respectively. We will be using the request library to get the html content before proceeding to scrape this html content with BeautifulSoup.

Next up, we will create two empty dictionaries one for the title and the meta description. We are using a dictionary so we can pass both the title (or meta description) and the site .

Now that we have our empty dictionaries, we can then create a list of sites that we want to scrape. In our case these are: booking, jumia, hotels and tripadvisor.

Once this is done we will write a for statement to loop through the list and fetch the html of each url in our list. We will use the request library to get the html content of the url. When we have done this, we can then use BeautifulSoup to scrape the websites.

You can check out the github gist for the rest of the code.

I hope you found this article useful. You can reach out to me on twitter . Thanks for reading