Web scraping is a prevalent technique to accumulate large amounts of data from publicly available websites.<\/p>\n\n\n\n
There are various techniques to do this. Yet, People use a few methods a lot more than the others.<\/p>\n\n\n\n
Many people use Selenium to navigate programmatically through web pages and pull data from them. Selenium is helpful, especially for scraping websites with lots of dynamically loaded components. Because these components load only after the page has loaded its static features, other techniques often fail to fetch them.<\/p>\n\n\n\n
But, programming a Selenium driver could be overkill for other websites. If the website renders data as static pages, you have more accessible ways to fetch tons of data in no time.<\/p>\n\n\n\n
We will use these techniques first and discuss ways to speed up selenium scripts.<\/p>\n\n\n\n
Before we move on, here’s something you should be aware of. Not all websites welcome bot behaviors. You can avoid legal issues by reaching out to the site administrators before you do any scraping.<\/p>\n\n\n\n
A quick check is to study the site’s `\/robots.txt` file. If you see something like the following, you could say for sure that you CANT scrape it.<\/p>\n\n\n\n