I was searching for flight tickets and noticed that ticket prices fluctuate during the day. I tried to find out when the best time to buy tickets is, but there was nothing on the Web that helped. I built a small program to automatically collect the data from the web — a so-called scraper. It extracted information for my specific flight destination on predetermined dates and notified me when the price got lower.

1. Python Developer Tool

Web scraping is a technique used to extract data from websites through an automated process.I learned a lot from this experience with Web scraping, and I want to share it.This post is intended for people who are interested to know about the common design patterns, pitfalls and rules related to the web scraping. The ariticle presents several use cases and a collection of typical problems, such as how not to be detected, dos and don’ts, and how to speed up (parallelization) your scraper.Everything will be accompanied by python snippets, so that you can start straight away. This document will also go through several useful python packages.

Use CasesThere are many reasons and use cases why you would want to scrape data. Let me list some of them:. to spot if some of the clothes you want to buy got discounted. of several clothes brands by scraping their pages. price of the flight tickets can vary during the day. One could crawl the travel website and get alarmed once the price was lowered.

analyze the action websites to answer the question if starting bid should be low or high to attract more bidders or if the longer auction correlates with a higher end bidTutorialStructure of the tutorial:. Available packages. Basic code. Pitfalls.

Dos and dont’s. Speed up — parallelizationBefore we start: Be NICE to the servers; you DON’T want to crash a website.

Available packages and toolsThere is no universal solution for web scraping because the way data is stored on each website is usually specific to that site. In fact, if you want to scrape the data, you need to understand the website’s structure and either build your own solution or use a highly customizable one.However, you don’t need to reinvent the wheel: there are many packages that do the most work for you. Depending on your programming skills and your intended use case, you might find different packages more or less useful.1.1 Inspect optionMost of the time you will finding yourself inspecting the the website. You can easily do it with an “inspect” of your bowser. The section of the website that holds my name, my avatar and my description is called hero hero-profile u-flexTOP(how interesting that Medium calls its writers ‘heroes’:)). The class that holds my name is called ui-h2 hero-title and the description is contained within the class ui-body hero-description.You can read more about, and differences between and.1.2 ScrapyThere is a stand-alone ready-to-use data extracting framework called. Apart from extracting HTML the package offers lots of functionalities like exporting data in formats, logging etc.

It is also highly customisable: run different spiders on different processes, disable cookies¹ and set download delays². It can also be used to extract data using API. However, the learning curve is not smooth for the new programmers: you need to read tutorials and examples to get started.¹ Some sites use cookies to identify bots.² The website can get overloaded due to a huge amount of crawling requests.For my use case it was too much ‘out of the box’: I just wanted to extract the links from all pages, access each link and extract information out of it.1.3 BeautifulSoup with RequestsBeautifulSoup is a library that allows you to parse the HTML source code in a beautiful way.

Along with it you need a Request library that will fetch the content of the url. However, you should take care of everything else like error handling, how to export data, how to parallelize the web scraper, etc.I chose BeautifulSoup as it would force me to figure out a lot of stuff that Scrapy handles on its own, and hopefully help me learn faster from my mistakes. Basic codeIt’s very straightforward to start scraping a website. Most of the time you will find yourself inspecting of the website to access the classes and IDs you need. Lets say we have a following html structure and we want to extract the mainprice elements. Note: discountedprice element is optional.

WatchPrice: $66.68Discounted price: $46.68Watch2Price: $56.68The basic code would be to import the libraries, do the request, parse the html and then to find the class mainprice. Pitfalls3.1 Check robots.txtThe scraping rules of the websites can be found in the file. You can find it by writing robots.txt after the main domain, e.g.

Python Developer Tool

These rules identify which parts of the websites are not allowed to be automatically extracted or how frequently a bot is allowed to request a page. Most people don’t care about it, but try to be respectful and at least look at the rules even if you don’t plan to follow them.3.2 HTML can be evilHTML tags can contain id, class or both. HTML id specifies a unique id and HTML class is non-unique.

Changes in the class name or element could either break your code or deliver wrong results.There are two ways to avoid it or at least to be alerted about it:. Use specific id rather than class since it is less likely to be changed. Check if the element returns None. However, because some fields can be optional (like discountedprice in our HTML example), corresponding elements would not appear on each listing. In this case you can count the percentage of how many times this specific element returned None to the number of listings. If it is 100%, you might want to check if the element name was changed.3.3 User agent spoofingEverytime you visit a website, it gets your via. Some websites won’t show you any content unless you provide a user agent.

Also, some sites offer different content to different browsers. Websites do not want to block genuine users but you would look suspicious if you send 200 requests/second with the same user agent. A way out might be either to generate (almost) random user agent or to set one yourself. By using a shared proxy, the website will see the IP address of the proxy server and not yours. A VPN connects you to another network and the IP address of the VPN provider will be sent to the website.3.7 HoneypotsHoneypots are means to detect or scrapers.These can be ‘hidden’ links that are not visible to the users but can be extracted by scrapers/spiders. Such links will have a CSS style set to display:none, they can be blended by having the color of the background, or even be moved off of the visible area of the page.

Once your crawler visits such a link, your IP address can be flagged for further investigation, or even be instantly blocked.Another way to spot crawlers is to add links with infinitely deep directory trees. Then one would need to limit the number of retrieved pages or limit the traversal depth. Dos and Don’ts. Before scraping, check if there is a public API available. Public APIs provide easier and faster (and legal) data retrieval than web scraping. Check out that provides APIs for different purposes. In case you scrape lots of data, you might want to consider using a database to be able to analyze or retrieve it fast.

Follow on how to create a local database with python. Be polite. As suggests, it is recommended to let people know that you are scraping their website so they can better respond to the problems your bot might cause.Again, do not overload the website by sending hundreds of requests per second. Speed up — parallelizationIf you decide to parallelize your program, be careful with your implementation so you don’t slam the server. And be sure you read the Dos and Don’ts section.

Check out the the definitions of parallelization vs concurrency, processors and threads and.If you extract a huge amount of information from the page and do some preprocessing of the data while scraping, the number of requests per second you send to the page can be relatively low.For my other project where I scraped apartment rental prices, I did heavy preprocessing of the data while scraping, which resulted in 1 request/second. In order to scrape 4K ads, my program would run for about one hour.In order to send requests in parallel you might want to use a package.Let’s say we have 100 pages and we want to assign every processor equal amount of pages to work with. If n is the number of CPUs, you can evenly chunk all pages into the n bins and assign each bin to a processor.

Each process will have its own name, target function and the arguments to work with. The name of the process can be used afterwards to enable writing data to a specific file.I assigned 1K pages to each of my 4 CPUs which yielded 4 requests/second and reduced the scraping time to around 17 mins.

Automating Cisco IOS By Kirk Byers 2015-05-26I recently started working on a method to automate various tasks in Cisco IOS using Python and Ansible. The general method consists of an SSH control channel and a separate SCP channel to transfer files:Once you have a reliable, programmatic file transfer mechanism, then there are several interesting automation use cases:. Loading new software images. Loading a device's initial configuration. Restoring a configuration (for a failed device). Loading configuration changes (configuration merge). Loading a completely new configuration file (configure replace).After some experimentation, I was able to add support into for a separate SCP file transfer mechanism.

This support consists of two new classes (SCPConn and FileTransfer).In addition to the new Netmiko classes, I also built an experimental Ansible module that transfers files to Cisco IOS devices using SCP. Automating Cisco IOS By Kirk Byers 2015-05-26I recently started working on a method to automate various tasks in Cisco IOS using Python and Ansible. The general method consists of an SSH control channel and a separate SCP channel to transfer files:Once you have a reliable, programmatic file transfer mechanism, then there are several interesting automation use cases:.

Loading new software images. Loading a device's initial configuration. Restoring a configuration (for a failed device). Loading configuration changes (configuration merge). Loading a completely new configuration file (configure replace).After some experimentation, I was able to add support into for a separate SCP file transfer mechanism.

This support consists of two new classes (SCPConn and FileTransfer).In addition to the new Netmiko classes, I also built an experimental Ansible module that transfers files to Cisco IOS devices using SCP.