user, Author at Donotsad where can learn any thing work project and make money

Aug 09

Beautiful Soup – find_all_previous Method

Beautiful Soup – find_all_previous() Method ”; Previous Next Method Description The find_all_previous() method in Beautiful Soup look backwards in the document from this PageElement and finds all the PageElements that match the given criteria and appear before the current element. It returns a ResultsSet of PageElements that comes before the current tag in the document. Like all other find methods, this method has the following syntax − Syntax find_previous(name, attrs, string, limit, **kwargs) Parameters name − A filter on tag name. attrs − A dictionary of filters on attribute values. string − A filter for a NavigableString with specific text. limit − Stop looking after finding this many results. kwargs − A dictionary of filters on attribute values. Return Value The find_all_previous() method returns a ResultSet of Tag or NavigableString objects. If the limit parameter is 1, the method is equivalent to find_previous() method. Example 1 In this example, name property of each object that appears before the first input tag is displayed. from bs4 import BeautifulSoup fp = open(“index.html”) soup = BeautifulSoup(fp, ”html.parser”) tag = soup.find(”input”) for t in tag.find_all_previous(): print (t.name) Output form h1 body title head html Example 2 In the HTML document under consideration (index.html), there are three input elements. With the following code, we print the tag names of all preceding tags before thr <input> tag with nm attribute as marks. To differentiate between the two input tags before it, we also print the attrs property. Note that the other tags don”t have any attributes. from bs4 import BeautifulSoup fp = open(“index.html”) soup = BeautifulSoup(fp, ”html.parser”) tag = soup.find(”input”, {”name”:”marks”}) pretags = tag.find_all_previous() for pretag in pretags: print (pretag.name, pretag.attrs) Output input {”type”: ”text”, ”id”: ”age”, ”name”: ”age”} input {”type”: ”text”, ”id”: ”nm”, ”name”: ”name”} form {} h1 {} body {} title {} head {} html {} Example 3 The BeautifulSoup object stores the entire document”s tree. It doesn”t have any previous element, as the example below shows − from bs4 import BeautifulSoup fp = open(“index.html”) soup = BeautifulSoup(fp, ”html.parser”) tags = soup.find_all_previous() print (tags) Output [] Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – get_text Method

Beautiful Soup – get_text() Method ”; Previous Next Method Description The get_text() method returns only the human-readable text from the entire HTML document or a given tag. All the child strings are concatenated by the given separator which is a null string by default. Syntax get_text(separator, strip) Parameters separator − The child strings will be concatenated using this parameter. By default it is “”. strip − The strings will be stripped before concatenation. Return Type The get_Text() method returns a string. Example 1 In the example below, the get_text() method removes all the HTML tags. html = ””” <html> <body> The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. </body> </html> ””” from bs4 import BeautifulSoup soup = BeautifulSoup(html, “html.parser”) text = soup.get_text() print(text) Output The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. Example 2 In the following example, we specify the separator argument of get_text() method as ”#”. html = ””” The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. ””” from bs4 import BeautifulSoup soup = BeautifulSoup(html, “html.parser”) text = soup.get_text(separator=”#”) print(text) Output #The quick, brown fox jumps over a lazy dog.# #DJs flock by when MTV ax quiz prog.# #Junk MTV quiz graced by fox whelps.# #Bawds jog, flick quartz, vex nymphs.# Example 3 Let us check the effect of strip parameter when it is set to True. By default it is False. html = ””” The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. ””” from bs4 import BeautifulSoup soup = BeautifulSoup(html, “html.parser”) text = soup.get_text(strip=True) print(text) Output The quick, brown fox jumps over a lazy dog.DJs flock by when MTV ax quiz prog.Junk MTV quiz graced by fox whelps.Bawds jog, flick quartz, vex nymphs. Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – Quick Guide

Beautiful Soup – Quick Guide ”; Previous Next Beautiful Soup – Overview In today”s world, we have tons of unstructured data/information (mostly web data) available freely. Sometimes the freely available data is easy to read and sometimes not. No matter how your data is available, web scraping is very useful tool to transform unstructured data into structured data that is easier to read and analyze. In other words, web scraping is a way to collect, organize and analyze this enormous amount of data. So let us first understand what is web-scraping. Introduction to Beautiful Soup The Beautiful Soup is a python library which is named after a Lewis Carroll poem of the same name in “Alice”s Adventures in the Wonderland”. Beautiful Soup is a python package and as the name suggests, parses the unwanted data and helps to organize and format the messy web data by fixing bad HTML and present to us in an easily-traversable XML structures. In short, Beautiful Soup is a python package which allows us to pull data out of HTML and XML documents. HTML tree Structure Before we look into the functionality provided by Beautiful Soup, let us first understand the HTML tree structure. The root element in the document tree is the html, which can have parents, children and siblings and this determines by its position in the tree structure. To move among HTML elements, attributes and text, you have to move among nodes in your tree structure. Let us suppose the webpage is as shown below − Which translates to an html document as follows − <html> <head> <title>TutorialsPoint</title> </head> <body> <h1>Tutorialspoint Online Library</h1> It”s all Free </body> </html> Which simply means, for above html document, we have a html tree structure as follows − Beautiful Soup – web-scraping Scraping is simply a process of extracting (from various means), copying and screening of data. When we scrape or extract data or feeds from the web (like from web-pages or websites), it is termed as web-scraping. So, web scraping (which is also known as web data extraction or web harvesting) is the extraction of data from web. In short, web scraping provides a way to the developers to collect and analyze data from the internet. Why Web-scraping? Web-scraping provides one of the great tools to automate most of the things a human does while browsing. Web-scraping is used in an enterprise in a variety of ways − Data for Research Smart analyst (like researcher or journalist) uses web scrapper instead of manually collecting and cleaning data from the websites. Products, prices & popularity comparison Currently there are couple of services which use web scrappers to collect data from numerous online sites and use it to compare products popularity and prices. SEO Monitoring There are numerous SEO tools such as Ahrefs, Seobility, SEMrush, etc., which are used for competitive analysis and for pulling data from your client”s websites. Search engines There are some big IT companies whose business solely depends on web scraping. Sales and Marketing The data gathered through web scraping can be used by marketers to analyze different niches and competitors or by the sales specialist for selling content marketing or social media promotion services. Why Python for Web Scraping? Python is one of the most popular languages for web scraping as it can handle most of the web crawling related tasks very easily. Below are some of the points on why to choose python for web scraping − Ease of Use As most of the developers agree that python is very easy to code. We don”t have to use any curly braces “{ }” or semi-colons “;” anywhere, which makes it more readable and easy-to-use while developing web scrapers. Huge Library Support Python provides huge set of libraries for different requirements, so it is appropriate for web scraping as well as for data visualization, machine learning, etc. Easily Explicable Syntax Python is a very readable programming language as python syntax are easy to understand. Python is very expressive and code indentation helps the users to differentiate different blocks or scopes in the code. Dynamically-typed language Python is a dynamically-typed language, which means the data assigned to a variable tells, what type of variable it is. It saves lot of time and makes work faster. Huge Community Python community is huge which helps you wherever you stuck while writing code. Beautiful Soup – Installation Beautiful Soup is a library that makes it easy to scrape information from web pages. It sits atop an HTML or XML parser, providing Pythonic idioms for iterating, searching, and modifying the parse tree. BeautifulSoup package is not a part of Python”s standard library, hence it must be installed. Before installing the latest version, let us create a virtual environment, as per Python”s recommended method. A virtual environment allows us to create an isolated working copy of python for a specific project without affecting the outside setup. We shall use venv module in Python”s standard library to create virtual environment. PIP is included by default in Python version 3.4 or later. Use the following command to create virtual environment in Windows C:usesuser>python -m venv myenv On Ubuntu Linux, update the APT repo and install venv if required before creating virtual environment mvl@GNVBGL3:~ $ sudo apt update && sudo apt upgrade -y mvl@GNVBGL3:~ $ sudo apt install python3-venv Then use the following command to create a virtual environment mvl@GNVBGL3:~ $ sudo python3 -m venv myenv You need to activate the virtual environment. On Windows use the command C:usesuser>cd myenv C:usesusermyenv>scriptsactivate (myenv) C:Usersusersusermyenv> On Ubuntu Linux, use following command to activate the

Aug 09

Beautiful Soup – Porting Old Code

Beautiful Soup – Porting Old Code ”; Previous Next You can make the code from earlier version of Beautiful Soup compatible with the lates version by making following change in the import statement − Example from BeautifulSoup import BeautifulSoup #becomes this: from bs4 import BeautifulSoup If you get the ImportError “No module named BeautifulSoup”, it means you”re trying to run Beautiful Soup 3 code, but you only have Beautiful Soup 4 installed. Similarly, If you get the ImportError “No module named bs4″, because you”re trying to run Beautiful Soup 4 code, but you only have Beautiful Soup 3 installed. Beautiful Soup 3 used Python”s SGMLParser, a module that has been removed in Python 3.0. Beautiful Soup 4 uses html.parser by default, but you can also use lxml or html5lib. Although BS4 is mostly backwards-compatible with BS3, most of its methods have been deprecated and given new names for PEP 8 compliance. Here are a few examples − replaceWith -> replace_with findAll -> find_all findNext -> find_next findParent -> find_parent findParents -> find_parents findPrevious -> find_previous getText -> get_text nextSibling -> next_sibling previousSibling -> previous_sibling Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – extract Method

Beautiful Soup – extract() Method ”; Previous Next Method Description The extract() method in Beautiful Soup library is used to remove a tag or a string from the document tree. The extract() method returns the object that has been removed. It is similar to how a pop() method in Python list works. Syntax extract(index) Parameters Index − The position of the element to be removed. None by default. Return Type The extract() method returns the element that has been removed from the document tree. Example 1 html = ””” <div> Hello Python </div> ””” from bs4 import BeautifulSoup soup=BeautifulSoup(html, ”html.parser”) tag1 = soup.find(“div”) tag2 = tag1.find(“p”) ret = tag2.extract() print (”Extracted:”,ret) print (”original:”,soup) Output Extracted: Hello Python original: <div> </div> Example 2 Consider the following HTML markup − <html> <body> The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs./p> </body> </html> Here is the code − from bs4 import BeautifulSoup fp = open(”index.html”) soup = BeautifulSoup(fp, ”html.parser”) tags = soup.find_all() for tag in tags: obj = tag.extract() print (“Extracted:”,obj) print (soup) Output Extracted: <html> <body> The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. </body> </html> Extracted: <body> The quick, brown fox jumps over a lazy dog. DJs flock by when MTV ax quiz prog. Junk MTV quiz graced by fox whelps. Bawds jog, flick quartz, vex nymphs. </body> Extracted: The quick, brown fox jumps over a lazy dog. Extracted: DJs flock by when MTV ax quiz prog. Extracted: Junk MTV quiz graced by fox whelps. Extracted: Bawds jog, flick quartz, vex nymphs. Example 3 You can also use extract() method along with find_next(), find_previous() methods and next_element, previous_element properties. html = ””” <div> HelloPython </div> ””” from bs4 import BeautifulSoup soup=BeautifulSoup(html, ”html.parser”) tag1 = soup.find(“b”) ret = tag1.next_element.extract() print (”Extracted:”,ret) print (”original:”,soup) Output Extracted: Hello original: <div> Python </div> Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – parent Property

Beautiful Soup – parent Property ”; Previous Next Method Description The parent property in BeautifulSoup library returns the immediate parent element of the said PegeElement. The type of the value returned by the parents property is a Tag object. For the BeautifulSoup object, its parent is a document object Syntax Element.parent Return value The parent property returns a Tag object. For Soup object, it returns document object Example 1 This example uses .parent property to find the immediate parent element of the first tag in the example HTML string. html = “”” <html> <head> <title>TutorialsPoint</title> </head> <body> Hello World “”” from bs4 import BeautifulSoup soup = BeautifulSoup(html, ”html.parser”) tag = soup.p print (tag.parent.name) Output body Example 2 In the following example, we see that the <title> tag is enclosed inside a <head> tag. Hence, the parent property for <title> tag returns the <head> tag. html = “”” <html> <head> <title>TutorialsPoint</title> </head> <body> Hello World “”” from bs4 import BeautifulSoup soup = BeautifulSoup(html, ”html.parser”) tag = soup.title print (tag.parent) Output <head><title>TutorialsPoint</title></head> Example 3 The behaviour of Python”s built-in HTML parser is a little different from html5lib and lxml parsers. The built-in parser doesn”t try to build a perfect document out of the string provided. It doesn”t add additional parent tags like body or html if they don”t exist in the string. On the other hand, html5lib and lxml parsers add these tags to make the document a perfect HTML document. html = “”” Hello World “”” from bs4 import BeautifulSoup soup = BeautifulSoup(html, ”html.parser”) print (soup.p.parent.name) soup = BeautifulSoup(html, ”html5lib”) print (soup.p.parent.name) Output [document] Body As the HTML parser doesn”t add additional tags, the parent of parsed soup is document object. However, when we use html5lib, the parent tag”s name property is Body. Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – wrap Method

Beautiful Soup – wrap() Method ”; Previous Next Method Description The wrap() method in Beautiful Soup encloses the element inside another element. You can wrap an existing tag element with another, or wrap the tag”s string with a tag. Syntax wrap(tag) Parameters The tag to be wrapped with. Return Type The method returns a new wrapper with the given tag. Example 1 In this example, the tag is wrapped in <div> tag. html = ””” <html> <body> The quick, brown fox jumps over a lazy dog. </body> </html> ””” from bs4 import BeautifulSoup soup = BeautifulSoup(html, “html.parser”) tag1 = soup.find(”b”) newtag = soup.new_tag(”div”) tag1.wrap(newtag) print (soup) Output <html> <body> The quick, <div>brown</div> fox jumps over a lazy dog. </body> </html> Example 2 We wrap the string inside the tag with a wrapper tag. from bs4 import BeautifulSoup soup = BeautifulSoup(“tutorialspoint.com”, ”html.parser”) soup.p.string.wrap(soup.new_tag(“b”)) print (soup) Output tutorialspoint.com Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – find_previous_siblings Method

Beautiful Soup – find_previous_siblings() Method ”; Previous Next Method Description The find_previous_siblings() method in Beautiful Soup package returns all siblings that appear earlier to this PAgeElement in the document and match the given criteria. Syntax find_previous_siblings(name, attrs, string, limit, **kwargs) Parameters name − A filter on tag name. attrs − A dictionary of filters on attribute values. string − A filter for a NavigableString with specific text. limit − Stop looking after finding this many results. kwargs − A dictionary of filters on attribute values. Return Value The find_previous_siblings() method a ResultSet of PageElements. Example 1 Let us use the following HTML snippet for this purpose − Excellent Python Tutorial In the code below, we try to find all the siblings of <> tag. There are two more tags at the same level in the HTML string used for scraping. from bs4 import BeautifulSoup soup = BeautifulSoup(“ExcellentPythonTutorial”, ”html.parser”) tag1 = soup.find(”u”) print (“previous siblings:”) for tag in tag1.find_previous_siblings(): print (tag) Output Python Excellent Example 2 The web page (index.html) has a HTML form with three input elements. We locate one with id attribute as marks and then find its previous siblings. from bs4 import BeautifulSoup fp = open(“index.html”) soup = BeautifulSoup(fp, ”html.parser”) tag = soup.find(”input”, {”id”:”marks”}) sibs = tag.find_previous_sibling() print (sibs) Output [<input id=”age” name=”age” type=”text”/>, <input id=”nm” name=”name” type=”text”/>] Example 3 The HTML string has two tags. We find out the siblings previous to the one with id1 as its id attribute. html = ””” ExcellentPythonTutorial ””” from bs4 import BeautifulSoup soup = BeautifulSoup(html, ”html.parser”) tag = soup.find(”p”, id=”id1”) ptags = tag.find_previous_siblings() for ptag in ptags: print (“Tag: {}, Text: {}”.format(ptag.name, ptag.text)) Output Tag: p, Text: Python Tag: b, Text: Excellent Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – insert_before Method

Beautiful Soup – insert_before() Method ”; Previous Next Method Description The insert_before() method in Beautiful soup inserts tags or strings immediately before something else in the parse tree. The inserted element becomes the immediate predecessor of this one. The inserted element can be a tag or a string. Syntax insert_before(*args) Parameters args − One or more elements, may be tag or a string. Return Value This insert_before() method doesn”t return any new object. Example 1 The following example inserts a text “Here is an” before “Excellent in the given HTML markup string. from bs4 import BeautifulSoup, NavigableString markup = ”Excellent Python Tutorial from TutorialsPoint” soup = BeautifulSoup(markup, ”html.parser”) tag = soup.b tag.insert_before(“Here is an “) print (soup.prettify()) Output Here is an Excellent Python Tutorial from TutorialsPoint Example 2 You can also insert a tag before another tag. Take a look at this example. from bs4 import BeautifulSoup, NavigableString markup = ”Excellent Tutorial from TutorialsPoint” soup = BeautifulSoup(markup, ”html.parser”) tag = soup.b tag1 = soup.new_tag(”b”) tag1.string = “Python “ tag.insert_before(tag1) print (soup.prettify()) Output Excellent Python Tutorial from TutorialsPoint Example 3 The following code passes more than one strings to be inserted before the tag. from bs4 import BeautifulSoup markup = ”There are Tutorials from TutorialsPoint” soup = BeautifulSoup(markup, ”html.parser”) tag = soup.b tag.insert_before(“many “, ”excellent ”) print (soup.prettify()) Output There are many excellent Tutorials from TutorialsPoint Print Page Previous Next Advertisements ”;

Aug 09

Beautiful Soup – Discussion

Discuss Beautiful Soup ”; Previous Next In this tutorial, we will show you, how to perform web scraping in Python using Beautiful Soup 4 for getting data out of HTML, XML and other markup languages. In this we will try to scrap webpage from various different websites (including IMDB). We will cover beautiful soup 4, python basic tools for efficiently and clearly navigating, searching and parsing HTML web page. We have tried to cover almost all the functionalities of Beautiful Soup 4 in this tutorial. You can combine multiple functionalities introduced in this tutorial into one bigger program to capture multiple meaningful data from the website into some other sub-program as input. Print Page Previous Next Advertisements ”;