VB.Net Web Scraper using HTMLAgilityPack – Part II

In this tutorial, we’ll be selecting the text inside <p> and <div> tags from an HTML page and save it to text file as a bonus. First, we create our HTML document, name it sample2.html.


<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<title>Sample 1</title>
</head>
<body>
	<div>First div - some more text here</div>
	<p>Paragraph1 - Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ornare velit vel ipsum consectetur facilisis. In iaculis tempor elit a porttitor. Etiam nisl eros, rutrum a purus a, placerat fringilla ante. </p>
	<div>Second div - some more text here</div>
	<p>Paragraph2 - Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ornare velit vel ipsum consectetur facilisis. In iaculis tempor elit a porttitor. Etiam nisl eros, rutrum a purus a, placerat fringilla ante. </p>
</body>
</html>

When you are finished with the HTML page, create a new VB.Net Project in Visual Studio. Select Console Application and name it GrabElements. Type the codes below.


Imports HtmlAgilityPack

Module Module1

Sub Main()
Dim doc As New HtmlDocument
doc.Load("C:\\Users\\allmankind\\Documents\\sample2.html")

Console.WriteLine("The div tags are: ")
For Each div As HtmlNode In doc.DocumentNode.SelectNodes("//div")
Console.WriteLine(div.InnerText)
Next

Console.WriteLine()
Console.WriteLine("The p tags are:")
For Each p As HtmlNode In doc.DocumentNode.SelectNodes("//p")
Console.WriteLine(p.InnerText)
Next

Console.ReadKey()
End Sub

End Module

When you create a new VB.Net console application in visual studio, there will be codes generated for you. Like the libraries that are imported at the top with the import keyword. The main class is also generated.

Add the .dll inside HTMLAgilityPack folder in your project via Add References under Project menu. Then add the code that imports the library like on the code above. Next, we initialize HTMLDocument with doc as its name as something that holds the html document. Then, we load the html page with its local address in your computer. Then we run a loop that reads all links inside it and outputs them on the console.

Now like what I did on the part 2 java tutorial, I am going to save the elements to a text file.

Create a new VB.Net Console Application in Visual Studio and name it GrabElementsTxt. Type the codes below.


Imports HtmlAgilityPack
Imports System.IO

Module Module1

Sub Main()
Dim file As New StreamWriter("C:\\Users\\allmankind\\Documents\\sample2.txt")
Dim doc As New HtmlDocument
doc.Load("C:\\Users\\allmankind\\Documents\\sample2.html")
file.WriteLine("The div tags are: ")
For Each div As HtmlNode In doc.DocumentNode.SelectNodes("//div")
file.WriteLine(div.InnerText)
Next

file.WriteLine()
file.WriteLine("The p tags are:")
For Each p As HtmlNode In doc.DocumentNode.SelectNodes("//p")
file.WriteLine(p.InnerText)
Next
file.Close()
End Sub

End Module

Thank you for reading. Don’t forget to share and leave a comment.

Advertisements

One thought on “VB.Net Web Scraper using HTMLAgilityPack – Part II

  1. Pingback: broken link checker google chrome

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s