Category Archives: C#

07.08.13
by

C# Web Scraper using HTMLAgilityPack – Part II

In this tutorial, we’ll be selecting the text inside <p> and <div> tags from an HTML page and save it to text file as a bonus. First, we create our HTML document, name it sample2.html.


<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<title>Sample 1</title>
</head>
<body>
	<div>First div - some more text here</div>
	<p>Paragraph1 - Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ornare velit vel ipsum consectetur facilisis. In iaculis tempor elit a porttitor. Etiam nisl eros, rutrum a purus a, placerat fringilla ante. </p>
	<div>Second div - some more text here</div>
	<p>Paragraph2 - Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ornare velit vel ipsum consectetur facilisis. In iaculis tempor elit a porttitor. Etiam nisl eros, rutrum a purus a, placerat fringilla ante. </p>
</body>
</html>

When you are finished with the HTML page, create a new C# Project in Visual Studio. Select Console Application and name it GrabElements. Type the codes below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using HtmlAgilityPack;

namespace GrabElements
{
class Program
{
static void Main(string[] args)
{
HtmlDocument doc = new HtmlDocument();
doc.Load(“C:\\Users\\allmankind\\Documents\\sample2.html”);
Console.WriteLine(“The div tags are: “);
foreach (HtmlNode div in doc.DocumentNode.SelectNodes(“//div”))
{
Console.WriteLine(div.InnerText);
}

Console.WriteLine(“\nThe p tags are:”);
foreach (HtmlNode p in doc.DocumentNode.SelectNodes(“//p”))
{
Console.WriteLine(p.InnerText);
}
Console.ReadKey();

}
}
}

When you create a new C# console application in visual studio, there will be codes generated for you. Like the libraries that are imported at the top with the using keyword. The main class is also generated.

Add the .dll inside HTMLAgilityPack folder in your project via Add References under Project menu. Then add the code that imports the library like on the code above. Next, we initialize HTMLDocument with doc as its name as something that holds the html document. Then, we load the html page with its local address in your computer. Then we run a loop that reads all links inside it and outputs them on the console.

Now like what I did on the part 2 java tutorial, I am going to save the elements to a text file.

Create a new C# Console Application in Visual Studio and name it GrabElementsTxt. Type the codes below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using HtmlAgilityPack;
using System.IO;

namespace GrabElementsTxt
{
class Program
{
static void Main(string[] args)
{
StreamWriter file = new StreamWriter(“C:\\Users\\allmankind\\Documents\\sample2.txt”);
HtmlDocument doc = new HtmlDocument();
doc.Load(“C:\\Users\\allmankind\\Documents\\sample2.html”);
file.WriteLine(“The div tags are: “);
foreach (HtmlNode div in doc.DocumentNode.SelectNodes(“//div”))
{
file.WriteLine(div.InnerText);
}

file.WriteLine(“\nThe p tags are:”);
foreach (HtmlNode p in doc.DocumentNode.SelectNodes(“//p”))
{
file.WriteLine(p.InnerText);
}
file.Close();
}
}
}

Thank you for reading. Don’t forget to share and leave a comment.

07.08.13
by

C# Web Scraper using HTMLAgilityPack – Part I

In this tutorial, we will be developing a simple web scraping program that scrapes the link names and its href within an HTML Page. For this series of tutorials, I will be using Visual Studio 2010 for the C# language and a library called HtmlAgilityPack.

You can download HTMLAgilityPack here http://htmlagilitypack.codeplex.com/releases/view/90925.

First, we’ll create our own html document to try-out the programs we are going to develop. Then, we’ll try the program with a valid webpage URL.

Here is our sample HTML document named sample1.html.


<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<title>Sample 1</title>
</head>
<body>
  <div id="wrapper">
  <a href="link1.html">This is link1</a>
  <a href="link2.html">This is link2</a>
    <div>
      <a href="link3.html">This is link3</a>
    </div>
  </div>
</body>
</html>

When you are finished with the HTML page, create a new C# Project in Visual Studio. Select Console Application and name it GetLinks. Type the codes below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using HtmlAgilityPack;

namespace GetLinks
{
class Program
{
static void Main(string[] args)
{
HtmlDocument doc = new HtmlDocument();
doc.Load(“C:\\Users\\allmankind\\Documents\\sample1.html”);
foreach(HtmlNode link in doc.DocumentNode.SelectNodes(“//a”))
{
Console.WriteLine(link.InnerText);
Console.WriteLine(link.Attributes[“href”].Value);
}
Console.ReadKey();

}
}
}

When you create a new C# console application in visual studio, there will be codes generated for you. Like the libraries that are imported at the top with the using keyword. The main class is also generated.

Add the .dll inside HTMLAgilityPack folder in your project via Add References under Project menu. Then add the code that imports the library like on the code above. Next, we initialize HTMLDocument with doc as its name as something that holds the html document. Then, we load the html page with its local address in your computer. Then we run a loop that reads all links inside it and outputs them on the console.

Now let’s test our program with an article in wikipedia for example. We’ll use this link http://en.wikipedia.org/wiki/Language. There will be slight changes to the code, see below.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using HtmlAgilityPack;

namespace GetLinks
{
class Program
{
static void Main(string[] args)
{
HtmlDocument doc = new HtmlWeb().Load(“http://en.wikipedia.org/wiki/Language&#8221;);
foreach(HtmlNode link in doc.DocumentNode.SelectNodes(“//a”))
{
Console.WriteLine(link.InnerText);
Console.WriteLine(link.Attributes[“href”].Value);
}
Console.ReadKey();

}
}
}

Thank you for reading. Don’t Forget to share and leave a comment.