Microcode: PowerShell Scripting Tricks: Scripting the Web (Part 3) (Resolve-Link, Get-WebPageLink)
The first post in this series was learning to crawl. I introduced Get-Web, which allows you to use System.Net.Webclient to download web sites in a variety of ways. The next post was learning to walk. I showed us Get-MarkupTag, which helps coerce parts of the web into XML. Now we can start to really have some fun with the data and run wild.
Pulling out semi-structured data is one thing, but it’s important to be able to pull out more complex information as well. One interesting case is pulling out all of the links from a webpage. This task breaks down into four smaller tasks:
- Downloading the page (done with Get-Web)
- Getting the <a> tags in a meaningful way (done with Get-MarkupTag)
- Extracting out the href attribute
- Determining if the link is relative or absolute
To determine if the link is relative or absolute, I made a Resolve-Link function. It takes a base url (e.g https://www.foo.com/blah/blah.asp) and a link found on it, and returns the real item it resolves to. It optionally returns a property bag with the type of link and the resolved link.
Here’s Resolve-Link:
**
function Resolve-Link([Uri]$uri,<br> [string]$link,<br> [switch]$returnLinkType) {<br> #.Synopsis<br> # Resolves a relative or absolute link to an absolute url<br> #.Description<br> # Takes a uri and a link to a page and returns the absolute url, or<br> # optionally returns a property bag with the link type<br> # (absolute, relative, or host relative) and the link<br> #.Parameter uri<br> # The uri the link is located on<br> #.Parameter link<br> # The original link text<br> #.Parameter returnLinkType<br> # The return link type<br> #.Example<br> # Resolve-Link http:/www.microsoft.com/ /technet/scriptcenter<br> if ($link.StartsWith("/")) {<br> # Relative to Host site<br> if ($returnLinkType) {<br> return New-Object Object |<br> Add-Member NoteProperty Type "Host Relative" -PassThru |<br> Add-Member NoteProperty Link ([uri]"$($uri.Scheme)://$($uri.DnsSafehost)$($link)") -PassThru<br> }<br> return "$($uri.Scheme)://$($uri.DnsSafehost)$($link)"<br> } else {<br> if ($link.StartsWith("$($uri.Scheme)://")) {<br> # Absolute Link<br> if ($returnLinkType) {<br> return New-Object Object |<br> Add-Member NoteProperty Type "Absolute" -PassThru |<br> Add-Member NoteProperty Link ([uri]$link) -PassThru<br> } <br> return $link<br> } else {<br> # Relative link<br> $realLink = $uri.AbsoluteUri.Substring(0,<br> $uri.AbsoluteUri.LastIndexOf("/")) + "/$link" <br> if ($returnLinkType) {<br> return New-Object Object |<br> Add-Member NoteProperty Type "Relative" -PassThru |<br> Add-Member NoteProperty Link ([uri]$realLink) -PassThru<br> }<br> return $realLink <br> }<br> } <br>}
Once Resolve-Link was written, making Get-WebPageLink is an incredible snap. It’s below, and it actually takes only 3 lines to do the real work and 11 lines to explain the work and give examples.
**
function Get-WebPageLink($url) {<br> #.Synopsis<br> # Returns all of the links within a webpage<br> #.Description<br> # Resolves all <A> references and returns a property bag with<br> # the text contained in the link, the page the link came from,<br> # and the type of link returned (absolute, host relative, or relative)<br> #.Parameter urltp<br> # The page to get links from<br> #.Example<br> # Get-WebPageLink https://blogs.msdn.com/<br> Get-MarkupTag a (Get-Web $url) | Foreach-Object {<br> Resolve-Link $url $_.Xml.Href -returnLinkType |<br> Add-Member NoteProperty Text $_.Xml."#text" -PassThru <br> }<br>}
Go ahead and give Get-WebpageLink a whirl:
**
Get-WebpageLink https://blogs.msdn.com
Ready for some real fun? Remember way back when I did a post about getting RSS feeds in PowerShell with Microsoft.FeedsManager (Get-Feed). If you have that script handy, go ahead and check out this one liner that will refresh every RSS item you’ve got and extract out all of the links from it.
**
Get-Feed -recurse -articles | Foreach-Object { Get-WebPageLink $_.Link }
That particular command line can take a while, depending on how many blogs you subscribe to, but it gives you a brand new view on blogs (as a simmering stew of scripts, rather than just text to be read and comprehended).
There’s more fun to come in unlocking the web, but these two scripts should get you started in extracting a little more into the wild world of the web.
Hope this Helps,
James Brundage [MSFT]