THIS TUTORIAL IS A DRAFT
By popular request, I decided to write a small tutorial on how to parse a webpage (also known as screen scraping), and grab the data you are looking for. The example is a script for the Homeseer automation software, and uses the hs.GetUrl function, but it's written in VBscript, and many of these functions can be found in other languages too, so this is a great start, no matter what language you want to use.
In this example, I am going to grab the data from http://www.randomjoke.com/topic/oneliners.php which shows a random one-liner/joke every time you load the page. The functions I used for this page are also the ones you would use when parsing other websites. Everyone has their own "coding" style, so there are different (and more efficient) ways of doing the same thing, but I am trying to keep things easy here.
These are the most important functions used in this example:
Most scripts have sub's and functions, a sub can execute many instructions, while a function is designed to do something with the data it was given, and return the result. All Homeseer scripts have a sub named main(), parameters can be given, but in this example won't be needed.
Before we do the actual processing, we have to declare (allocate memory space) the variables we want to use in this script. It's a good habit to have certain variable naming conventions, to keep things organized, in this case, instead of using "Site", "Document" and "Body", I used "strSite", "strDocument" and "strBody", showing that these variables are of the type String (some of the other types are Integers, Variants, Booleans).
'hs.GetUrl' requires us to split up the url in several parts, the host (strSite), the document we want to retrieve (strDocument), and the port (80, default http port).
We use the 'hs.GetUrl' function to connect to the website and retrieve the HTML code, and store it in 'strBody'. Once we have the data, we remove all the Tab's and Carriage Returns, by replacing them with "" (nothing). chr(9) is the ASCII code for the Tab character, while chr(10) and (chr(13) are the ASCII values for Carriage Returns & Line Feeds (the enter key on your windows keyboard). This doesn't have to be done, but it does make parsing easier (and since you are removing all CR's and LF's you are putting everything on 1 line).
This is a snippet of the HTML source code that contains the actual quote. You can view the source by going to the site with your browser, right click and select 'View Source'. I'll highlight the keywords we are using here to grab the data.
The Instr() function will tell us the position of a given string, and Mid() allows us to select a part of a string, by specifying the start and end position (which is where you use Instr to figure out where the data you are looking for starts). When you are looking for data in a webpage, you have to try to use a unique 'keyword', in this case, "topic lists" occurs only once and brings us pretty close to the data we are looking for. Once you get to where the quote starts, you have to figure out where the quote ends, so you can chop off the rest of the useless html data.
In this example, I am just going to print the result to the log view window, but you can use i.e. hs.SetDeviceString "v5", strBody to update a device with the status
This is what the finished script looks like:
Some good VBscript references:
http://www.devguru.com/Technologies/vbscri...ript_intro.html
By popular request, I decided to write a small tutorial on how to parse a webpage (also known as screen scraping), and grab the data you are looking for. The example is a script for the Homeseer automation software, and uses the hs.GetUrl function, but it's written in VBscript, and many of these functions can be found in other languages too, so this is a great start, no matter what language you want to use.
In this example, I am going to grab the data from http://www.randomjoke.com/topic/oneliners.php which shows a random one-liner/joke every time you load the page. The functions I used for this page are also the ones you would use when parsing other websites. Everyone has their own "coding" style, so there are different (and more efficient) ways of doing the same thing, but I am trying to keep things easy here.
These are the most important functions used in this example:
Most scripts have sub's and functions, a sub can execute many instructions, while a function is designed to do something with the data it was given, and return the result. All Homeseer scripts have a sub named main(), parameters can be given, but in this example won't be needed.
Before we do the actual processing, we have to declare (allocate memory space) the variables we want to use in this script. It's a good habit to have certain variable naming conventions, to keep things organized, in this case, instead of using "Site", "Document" and "Body", I used "strSite", "strDocument" and "strBody", showing that these variables are of the type String (some of the other types are Integers, Variants, Booleans).
'hs.GetUrl' requires us to split up the url in several parts, the host (strSite), the document we want to retrieve (strDocument), and the port (80, default http port).
We use the 'hs.GetUrl' function to connect to the website and retrieve the HTML code, and store it in 'strBody'. Once we have the data, we remove all the Tab's and Carriage Returns, by replacing them with "" (nothing). chr(9) is the ASCII code for the Tab character, while chr(10) and (chr(13) are the ASCII values for Carriage Returns & Line Feeds (the enter key on your windows keyboard). This doesn't have to be done, but it does make parsing easier (and since you are removing all CR's and LF's you are putting everything on 1 line).
This is a snippet of the HTML source code that contains the actual quote. You can view the source by going to the site with your browser, right click and select 'View Source'. I'll highlight the keywords we are using here to grab the data.
The Instr() function will tell us the position of a given string, and Mid() allows us to select a part of a string, by specifying the start and end position (which is where you use Instr to figure out where the data you are looking for starts). When you are looking for data in a webpage, you have to try to use a unique 'keyword', in this case, "topic lists" occurs only once and brings us pretty close to the data we are looking for. Once you get to where the quote starts, you have to figure out where the quote ends, so you can chop off the rest of the useless html data.
In this example, I am just going to print the result to the log view window, but you can use i.e. hs.SetDeviceString "v5", strBody to update a device with the status
This is what the finished script looks like:
Some good VBscript references:
http://www.devguru.com/Technologies/vbscri...ript_intro.html
