Last year, I bought a DP-558 KiSS hard disk recorder. One cool feature the player has is the ability to schedule recordings
\nthrough an Electronic Programme Guide (EPG) on the internet. This is
\nvery simple, using your own userid you log on to the site browse for
\ninteresting programs and record them. The hard disk recorder, which is
\nconnected to the internet regularly polls a KiSS server to find out about which
\nrecordings have been scheduled. This is possible since the hardware id
\nof the KiSS player is linked to your user id. Now this is cool but
\nwouldn’t it be even more cool to automatically record certain shows
\nwhen they appear and to send notifications by mail about possibly
\ninteresting programs? <\/p>\n
This is how I came up with the idea to crawl the KiSS EPG site for
\nprogramme information and autmatically record interesting programs.
\nEvery day at a very early time (e.g. 5 AM) it crawls the site for
\ninteresting shows. To implement the crawling I developed a very simple
\nweb crawling framework. The framework defines the concepts of Page and
\nAction. A Page is basically an HTML page transformed into an XML
\ncontent model of the page. An action is basically a hyperlink with a
\nuser friendly (content based) name. To implement crawling a program
\nmust be written that performs the necessary navigation using pages and
\nactions, extracting the content as it progresses. <\/p>\n
The framework creates Page objects by first tidying up the HTML page into XHTML (using jtidy)
\nand afterwards providing the interpretation of the HTML (=content +
\npresentation) as content only in XML. The framework determines which
\nXSLT to use based on configuration based on page type or URL. If the
\naction in the transformed page contains a page type, then that is used
\nto determine the XSLT. Otherwise, a URL match determines which XSLT to
\nuse.<\/p>\n
The KiSS crawler is just an application of this simple web crawling
\nframework. It consists of XSLT transformations, a program for
\nnavigating the KiSS site, configuration of the basic crawler, and a
\nconfiguration at application level defining which programs should be
\nrecorded and for which programs a notification should be sent. The
\ncrawler works like a charm. It retrieves detailed program information
\nincluding time, title, description, and categorization and sends a
\ndetailed notification afterwards. The source code will be made open
\nsource under the wamblee.org flag. In fact it can be accessed already
\nusing the subversion URL https:\/\/wamblee.org\/svn\/public\/utils. <\/p>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[7],"tags":[],"_links":{"self":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts\/115"}],"collection":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/comments?post=115"}],"version-history":[{"count":0,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/posts\/115\/revisions"}],"wp:attachment":[{"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/media?parent=115"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/categories?post=115"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/brakkee.org\/site\/wp-json\/wp\/v2\/tags?post=115"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}