scrape

NAME
     scrape - display a snippet of text parsed from a web page.

SYNOPSIS
     scrape -tokens TOKENS -dirs DIRECTIONS -url URL [options]

DESCRIPTION
     Mandatory arguments:
     -tokens <token_list_string>
        A list of tokens (delimited by spaces) which the user has
        determined to be sufficient to consistently point the
        parser to the exact location of the required data. For
        example
          -tokens <title> </title>
        would return all characters between these two HTML tags.

     -dirs <direction_list_string>
        A list of directions associated with each token. 0
        instructs the parser to look in the forward direction,
        while 1 instructs the parser to look in the reverse
        direction. In the above example, we would have
          -dirs 0 0
        since we will look for <title> from the beginning of
        the file in the forward direction, and then continue
        in the forward direction to find </title>. However,
        we could also specify
          -tokens </title> <title -dirs 0 1
        which would first find </title> and then reverse search
        for <title. This is useful if an HTML tag has attributes
        and you therefore cannot assume that the ">" in <title>
        will be present.
        
     -url <url>
        The full URL to the desired web page using the same format
        regardless of which HTTP variable-sending method is needed.
        Example: -url http://www.site.com?var1=${a}&var2=${b}
        Example: -url http://www.site2.com?var=%s
        Example: -url http://www.site3.com/%s.html
        The website name and any HTTP variables must be separated
        by a "?" and subsequent HTTP variables by a "&".

     Optional arguments:
     -method <http_method = get>
        If not included, the variables will be sent using the HTTP
        GET method. If set to anything else (including of course,
        post) the variables will be sent using the HTTP POST method.

     -textonly <display_method = 1>
        If not included, the scraped text will be returned as a
        simple string of text, easily fed into other YubNub functions.
        This is the default value of 1. If set to anything else,
        scrape will return the scraped text to a "mock" YubNub
        command line (see the defw, defn, and postalcode commands.)

     -debug <debugger = 0>
        If not included, no debugging information will be returned.
        If set to 1, some debugging information will be returned.
        This may help you see why a certain scrape is not working.

     Forthcoming arguments:
     -a character offset argument
     -a length argument
     -a word offset argument
     -a word delimiter argument
     -a numwords argument

EXAMPLE
     1. The qpostal command sends the user to the Canada Post site
     which displays the requested postal code:

        qpostal -n 1708 -s charles -t court -c val caron -p ON

     To scrape this postal code from this web page, you would
     have to examine the HTML of the above Canada Post site, and
     identify various tokens that will consistently guide the
     parser to the postal code string. One could define a command
     called qpostal_lite in this way:

        scrape -tokens >Postal Code< <tr> </tr> tblcell <br> >
               -dirs 0 0 0 1 0 1 -method post
               -url http://www.canadapost.ca/...
                  (see the yndesturl variable of the
                   qpostal command for the full URL)

     Typing:

        qpostal_lite -n 1708 -s charles -t court -c val caron -p ON

     will now return a simple string to your browser, which can
     now be piped into other commands which expect a postal code
     as an argument: cbc_pc {qpostal_lite ...}

     2. The wikt commands sends the user to the Wiktionary site
     which displays the requested definition:

        wikt eon

     To scrape this definition from this page, you would again
     examine the HTML of the above Wiktionary site, and
     identify various tokens that will consistently guide the
     parser to the definition string. This is often difficult.
     The following command gets the first def'n, which is not
     always expected. Define wikt_lite as:

        scrape -tokens </ol> <ol> -dirs 0 1 -textonly 0
               -url http://en.wiktionary.org/wiki/%s

     Typing wikt_lite eon will now return the definition to a "mock"
     YubNub command line prompt. If the textonly switch is dropped,
     the definition is returned to the browser, and can also be
     piped into other YubNub commands expecting a string as an
     argument: fspell {wikt_lite eon}

NOTES
     This command is very beta at the moment. Please bear with
     me. Comments and suggestions are welcome at the YubNub Google
     group (type yubgroup at the YubNub command line.)

AUTHOR
    Sean O'Hagan