Watch

Poll a page and get only the links you haven't seen before.

watch reads a listing, feed or search page and returns the same-site links that are new since your last poll. You decide when to poll and what a link means.

Filter links with a regular expression.
The first poll returns every current link and sets first_poll, so you can treat it as a baseline.
Each poll is one ordinary read: paced, receipted, and able to climb past walls.

poll = await web.watch("https://news.ycombinator.com/", link_pattern=r"item\?id=\d+")
for link in poll["new"]:
    print(link["url"])

frankensurf watch https://news.ycombinator.com/ --link-pattern 'item\?id=\d+'

{
  "url": "https://news.ycombinator.com/",
  "status": "observed",
  "first_poll": true,
  "new": [
    { "url": "https://news.ycombinator.com/item?id=49969183", "text": "4 hours ago" },
    { "url": "https://news.ycombinator.com/item?id=49971230", "text": "1 hour ago" }
  ],
  "links_on_page": 30,
  "seen_count": 30,
  "receipt": { "status": "observed", "method": "http" }
}

Options

Option	Default	Description
`link_pattern`	all same-site links	Regular expression a link URL must match.
`state_key`	the URL	Name of the watch, if you watch one URL several ways.
`max_seen`	`5000`	Links remembered per watch.

Seen links are stored under <state>/watches/.