Reading a web page as Markdown, from inside the browser

Screenshotline takes screenshots of web pages. It also returns the same page as Markdown, from the same render, and that second endpoint gets more use than I expected when I built it.

The reason is simple. A screenshot is for a person. If the thing reading the page is a script or a model, handing it a PNG is the wrong shape: it has to run OCR or a vision model to recover text that was sitting in a DOM a moment earlier. That is slower, lossier and more expensive than just giving it the text.

So /extract runs the identical pipeline as /take and returns Markdown instead of pixels. Everything that makes the picture correct — the adaptive settle, the consent sweep, the collapsed ad slots, the scroll that triggers lazy images — makes the text correct for exactly the same reasons.

This post is about the part I got wrong first: where the extraction should happen.

Parse the DOM, not a copy of it

The obvious approach is to pull the HTML out of the browser and run it through a good server-side library. Readability to find the article, Turndown to convert it. Both are better than anything I would write at handling edge cases.

I went the other way: a dependency-free walker that runs inside the page, via page.evaluate, on the DOM that is already rendered. Not because the libraries are bad, but because by the time we are extracting, the browser has already done work that a serialised copy throws away.

Here is the clearest example. On danluu.com, the list of posts is written like this, with no whitespace at all between the elements:

<d>09/26</d><a href="...">Some post title</a>

<d> is not a real element, and the page styles it as a 4em flex item. Read the markup as text and you get 09/26Some post title. Read the computed layout and you can see the browser gives <d> a box, so there is a visual gap, so the text needs a space:

const LAID_OUT = /^(block|flex|grid|table|list-item|flow-root|table-cell|inline-block|inline-flex)/;
const laidOut = (el) => {
  const style = styleOf(el);
  return style ? LAID_OUT.test(style.display) : false;
};

Any unknown element the browser lays out as a box gets a space around its contents. That rule only exists because the extraction can ask the browser what it actually did.

Three more things come free in the same place:

Hidden content stays out. display: none, visibility: hidden and opacity: 0 are computed values. A tab panel that isn't open, or a mobile menu that only exists at narrow widths, would otherwise be indistinguishable from real content in the markup.

Shadow roots are reachable. innerText and querySelectorAll do not cross a shadow boundary, so a site built out of web components extracts as an empty document. Walking into open shadow roots is a few lines:

const childNodesOf = (node) => {
  const shadow = node.shadowRoot;
  if (shadow) return [...shadow.childNodes, ...node.childNodes];
  return [...node.childNodes];
};

Closed roots are unreachable by design, and nothing can be done about those. The same blind spot once hid a full-page consent modal from the screenshot sweep on dw.com, so it is worth knowing about in both directions.

The consent banner is already gone. By extraction time the sweep has removed it, and ad slots that were blocked have been collapsed and marked, so the walker can skip them by attribute. Nothing has to recognise a cookie wall twice.

The text arrives later than the page does

This is the part people underestimate, and it has nothing to do with Markdown.

timesofindia.indiatimes.com returns 446 characters of text if you extract after a 5-second settle, and 48,755 if you wait 15. Both responses are a clean 200. Nothing in either one says "this page wasn't finished".

That is why the extractor reuses the render pipeline's settle budget instead of grabbing text as soon as the DOM is ready: the budget keeps waiting while the page's text is still growing and stops early once there is plainly enough. A Markdown endpoint with a fixed 3-second wait would look fast and quietly return a header and a spinner on every heavy news site.

The same logic protects the sweep. The consent heuristic refuses to remove any element carrying more than 1,500 characters of text, because on france24.com it was eating about 1,200 characters of actual article.

Markdown that survives real pages

A few structural decisions, each from a page that broke the naive version.

Link-wrapped headings. Most news front pages are built as a card where an <a> wraps an <h2>. Emitted in the obvious order, that becomes [# Headline](url), which is neither a heading nor a link. So a heading inside a link is hoisted back out and the link goes inside it:

const heading = raw.match(/^\s*(#{1,6})\s/);
if (heading) return `\n\n${heading[1]} [${text}](${href})\n\n`;

Relative URLs are resolved against document.baseURI, for links and for images, because a Markdown document that leaves the page behind needs absolute links. data: image sources are dropped rather than inlined — a base64 blob is not useful to a reader and can be enormous.

Tables become pipe tables, with | inside a cell escaped. Lists keep their nesting, and continuation lines are indented to line up under the marker, so a multi-line list item does not break out of its list.

Page furniture is dropped, but carefully. With strip_chrome on, which is the default, nav, header, footer, aside, the matching ARIA roles and aria-hidden="true" are removed. Then there is a check that matters more than the rule itself:

let md = render();
if (options.stripChrome && md.replace(/\s/g, '').length < 200) {
  // Stripping took the page with it.
  for (const [el] of stripped) el.removeAttribute('data-screenshotline-hidden');
  md = childNodesOf(root).map((n) => walk(n, { depth: 0 })).join('');
}

Plenty of real sites put their only content inside an <aside> or hang it off a <nav>. If stripping the furniture empties the document, the whole thing is rendered again with the furniture left in. Returning a stub because the page's markup was unusual is worse than returning something slightly noisy.

There is a related lesson from the screenshot side. The consent sweep once ate apnews.com's entire site header — 1,022 links and two <nav> elements — because a trending headline said "Girl Scout cookies" and the keyword test matched cookies. Structural rules ("a consent banner is never the site's navigation") hold up where keyword rules do not.

Using it

# Markdown, as text
curl "https://api.screenshotline.com/extract?url=https://example.com" \
  -H "X-Access-Key: $KEY"

# Or JSON, with the title and character count
curl "https://api.screenshotline.com/extract?url=https://example.com&response=json" \
  -H "X-Access-Key: $KEY"

strip_chrome=false keeps the navigation. max_chars=20000 truncates and appends a [truncated] marker, which is handy when the text is going into a context window. The blocking options (block_ads, block_cookie_banners, block_chats, block_popups) work the same as for a screenshot. X-Text-Length on the response tells you how much text came back without parsing the body.

For agents there is a hosted MCP server with the same two operations, screenshot and read_page:

claude mcp add --transport http screenshotline \
  https://api.screenshotline.com/mcp --header "X-Access-Key: $KEY"

What this is not

It is not a readability clone. It does not try to find the one true article and throw everything else away, because a front page, a docs page and a product listing are all legitimate things to read and none of them has an article in the middle. It strips furniture, keeps structure, and leaves the judgement to whatever is reading.

It does no summarising, and no cleverness beyond the DOM. There is no model in this path at all — which I mention because "web page to Markdown for LLMs" usually implies one.

And it does not get past bot walls. If a page shows a captcha, you get the captcha's text, and a blank page is flagged in a response header instead of being returned as a success.

The code is one file, src/extract.js, about 250 lines, no dependencies: read it here. The hosted API has 500 free renders a month and no card, and the whole thing self-hosts with Docker.