An infinite-scroll extraction needs a stop condition and an item identity. Waiting longer or scrolling forever does not establish completeness. On a page you are permitted to automate, cap the run, detect whether new records actually appeared and preserve the reason it stopped.
Use explicit completion signals
| Signal | Action |
|---|---|
| End marker or disabled load control | Stop and record the page’s completion signal |
| Repeated item identity | Deduplicate without counting it as new content |
| No new items across a bounded check | Stop or report an incomplete run |
| Maximum rounds or deadline reached | Return partial output with its boundary |
| Request or page error | Record the error separately |
A controlled Playwright fixture
The owned browser fixture below uses a local page with three loading batches, a repeated record and an explicit disabled end control. The bounded run collected seven distinct fixture items and stopped after three successful loading rounds. These are fixture observations, not a commercial-site coverage benchmark.
Select items by stable source identity rather than screen position. After each loading action, wait for a specific new-item or end signal under a finite deadline. Use the source’s actual authorized loading mechanism; a button fixture does not prove that an arbitrary production scroll handler behaves the same way.
from playwright.sync_api import sync_playwright
html = "<html><body><ul id=\"items\"></ul><button id=\"more\">Load next</button>\n<script>let batch=0; function load(){batch++;const list=document.querySelector('#items');\nfor(let i=0;i<3;i++){let e=document.createElement('li');e.setAttribute('data-id', String((batch-1)*2+i));e.textContent='item '+e.dataset.id;list.appendChild(e);}\nif(batch===3){document.querySelector('#more').disabled=true;}}\ndocument.querySelector('#more').onclick=load; load();</script></body></html>"
with sync_playwright() as p:
browser = p.chromium.launch(channel="chrome", headless=True)
page = browser.new_page()
page.set_content(html)
found, rounds = {}, 0
for _ in range(5):
rounds += 1
for item in page.locator("#items li").all():
found[item.get_attribute("data-id")] = item.inner_text()
control = page.locator("#more")
if control.is_disabled():
break
control.click()
assert rounds == 3 and len(found) == 7
browser.close()Run this owned example with Playwright and an installed Chrome browser. It creates a separate headless browser context; it does not use an existing logged-in profile. Adapt the loading action and readiness signal to the permitted live page.
Adapt the test before using a live page
A Reader rendering mode is a selected-page extraction operation. It does not replace an application-specific browser automation task that must traverse every scroll state.
- Replace fixture selectors with selectors verified on the permitted page.
- Set a deadline, a round cap and a maximum retained-item budget.
- Record duplicate counts and the final stop reason.
- Compare the collected item identities with a known authorized sample.
Reference documentation: playwright.dev — /python/docs/api/class-page; www.searchcans.com — /reader-api.
