urllist.txt

urllist.txt generator

A plain text list of every indexable URL on your website – one per line.

1. Crawl your website

Website URL Max pages

Pages come from your sitemap first, then from links. Crawl once – every generator below reuses the result.

Include lastmod dates changefreq priority

Google ignores changefreq and priority; lastmod is used when accurate.

Page title Group by section

Business / website name

One-line summary (shown as the quote) Extra details (optional: services, areas served, contact) Add page descriptions

WordPress: block /wp-admin/ (keep admin-ajax) Block internal search results Block cart / checkout / my-account

Other paths to block (one per line, e.g. /private/)

Bot	Used for	Rule

Sitemap URLs (one per line)

Include non-indexable URLs too

Feed title Items File name

Feed description Use all pages (no blog found / no dates)

Generated file

URL or path Bot

Current live robots.txt

What is urllist.txt?

An old, simple sitemap format: one URL per line. Few search engines still use it, but it is handy for audits, migrations and bulk tools.