Crawl a site into clean, chunked text for AI
Crawls the demo newspaper one link deep, with robots.txt honoured, and turns every page into clean Markdown: the menus, cookie banner, sidebars and footers are stripped and the story is kept. The text is then cut into chunks with stable ids, which is the input a vectorUpsert or an llm node wants. Add an ai and a vectordb connection and a vectorUpsert node, or start from the 'Website → clean text → vector store' template, to finish the knowledge base.
The pipeline
This is the actual graph the template creates — 2 steps.
Connects to
- HTTP API
How you know it worked
Open the run on the canvas and click each node.
Try this templateUsed in: Education & government
Related templates
Database → transform → database (upsert)
Reads customers out of the demo database, reshapes them with a mapping, and upserts them into demo.orders keyed on order_ref. Running it twice is…
LDAP directoryDirectory group → email report
Looks a group up in the directory and emails its membership as an attachment. Reads the group entry's own member list, so it works against a plain…
A published app with a multi-page front end on a file databaseSchool management system (app + front end)
A whole small app on a local SQLite file: students, classes and attendance, with a multi-page front end you open in the browser. The 'setup'…
Stop moving data by hand.
Start free with three projects, or talk to us about running it across your organisation.