OpenVibeBot

OpenVibe's crawler, and how to steer it

OpenVibeBot reads public pages that describe themselves: product pages with prices, feeds and sitemaps. OpenVibe Deals, Search and the other OpenVibe sites use what it reads to link back to those pages, with where and when each fact was read. It follows your robots.txt every time, and you can stop it in one line.

User agent: Mozilla/5.0 (compatible; OpenVibeBot/1.0; +https://openvibe.network/bot)
robots.txt names: OpenVibeBot or OpenVibeSourcesOpenVibeSources is its older name and is still obeyed.

What it reads

Your sitemap first.The Sitemap lines in robots.txt, else /sitemap.xml. Pages a sitemap says changed are read sooner; unchanged ones less and less often.
Structured data.schema.org JSON-LD on the page: a product's name, price, list price, currency, availability, image and address. A page without it is skipped.
Links, only if asked.When a site has no sitemap, links on the same site, at most two steps from the home page.

It keeps facts and a link, never a copy of your page, and nothing about people. It never leaves the site it was pointed at.

What it obeys

robots.txt (RFC 9309), before every request and again on every redirect. Rules for OpenVibeBot or OpenVibeSources apply; otherwise the rules for *.
Crawl-delay, as the least time between two requests.
noindex and nofollow in <meta name="robots">, in <meta name="OpenVibeBot"> and in the X-Robots-Tag header.
rel=canonical: the page's own address wins over the one it was reached by.
429, or 503 with Retry-After: the whole site waits, not just that page.
Gone pages: a 404 twice or a 410 once removes what came from that page.

1 sleast gap between two requests to one site

5 sgap when it reads a shop

2sites read at the same time, at most

24 hlongest a robots.txt is remembered

Block it or slow it down

Changes reach it within 24 hours, the longest it keeps a robots.txt. Nothing else to set up and nobody to ask.

Keep it off the whole site (robots.txt)

User-agent: OpenVibeBot
Disallow: /

Keep it out of some paths (robots.txt)

User-agent: OpenVibeBot
Disallow: /checkout/
Disallow: /account/

Slow it down (robots.txt)

User-agent: OpenVibeBot
Crawl-delay: 30

Keep one page out (HTML)

<meta name="OpenVibeBot" content="noindex, nofollow">

Keep a file out (HTTP header)

X-Robots-Tag: OpenVibeBot: noindex

What it never does

Solve CAPTCHAs.
Sign in or read anything behind a sign-in.
Get past cookie walls or paywalls.
Hide what it is: every request carries the user agent above.

How a site gets read at all

A short check of a site's public answers decides, the same way for every site: if its pages answer, they are read; if only its sitemap or feeds answer, OpenVibe uses feeds, the site's own API or a partnership instead and leaves the pages alone; if nothing answers, nothing is read and the site is checked again a day later.

Want your shop listed from a feed instead?

A product feed, an affiliate program or an API is faster for you and more accurate for shoppers than any crawl. Tell us where it is and the crawler steps aside. Questions, a request that hit you too hard, or a correction: write to [email protected].

For machines: /bot.json says all of this as JSON.