Where the evidence comes from, and what we will not do to get it.
Blue Water Index is built primarily from boat advertisements that brokers, dealers and marketplaces make publicly accessible in order to sell boats. Publicly accessible does not mean unrestricted, which is why every source is assessed before collection begins — and why most of the work of running this company responsibly happens before a single price is read.
For what the evidence becomes once it is collected, see Methodology & Data.
Collection from a public source does not imply a partnership, an endorsement or a data-sharing relationship with that source. Blue Water Index independently determines source eligibility under the policy described here.
The presence of a vessel in our dataset does not mean that Blue Water Index has verified every fact in the underlying advertisement. Source observations are normalized, reconciled and analysed as described in Methodology & Data.
Three things have to be true before we read a site.
Under our collection policy, all three must hold. Its robots.txt permits the exact listing and discovery paths we intend to fetch. Its content is reachable without a login, a paywall or an anti-bot control. And its terms of use do not prohibit automated collection.
Any one failure is a no. The gate runs before the first listing is read, and it runs again when a site changes — a source that passed last year is not a source that passes today.
One crawler, one name, one request at a time.
BlueWaterIndexBot identifies itself on every request, carries a contact address in its user-agent string, and has a page of its own for whoever finds it in a log file. It never pretends to be a browser.
Requests to a site run strictly one at a time, around three seconds apart. Where a robots.txt states a longer crawl-delay, that becomes the floor for that site — six seconds, ten seconds, whatever is asked. A floor is only ever raised, and it is set when the source is assessed rather than decided mid-crawl.
A 429 or a 503 is an instruction, not an error. We record it and move on; we do not attempt to circumvent it.
When a site says no, that is the end of it.
There is a well-known set of ways past a refusal, and none of them are difficult. We do not use them. A refusal ends collection immediately, and we do not look for a technical way around it. A source is reconsidered only if the publisher changes its published position, or provides access through a partnership, a licensed feed or an API.
More than thirty sources are recorded as refused. Some are sources we would badly like to have, including two of the largest listing portals in the business. Others we had been reading for months before they changed their minds. They stay on it.
The crawler sends one honest name on every request. It does not impersonate Chrome to get past a check that is there to spot it.
Where a challenge exists to tell a browser from a crawler, rendering it in a headless browser is not a technical workaround. It is the refusal, answered with a lie.
We crawl from one place, and a site that blocks it has blocked us. Coming back from somewhere else is the same request wearing a hat.
One blocked marketplace publishes the identical database under three other domains. Reading those would be reading a refusal sideways, so none of them were ever requested.
An open door is not an invitation.
A Greek brokerage’s robots.txt permitted everything. Its terms of use prohibited scraping and data mining. Under our policy, the terms control: collection stopped, and every page already fetched was deleted without a single field being extracted from it.
A broker’s robots.txt was malformed — sixty bytes, and two defensible readings, one of which permitted us and one of which blocked the entire site. We took the one that blocked us.
A Finnish publisher uses the machine-readable rights reservation provided for under EU copyright law. Nothing technical stands in the way. We treat that reservation as dispositive for our collection policy and do not collect from the site — and the same goes more broadly: where a publisher has made a machine-readable reservation that reasonably applies to our activity, we treat it as a refusal.
The measurement, not the page.
What survives collection is a set of measurements: builder, model, year, length, asking price and currency, where the vessel lies, every change to the price, and the day the listing ended. Listing text and photographs are not republished. The Index is not a place to browse boats for sale, and is not built to become one.
Source material may be cached where that is necessary for collection integrity, for auditability, and to avoid unnecessary repeat requests — re-reading our own copy costs a site nothing. A cached page is working material: it is not republished, and it is not a public archive.
Why the source list is not published.
It is a fair thing to ask of a page like this one, so here is the answer rather than a silence. A published list would cost the sites on it more than it would tell you.
Collection is not a partnership, as the note at the top of this page says — but a list of names reads like one, and it would put brokers in the position of appearing to endorse a product they have never been asked about. It would also hand a ready-made target list to crawlers operating with none of the discipline above, which is precisely the traffic a robots.txt exists to manage.
What we will do is answer directly. If you operate a listings site and want to know whether we read it, what we hold and how to change either, ask us. You will get a specific answer about your site rather than a list about everyone else’s.
The shape of it is not a secret, and it is restated in the Market Pulse. As last published, at 31 August 2026:
Two lines, and we stop.
That is enough. You do not need a firewall rule, and you do not
need to email us first — though we would rather you did, because there is
usually a rate or a schedule that suits us both better than a block does. Where a
site specifies a Crawl-delay, we honour it.
If you operate a source we have collected from and want to discuss what we retain, or request removal of source material, contact us. We will review and action the request in accordance with our retention and legal obligations.
robots.txt permits it. We treat that as a no and leave. If it was
not meant as one, allowlisting the user-agent is enough — write to us and
we will confirm our addresses.