# Collecting data from hostile websites with AI agents

European road freight has no public price index. We used Talona to gather 209,342 truck postings across 1,370 corridors, then published the whole thing as an open dataset.

Kacper Gadomski | Aug 27, 2026 | 8 min read

Source: https://talona.ai/blog/european-road-freight-dataset

Ask what it costs to move a truck from Warsaw to Duisburg this week and you will not find a public answer. Road freight carries more goods across Europe than rail, water and air combined, and its prices are private. They sit in commercial feeds sold by the seat, or scattered across freight exchanges that show you one page at a time behind a login.

So we collected it: 209,342 truck capacity postings, 1,370 country corridors, weekly and monthly spot rates in EUR per kilometre, November 2025 through June 2026. It is on Hugging Face, free, with the build scripts and a validator that runs 24 checks on every rebuild. This is the story of getting it.

## The market has no API

Freight exchanges are where European road pricing is actually discovered. Carriers post empty trucks, shippers post loads, and the two sides meet at a rate. That makes the exchange the closest thing the market has to a tape.

None of them offer a public API. They are browser products, built for dispatchers who sit in a tab all day. The data products that do exist are the exchanges’ own, sold back to the market at a price most of the market cannot pay. The page is not a workaround for a missing endpoint. It is the only door there is.

This is the shape of a great many valuable datasets. Not secret, not even hard to look at, just published in a form that assumes a person is doing the looking.

## What a hostile source actually looks like

Hostile rarely means a wall. It usually means a hundred small decisions that were reasonable for the site and ruinous for anyone reading it at volume. Every row in this table cost real time.

| What the source did | What it cost |
| :--- | :--- |
| Everything behind a login | Session state has to survive between runs, or every run begins at a sign-in form. |
| City names localised to Polish | Wiedeń, Mediolan, Praga. Vienna, Milan and Prague, in a feed that had to key to a gazetteer. |
| Rollup rows mixed into detail rows | A body_group of ALL sits beside the per-type rows. Sum without excluding it and every total is roughly double. |
| Postal codes in two formats | 110 00 in one table, 11000 in another. Normalising the non-alphanumerics lifted the match rate from 84.9% to 88.2%. |
| Country names in the city column | 1,970 rows said Germany where a city belonged. Another 2,400 said nothing at all. |
| Impossible measurements | A 2,513 tonne truck. 166 rows with bad payload, 93 with bad loading metres. |
| Codes from outside the continent | AF and PK, in a European freight feed. |
| An upstream that forgets | Source records are not retained past a window. Miss a week and that week is gone permanently. |

The last row is the one that changes the engineering. A source that forgets turns collection from a task into a standing obligation. You are not scraping a site once and cleaning up afterwards. You are running every week, forever, against a page that owes you nothing and will be redesigned without warning.

## Why an agent, and not a scraper

A hand-written scraper is a bet that the markup holds. It is a good bet most weeks and a total loss the week it fails, because the data behind a source that forgets cannot be backfilled. The cost of a broken selector is not an afternoon of maintenance. It is a permanent hole in the series.

Talona is built for the other shape. The first run goes through Aurora, our browser agent, which reads the page the way a person does and decides what to click. That run is recorded. Later runs are promoted from replaying the interface toward calling the endpoints the interface was calling all along.

The economics of that matter at this cadence. The expensive model-driven pass happens once, when the shape of the site is being learned. The cheap deterministic pass happens every week after. And when the site changes underneath the cheap path, the run falls back to the agent instead of failing, which is the difference between a gap in the data and a slower Tuesday.

> **On the authenticated session**
>
> Browser state lives in a profile rather than in a cookie jar bolted to a script. The practical effect is that the login survives restarts, redeploys and the six weeks between someone touching the job, which is most of why long-running collection breaks.

## From pages to something publishable

Collecting the data is half the work. Publishing it without exposing the people who posted it is the other half, and it is the half that decides whether a dataset can be open at all.

Every carrier identifier is gone: name, numeric id, ratings, review counts, feed row ids. There is no stable pseudonym either, so two rows from the same carrier cannot be linked back together. A row is published only where both its pickup coordinate and its resolved locality are used by at least five distinct carriers. Coordinates are city centroids, never depots.

The locality threshold is the one that earns its keep. Resolving postal codes to place names is finer grained than centroids, and without a threshold on it, 4,406 rows across 210 localities would have carried a single carrier each. 8,678 carriers are represented in the aggregate. An audit query asserts the floor on every build.

```rebuild the dataset from source
./build/build.sh dataset            # regenerate all six CSVs
python3 build/validate.py dataset   # 24 invariant checks, exit 1 on failure
psql "$PGURI" -f build/sql/audit_suppression.sql   # expects PASS
```

## What the market looks like

The overall level lands at €1.34 per kilometre, in line with published European benchmarks. That agreement is the real validation of the collection. A pipeline reading a page wrong does not accidentally reproduce a number the industry already knows.

The most useful structure in the data is directional. A truck running Germany to Poland prices around €1.07 per kilometre. The same truck running Poland to Germany prices around €1.45. That gap is not an error in the aggregation. It is backhaul imbalance: the leg heading back toward where the freight originates is bid down, because an empty return is worse than a cheap one.

- 34,558 weekly and 16,716 monthly rate observations, corridor by body type, volume weighted.
- 4,765 supply and demand cells. A ratio above 1 means more trucks than loads, which is leverage for shippers.
- 1,755 weekly diesel prices across 27 countries, from the EU Weekly Oil Bulletin, for pass-through analysis.
- 262 HGV driving bans. Weekend and holiday restrictions suppress capacity and make a clean covariate.
- The median vehicle is 24 tonnes and 13.6 metres, which is the standard EU semi-trailer.

> **Read this before modelling**
>
> These are asking rates posted to an exchange, not settled contract prices, so treat the series as a directional index rather than a price reference. Price coverage at the individual posting level is 1.9%, which is why rate analysis belongs on the aggregate files and the posting file is for geography and equipment.

## Take it

The dataset is public, with a full column dictionary, the cleaning notes, the de-identification detail and the SQL that produced every file.

```python
from datasets import load_dataset

rates = load_dataset("k4cperg/european-road-freight-spot-rates", "spot_rates_weekly")
trucks = load_dataset("k4cperg/european-road-freight-spot-rates", "truck_capacity")
```

[k4cperg/european-road-freight-spot-rates](https://huggingface.co/datasets/k4cperg/european-road-freight-spot-rates) on Hugging Face.

There is nothing exotic about this market. It is one of a very large number of them where the numbers are in public view, on a page, in the wrong language, behind a login, in a system that discards last month. If you have one of those, that is the thing we build for.