Airfare data is not a price: what makes flight fare aggregation hard

A price on an e-commerce page is a field on a record. Someone typed it, it sits in a database, and if you read it twice in an hour you get the same answer twice. Almost every instinct people bring to competitive pricing data comes from that world, and almost none of it survives contact with airline fares.

A fare is not stored anywhere. It is computed, on demand, in response to one specific question — this route, these dates, this many passengers, this cabin, bought from this country in this currency through this channel — and it is true for about as long as it takes to read it. Change any term in that question and you get a different number, legitimately. There is no single price for a seat. There is only the answer to the query you actually asked.

Everything difficult about this subject comes from that one fact.

A fare is a function, not a value

Ask a pricing engine for a Tuesday departure and a Wednesday departure and you have asked two unrelated questions. The same is true across nearly a dozen dimensions, and each one is load-bearing:

• Origin, destination and date — including the return date, because a round trip is not two one-ways and is frequently cheaper than one of them.
• Passenger count and mix. Two adults is not twice one adult. Availability is consumed in blocks, and asking for four seats can push you into a higher booking class than asking for one.
• Point of sale. The same seat is routinely priced differently for a buyer in Singapore and a buyer in Germany. Not a bug — deliberate market segmentation.
• Currency, which is not simply the POS price converted, because the fare may be filed separately in each.
• Channel. Direct, agency, metasearch and corporate all see different inventory and different fares, and the direct price is often not the cheapest one.
• Cabin, booking class and fare basis — three different things, discussed below, that people routinely treat as one.
• Ancillaries. A headline fare without bags, seat selection or a change fee is not comparable to one with them, and the gap can exceed the difference you were trying to measure.

A naive collector picks one value for each of these, holds them fixed, and produces a tidy daily series. The series is internally consistent and describes a market nobody is buying in.

The combinatorial problem

This is the part that kills most in-house efforts, and it is just arithmetic.

Take a modest brief: two hundred routes, shopped across a 120-day booking window, refreshed twice a day. That is 48,000 quotes a day, for one competitor set. Add three cabins and two passenger configurations and you are at 288,000 — and you have not yet added a second point of sale, or the return-date dimension, which multiplies rather than adds.

Routes × dates × 2 refreshes

48K/day

With cabins and pax mix

288K/day

Useful life of a quote

Minutes

Every one of those is a live request to a system that does not want to serve it at that volume. The engineering problem is not "can we read a fare off a page". It is scheduling six figures of queries a day against sources with rate limits and bot defences, prioritising the ones whose answers have actually changed, and finishing before the answers go stale — which is the real constraint, because a fare captured eight hours ago is not a slightly worse data point. It is a wrong one.

Four things that break a naive scraper

The quote expires. Caching is the normal answer to volume and here it is poison. If your pipeline retries a failed batch four hours later and stamps the results with the run time, you have silently mixed two markets into one row set and nothing downstream can tell. Capture time and run time have to be separate fields, always.

Booking class is not cabin. Cabin is where you sit. Booking class is a single-letter inventory bucket inside that cabin — each with its own price and its own small allocation of seats. Fare basis is the rule being sold, including whether it can be changed or refunded. A competitor showing a fare $40 below yours in a bucket with two seats left is not undercutting you; they are advertising. Inventory depth is what separates a real price from a billboard, and a collector that records only the number cannot tell the difference.

Point of sale and currency shift under you. Many sources infer POS from the requesting IP address. Change where your infrastructure runs — or let a provider rotate you somewhere new — and your historical series develops a step change that looks exactly like a competitor repricing. Pinning POS deliberately, and recording it on every row, is the only defence.

The source changes without warning. Layouts change, fields move, bot defences escalate, and none of it is announced. A pipeline that assumes stability does not fail loudly; it fails by returning fewer rows, or nulls, or last week's structure parsed into the wrong columns. The engineering that matters is the monitoring that notices a 4% drop in expected row count on a Tuesday, not the parser itself.

Where the data actually comes from

There is no single source of truth, and anyone who tells you otherwise is selling one source.

• GDS gives broad, structured coverage and is the closest thing to a canonical view of filed fares, but it is expensive per query and does not see direct-only or channel-specific pricing.
• Airline direct and NDC gives the fare the airline actually wants to sell, including the offers that never reach a GDS — and has to be collected carrier by carrier, each with its own defences.
• OTAs show what is being merchandised to real buyers, including agency markups and private fares, which is sometimes the number your customer is actually comparing you against.
• Metasearch is the cheapest coverage per route and the least trustworthy per data point: you see the lowest fare some subset of agencies chose to publish at that moment, with no inventory depth behind it.

A usable feed blends them and records which source each row came from, because the right answer depends entirely on the question. "What is the market" and "what is the buyer seeing" are different questions with different correct sources.

A worked example

A regional carrier flies one route with a 180-seat aircraft. Their average fare on it is $240 and they are running a 31% load factor — 56 seats sold, around $13.4K of revenue per departure.

The commercial team's read is that the route is soft and the fare is already generous. What the fare data shows is narrower and more actionable: a competitor added four weekly frequencies on the route six weeks earlier and repriced the advance-purchase window beneath them. The carrier is sitting $51 above the market in the 21-to-60-day booking band, which is where most of the demand on this route commits. They are not expensive on the day of travel. They are invisible two months out, which is when the seats are actually sold.

Repricing into the market at $189 fills 158 of 180 seats — an 88% load factor and about $29.9K per departure.

Priced blind

$13.4K

Priced on market data

$29.9K

Per departure

+122%

A 21% lower fare produced 122% more revenue. That arithmetic only works because the inventory is perishable: the aircraft flies with the empty seats aboard, and an unsold seat is not inventory carried to tomorrow. It is a write-off that happens on a schedule.

The point of the example is not the size of the number. It is that none of the information needed to find it is available from a fare captured once a day on the departure date. You need the booking curve, the competitor's whole advance-purchase ladder, and the capacity change that explains both.

What good looks like in a feed

After enough of these, the specification converges. A fare feed is usable when it has:

• Capture timestamp distinct from run timestamp, on every row, with no exceptions.
• A stable schema. The column set does not change shape because a source page did. Breaking changes are versioned and announced, not discovered.
• Source attribution per row — GDS, direct, OTA, metasearch — so a consumer can weight them.
• Cabin, booking class and fare basis as separate fields, plus inventory depth wherever it can be observed.
• POS and currency as queried, recorded rather than inferred later.
• Deduplication, because the same fare reached through two paths is one observation, and counting it twice quietly distorts every average built on top.
• Backfill on failure, so a bad afternoon shows up as a gap that gets filled rather than as a dip in the market that never happened.
• The same shape every morning, which sounds like the least interesting item here and is the one pricing teams care about most, because it is what lets them automate anything.

That list is the actual product. The scraping is a prerequisite, not the work — the same way the hard part of hotel rate data turns out to be matching room types rather than reading a number off an OTA.

---

We build and run this as a feed: air fare intelligence, by route, cabin and booking class, delivered on the cadence you reprice on. If you would rather have the engineers than the feed, that is forward-deployed engineering, and it is the same team.