Geospatial Foundation Model is trained on large, generalized, unlabelled geospatial data for downstream tasks. With many similarities to LLMs, there are some key differences with respect to the nature of geospatial applications: multimodality of input and output data, and geospatial data representation.
Trajectory Foundation Models aim at the task of generating a trajectory from a learned prior. However, the paradigms for representation and training are diverse. Some open research questions include if the destination should be in the context, how granular the representation should be, and should generation be based on pure GPS locations or road networks.
There is no agreement in the research community on what the foundation model should be, most leaning towards multiple foundation models based on classes of downstream applications.
Routing is a problem of finding the optimal path for a vehicle to visit a set of customers $N$. State-of-the-art solutions use solvers to find the shortest path, or as people may refer to it, the traveling salesman problem (TSP) - and the representation of a path is less granular, where only stop locations are treated as token.
While solvers have evolved to be fast and efficient, one challenge is that 80% of commercial drivers do not follow the route. This, in the foundation model context, is interpreted as driver tacit knowledge. A solver can only represent fixed real-world priors, which foundation models are actually good at capturing. Put another way, the solver’s planned theoretical optimal path may not be the real-world optimal, but experienced drivers tend to pick a satisfactory path that they are actually going to take.
Traditional optimization relies on fixed objective functions and constraints. In real-world applications, however, these fixed functions act as a “lossy compression” of complex reality.
If we represent the real-world objective function as $C_{real}(x)$ and the solver’s objective as $C_{solver}(x)$, the difference is the Optimality Gap, representing the driver’s tacit knowledge $\mathcal{T}(x)$:
\[C_{real}(x) = C_{solver}(x) + \mathcal{T}(x)\]Under scaling laws, as model capability increases, this optimality gap should decrease and eventually smooth out at a plateau of $x\%$. This $x\%$ directly represents the captured tacit knowledge.
Current SOTA trajectory prediction models (like Prt-Net, Pointerformer, and DIFUSCO) rely heavily on Reinforcement Learning paradigms. The justification is usually the lack of massive, labeled real-world datasets. Consequently, most of these models are trained on synthetic data, which fails to reflect actual operational realities.
Instead of human-engineered rules, we should lean into The Bitter Lesson: simple paradigms paired with massive compute will always outperform complex, hand-crafted logic.
The Scaling Mindset:
Scaling laws essentially predict the relationship between model parameters $N$, training data $D$, compute $C$, and the test loss. By feeding a simple autoregressive model massive amounts of raw fleet data, we simplify the task: prove that scaling works for trajectory prediction better than RL.
Unlike LLMs, which predict the next semantic ID (a high-dimensional compression of human language), routing predicts the next physical state. Every customer visit possesses a state (average dwell time, local road networks, ETAs, etc.).
If we apply the philosophy of “Token is all you need,” we must address three design questions:
Tokenization standardizes the foundation model on discrete outputs. While using raw GPS coordinates creates engineering and continuous-value modeling challenges, tokenization offers significant privacy and engineering advantages. The tradeoff is quantization fragmentation (results are only as precise as the token grid), but for macro-routing, this is acceptable.
We need a minimal, non-derivable set of information:
[L3][L4][L5][L6][L7][L8][L9]. Geohash is simpler and has a built-in hierarchy compared to S2 or H3, making it the most straightforward architecture for a transformer to learn spatial relationships.[Dwell]: How long the vehicle stops.[TT]: Travel time from the last point.[DOW] (Day of Week) and [TOD] (Time of Day) for cyclical signals.[WC] and vehicle type [VT].[BOS] (Begin), [EOS] (End), and [Newday] to segment continuous data.The high-level autoregressive sequence looks like this:
[BOS]
[WC] [VT]
[Newday] [DOW]
[L3] [L4] [L5] [L6] [L7] [L8] [L9] [TT] [TOD] [Dwell]
[L3] [L4] [L5] [L6] [L7] [L8] [L9] [TT] [TOD] [Dwell]
...
[EOD]
How do we use this for actual last-mile delivery (e.g., Amazon Last Mile Challange Dataset 2021)?
If we just let the model run, it’s doing “Free Inference”—generating synthetic trajectories. But routing is a combinatorial TSP problem. To solve it, we use Constrained Decoding. If we are restricted to 200 specific locations, we simply mask all other locations during generation.
This feeds into a Hybrid Search architecture:
When defining a “document” for training, we use a fixed context size (e.g., 8192 tokens) tracing back up to 7 days of historical fleet movement.
But there’s a real-world data glitch: what happens when a vehicle stays at a depot, loading and unloading, generating multiple consecutive GPS pings? In an LLM, this is like the model stuttering: “China is so so so so fun!” In routing, this creates massive noise because the model doesn’t need to know you moved 5 meters ten times inside a depot; it just needs to know you arrived, stayed for 2 hours, and left.
A simple ST-DBSCAN clustering won’t work because time windows are long-tail distributions (depot stops are naturally long), and DBSCAN will split them. The solution is a Gaps and Islands approach combined with spatial clustering. If there are continuous stops within an [L7] Geohash (approx 153m x 153m), we weight them into a single token and aggregate the [Dwell] time. This cleans the trajectory so the model learns true sequential routing without the noise.