Why Fintechs Are Replacing Hand-Built Features With Foundation Models
Manually engineered features don't transfer between fraud, credit, and recommendation models. A transformer trained on raw transactions can.
On this page
A bank processing billions of transactions faces the same modeling problem for fraud detection, credit scoring, and product recommendation: understand what a customer’s spending behavior actually means. The traditional answer is a team of analysts hand-designing features — average transaction size, spending category ratios, recency counts — one painstaking set per task, rarely reusable across tasks. The foundation-model answer borrows a lesson from language modeling: instead of hand-designing what matters, train a large model directly on raw sequences and let it learn general-purpose representations that transfer.
Why hand-engineered features hit a ceiling
Traditional financial ML pipelines transform raw transaction data into structured features by hand — income brackets, spending categories, transaction frequency — then feed those into a predictive model. This works, but has two structural limits that compound as an organization scales its ML efforts:
- Manual feature engineering doesn’t scale with team size. Every new feature requires domain expertise and iterative trial and error, and that cost is paid again for every new modeling effort.
- Features don’t generalize across tasks. A feature set carefully tuned for credit-default prediction has little relevance to a fraud-detection model or a product-recommendation engine, even though both are ultimately trying to understand the same underlying thing: how this customer behaves.
A foundation-model approach targets both limits directly: train once on raw data with no manual feature design, and produce a general-purpose representation — an embedding — that many different downstream tasks can reuse.
The architecture, end to end
- Transaction ingestion
Raw transaction data — amount, timestamp, merchant description — is collected per user as a time-ordered sequence, at a scale spanning trillions of transactions across over 100 million users.
- Tokenization via a hybrid encoding scheme
Each transaction is converted into a compact sequence of discrete tokens that preserves both structured and textual information — the hardest engineering problem in the whole pipeline, covered in detail below.
- Transformer backbone, trained self-supervised
The tokenized sequences feed a transformer trained with no labels at all, using Next Token Prediction and Masked Language Modeling — the same self-supervised objectives that pretrain large language models, applied to spending behavior instead of text.
- Downstream fine-tuning and fusion with tabular data
The pretrained model produces a fixed-length user embedding, which is then fine-tuned for specific tasks (credit default prediction, fraud detection) and combined with existing hand-engineered tabular features rather than replacing them outright.
The genuinely hard problem: turning a transaction into tokens
A transaction mixes structured fields (a numeric amount, a date) with free text (a merchant description) — and neither obvious encoding approach works well:
The chosen middle path: a hybrid encoding scheme that keeps each field’s structure explicit instead of flattening everything into text.
- Amount sign — one categorical token: positive (deposit) or negative (purchase).
- Amount bucket — the transaction’s magnitude, quantized into a manageable set of buckets, reducing numeric variance to a small vocabulary.
- Date tokens — separate tokens for month, weekday, and day of month.
- Merchant description — tokenized with a standard subword tokenizer (Byte Pair Encoding), so the model can recognize patterns in merchant names without needing to have seen that exact merchant before.
This keeps the token sequence compact (protecting against the quadratic attention cost) while remaining generalizable to transactions the model has never encountered — solving both failure modes of the two naive approaches at once.
The hybrid scheme isn’t a compromise for its own sake — it directly targets the two things that broke the naive approaches: token-count explosion (solved by keeping structured fields as single tokens instead of text) and cold-start failure on unseen transactions (solved by subword tokenization of the one genuinely open-ended field, the merchant description).
Learning without labels: what self-supervision actually teaches
With no manual labels required, the model trains on tasks it can generate from the data itself:
- Next Token Prediction: given a user’s transaction history so far, predict the next transaction — the same structure as a language model predicting the next word, applied to spending behavior. Coffee, then lunch, then — the model learns to guess dinner.
- Masked Language Modeling: hide random tokens in a sequence and train the model to reconstruct them, forcing it to learn relationships between context and content — the connection between a weekday and a spending category, or between a merchant type and a typical amount range.
Scale matters here in the same way it does for language models: larger models with longer context windows perform measurably better, and architectural choices compound — a switch to a larger causal transformer with optimized attention produced a real double-digit percentage-point gain on downstream task performance in practice.
Getting the new embeddings to actually help existing systems
A pretrained foundation model is only useful if it improves the predictions teams already care about — credit risk, fraud, recommendations — which almost always still rely on existing tabular features too. Two ways to combine them:
- Late fusion: freeze the pretrained embeddings, concatenate them with tabular features, and feed both into a traditional model like LightGBM. Simple to implement and reuses existing tooling, but the embeddings never adapt to the specific downstream task — there’s no interaction learned between the two data sources during training.
- Joint fusion: train the transformer and a tabular-feature model (an architecture like DCNv2, designed for structured inputs) together, end-to-end, on the same prediction target. More powerful because both components optimize for the same task jointly, at the cost of a more involved training setup.
Takeaway
The deeper lesson here isn’t “transformers work on transaction data too” — it’s that the same insight driving foundation models in language and vision applies wherever hand-engineered features are the bottleneck: a general-purpose representation, learned once from raw data via self-supervision, can outperform and outlast task-specific feature engineering that has to be rebuilt for every new use case. The hard engineering work isn’t the transformer itself — it’s the tokenization scheme that makes raw, messy, structured-plus-text data trainable at scale without losing the ability to generalize to what the model hasn’t seen yet.
Frequently asked questions
What's wrong with manually engineered features for financial ML models?
Manual feature engineering is time-intensive, requires deep domain expertise, and is brittle to changing patterns. Worse, features designed for one task — say, credit scoring — often don't transfer to a different task like fraud detection or recommendations, so teams end up duplicating feature engineering effort across every new model they build, with none of the work compounding.
How do you turn a bank transaction into something a transformer can process?
A transaction has structured fields (amount, date) and free text (merchant name) — neither a pure-text nor a pure-structured encoding handles both well. A hybrid scheme tokenizes each field separately: a categorical token for the amount's sign, a bucketed token for its magnitude, separate date tokens, and subword tokens (like Byte Pair Encoding) for the merchant description — producing a compact sequence that preserves structure without ballooning token count.
Why not just assign each unique transaction its own ID, like a recommendation system does with products?
Transactions are far too varied for that — nearly every combination of merchant, amount, and date is unique, so the ID space would be enormous and full of transactions the model has never seen during training. The model would fail on any transaction it hasn't memorized an ID for, which is a hard cold-start failure, unlike a hybrid tokenization scheme that generalizes to unseen combinations.
What is self-supervised learning doing when there's no labeled data?
Instead of labels, the model is trained on prediction tasks it can generate from the raw sequence itself: Next Token Prediction (predict the next transaction given the history) and Masked Language Modeling (guess transactions that were hidden from the sequence). Both objectives force the model to learn real structure in spending behavior — recurring payments, spending cycles — without any human labeling a single example.
What's the practical difference between late fusion and joint fusion when combining embeddings with tabular data?
Late fusion freezes the pretrained embeddings and feeds them alongside tabular features into a separate model like LightGBM — simple, but the embeddings never adapt to the specific downstream task. Joint fusion trains the transformer and the tabular model together end-to-end, letting the transformer learn to extract exactly the signal that complements the tabular features for that specific prediction task, which is more powerful but requires a more involved training setup.


/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.