Production experience / Walmart
Serving models.
Scaling search.
Moving Search from single-tenant clusters to shared model-serving infrastructure, with routing across tenants, locales, and serving paths.
01 / Architecture
From clusters
to a platform
The constraint
I helped drive the move from single-tenant Kubernetes model-serving clusters to a shared multi-tenant architecture. Search already served hundreds of millions of daily queries, so improving infrastructure efficiency and scalability had to preserve latency, availability, and a controlled blast radius.
The decision and trade-off
The core decision was how much tenant isolation to preserve. Keeping every workload in its own cluster minimized blast radius, but retained the operational and infrastructure overhead we were trying to remove.
I rejected full cluster isolation as the long-term architecture. I chose strong isolation around tenant compute and resource usage, while sharing selected platform capabilities, such as routing, where the efficiency benefit justified it.
The rollout and failure handling
I rolled the change out incrementally with the team, starting with lower-risk workloads and locales. We validated isolation, routing, and failover behavior at each stage before expanding to higher-volume traffic.
I treated the fast and deep-inference paths separately so pressure in the more expensive execution path would not unnecessarily affect unrelated low-latency traffic. Staged deployments and rollback practices were part of the rollout from the start.
Request context
Tenant + localeContext for a search request.
Routing
Multi-tier inferencePath selection and failover across environments.
Serving path
Low-latency cacheServing path
Deep model executionShared routing sits alongside strong tenant compute and resource isolation. The fast and deep-inference paths are treated separately to limit the effect of pressure in expensive execution. This view leaves internal thresholds and topology out.
02 / Personal ownership
The platform
and its practices
What I contributed
- Helped drive the multi-tenant migration, choosing tenant resource isolation with selected shared platform capabilities.
- Designed routing between cache serving and deep model execution, with failover across environments.
- Rolled the change out incrementally with the team, validating isolation, routing, and failover before expanding traffic.
- Built query normalization and tokenization adopted across Search teams.
- Established architecture governance and distributed systems training.
Work that extended across teams
Query normalization and tokenization became shared capabilities across Search teams. Architecture governance and training extended the scope beyond the serving implementation itself.
03 / Platform outcomes
Results in
their context
The broader platform work reached 99.9% availability and reduced inference latency by 25%. Deployment practices associated with the architecture contributed to a 40% improvement in deployment cycle time.
- 25%
- lower inference latency
- 99.9%
- model-serving platform availability
These are outcomes of the broader platform work. My decision, routing, and rollout contributions are described separately above.
Read the full experienceContinue the conversation
Building a
shared platform?
Let’s talk about search infrastructure, platform reliability, and the engineering practices that support them.