One of the biggest surprises for teams building with AI is not that it works.
It is how quickly it becomes expensive, slow, and difficult to scale.
What starts as a promising prototype often turns into a constrained system. Latency creeps in. Costs rise. Concurrency becomes limited. And suddenly, something that felt like a breakthrough is hard to roll out broadly across a product.
At a recent AIConf in Ahmedabad, Rajiv Mehta, a Machine Learning Specialist at Bacancy Technology and AWS Certified ML Specialist, explained why this happens. Getting a model to run is trivial. Getting it to run efficiently, at scale, and in a way that makes economic sense is where the real work begins.
For growth-stage companies, that distinction is everything.
Why the First Version Is Misleading
The reason this catches teams off guard is simple. The first version of any AI system usually works. It works in a notebook, in a demo, and often even with a handful of users. That early success creates a false sense of readiness.
What is invisible at that stage are the constraints that show up later. Memory limits, latency, concurrency, and cost all begin to compound as usage increases. What looked like a breakthrough quickly becomes a bottleneck.
Rajiv Mehta illustrated this with a simple but powerful comparison. The same 4B parameter model, loaded in a standard way, consumes significant memory and supports only a handful of users. Optimized correctly, that same model can handle an order of magnitude more users at significantly higher throughput.
Same model. Completely different outcome.
For growth-stage startups, this is the difference between a feature that works and a product that scales.
The Real Cost of Doing It the “Default” Way
One of the most important themes from Mehta’s session is that the default path is almost never the production path.
Most developers load models the simplest way possible using standard precision, standard libraries, and standard configurations. That approach is fine for experimentation, but it creates problems quickly when systems need to scale.
High memory usage limits concurrency. Slow throughput impacts user experience. Inefficient systems drive up infrastructure costs. For a growth-stage company, those are not minor issues. They directly affect margins, pricing, and the ability to expand AI-driven features across the product.
The key insight is that performance is not just about what the model can do. It is about how efficiently you run it.
Small Decisions, Massive Impact
What makes this space interesting is that the biggest gains do not come from changing the model. They come from changing how it is deployed.
Rajiv Mehta walked through a set of optimizations that, taken together, dramatically shift performance.
Quantization reduces memory footprint without meaningfully impacting output quality. Instead of consuming massive VRAM, models can run in a fraction of the space, unlocking far greater concurrency.
Memory management techniques like PagedAttention eliminate fragmentation and allow systems to use available resources far more efficiently. This becomes critical as workloads increase and systems move beyond simple use cases.
Inference engines also matter more than most teams realize. Tools like vLLM, llama.cpp, and others are purpose-built for serving models at scale. Using general-purpose frameworks leaves performance on the table, not because teams are doing something wrong, but because the tools were not designed for this use case.
Even at the compute level, optimizations like FlashAttention fundamentally change performance by reducing how often data needs to move between memory layers. This directly impacts latency and throughput, especially in real-time applications.
Individually, each of these decisions improves performance. Together, they completely change what is possible on the same hardware.
AI Is an Economics Problem as Much as a Technical One
One of the most important takeaways for growth-stage companies is that AI is not just a technical problem. It is an economic one.
Every token has a cost. Every millisecond of latency impacts user experience. Every inefficiency compounds as usage grows.
Rajiv Mehta highlighted how dramatically costs and performance can shift based on architecture decisions alone. Systems that are not optimized quickly become expensive to operate, limiting how broadly AI can be deployed across a product.
On the other hand, well-optimized systems unlock something much more valuable. They allow companies to scale AI capabilities without scaling cost at the same rate.
That is where real leverage comes from.
Avoiding Lock-In as You Scale
Another area Mehta emphasized is flexibility.
Most teams build directly against a single model provider’s API. It is fast to get started, but it creates long-term constraints. Switching models or adding new ones requires reworking large parts of the system.
The alternative is to introduce a routing layer that abstracts the underlying models. This allows teams to direct different types of requests to different models based on cost, complexity, or sensitivity.
Simple queries can be handled by smaller, faster models. More complex reasoning tasks can be routed to larger models. Sensitive workloads can remain on-premise.
This approach does more than improve performance. It gives companies control.
For growth-stage startups, that flexibility becomes increasingly important as products evolve and usage patterns change.
Where Most Teams Get It Wrong
If there is one takeaway from Mehta’s session, it is this.
Most teams over-index on the model and under-invest in everything around it.
As he put it, the model is roughly 20 percent of the solution. The inference engine, memory management, and routing architecture make up the other 80 percent.
That imbalance shows up everywhere. Teams spend time evaluating models, experimenting with prompts, and testing outputs, but they do not invest enough in the systems required to run those models effectively.
For growth-stage companies, this is a critical mistake. Because the challenge is not getting AI to work once. It is getting it to work consistently, efficiently, and at scale.
The Bottom Line
The hardest part of AI is not building something that works.
It is building something that keeps working as usage grows.
Rajiv Mehta’s session made that clear. The difference between a prototype and a production system is not the model. It is everything that surrounds it. Memory, inference, routing, and cost management all determine whether a system can scale.
For growth-stage companies, the opportunity is clear. The teams that invest early in how their systems run will be the ones that can deploy AI broadly and sustainably.
Because in the end, AI is not just about intelligence.
It is about execution.
To stay up-to-date on all upcoming York IE events, follow us on LinkedIn.
