Explanation
Situation
-
Enterprise Adoption: At Deloitte, while working with enterprise stakeholders across AWS, Azure and GCP, I saw that different teams were starting to build GenAI and RAG applications independently. That created several architecture problems:
- inconsistent identity and security controls,
- different approaches to RAG,
- fragmented observability,
- difficulty controlling model usage and cost, and
- challenges in standardizing deployment across cloud environments.
-
Resuability: The requirement was therefore not simply to build another GenAI application. We needed a reusable enterprise GenAI foundation that application teams could consume.
-
Economics: The other challenge was economics. LLM inference was start taking a significant operational cost at enterprise scale, so we needed to design the platform with cost optimization as an architectural concern rather than treating it as an afterthought.
Task
-
Reusability: My responsibility was to architect a reusable Enterprise GenAI Cloud Platform that could support RAG and agentic AI workloads while providing enterprise-grade security, centralized identity, governance, observability, and operational controls.
-
It also needed to support multi-cloud environments without forcing every application team to reinvent the platform architecture.
-
Stakeholders: I also needed to work with product and executive stakeholders to translate business requirements into a scalable architecture and establish reusable engineering standards that multiple teams could adopt.
-
FinOps: A major requirement was to make the platform economically viable at scale, so I also took ownership of the FinOps aspects of the GenAI architecture.
Action
-
Common GenAI Architecture: I started by defining a common GenAI architecture rather than allowing each team to build an independent solution.
-
RAG Architecture: At the application layer, I defined a standard RAG architecture covering document ingestion, chunking, embeddings, vector search, retrieval, prompt orchestration, and LLM response generation, enabling the platform to support RAG-based applications.
-
AI Service Layer: For the AI service layer, I designed architectures using Amazon Bedrock and Azure AI, allowing teams to consume enterprise AI capabilities through a common architectural pattern.
-
I then focused heavily on the enterprise control plane.
-
Enterprise Security: For identity and enterprise security, I incorporated zero-trust IAM / RBAC secrets management, encryption, network isolation, and policy-based access controls, with AI governance guardrails and AI governance guardrails. Access to AI services and enterprise data was governed through identity and policy controls rather than giving applications unrestricted access.
-
Governance and Observability: I also designed the platform with centralized governance and observability so that platform teams could understand how AI applications were being used and operated.
-
GenAI Economics: A particularly important part of my contribution was the economics of GenAI.
-
FinOps Initiatives: I spearheaded FinOps initiatives and introduced semantic caching to reduce unnecessary repeated LLM inference. The objective was to avoid sending semantically equivalent requests to the model when an appropriate previous response could be reused.
-
Cost Optimization by Design: This became part of the architecture rather than an operational optimization added later.
-
Stakeholder Alignment: I also worked with product and executive stakeholders to translate business requirements into architecture decisions and establish reusable engineering standards.
-
Technical Leadership: As the platform evolved, I provided technical leadership across four distributed teams and more than 25 engineers, helping establish common GitOps practices, architecture standards and runbooks.
Result
-
Enterprise Foundation: The platform established a reusable enterprise foundation for GenAI workloads, including RAG and AI-agent architectures, with security, governance and operational controls built into the architecture.
-
Cost Reduction: The most measurable outcome was the 80% reduction in LLM inference costs achieved through the FinOps initiatives and semantic caching.
-
Platform Transformation: Beyond the cost reduction, the larger architectural outcome was that GenAI moved from individual application experiments toward a reusable enterprise platform model.
-
Foundation for Agentic Workloads: That platform subsequently provided the foundation for agentic workloads such as the Enterprise Autonomous Engineering Platform.
-
Key Lesson: The key lesson I took from this work was that enterprise GenAI architecture is not just about selecting an LLM. I have to architect the complete system around it—data retrieval, identity, security, governance, observability and economics.
