Story 2 — 80% LLM Inference Cost Reduction

This should be your primary Frugality + Dive Deep + Deliver Results story. Your resume explicitly states that you spearheaded FinOps initiatives and semantic caching, reducing LLM inference costs by 80%.

The key is to present this as an architecture and engineering optimization problem, not simply as "we added caching."

1. Questions this story can answer

Primary question

"Tell me about a time when you reduced costs significantly."

Other questions

• Tell me about a time you demonstrated Frugality.
• Tell me about a time you Dived Deep into a problem.
• Tell me about a time you delivered a significant measurable result.
• Tell me about a time you optimized an architecture.
• Tell me about a time you identified an inefficiency others hadn't noticed.
• Tell me about a time you used data to make an engineering decision.
• Tell me about a time you had to balance cost and performance.
• How would you optimize LLM costs?
• How do you approach GenAI FinOps?
• How would you reduce token consumption?
• How would you design cost controls for an enterprise GenAI platform?
• Tell me about a difficult technical trade-off.
• Tell me about a time you challenged the existing architecture.
• Tell me about a time you improved an AI system without changing its core functionality.

2. The story in one sentence

Memorize this:

"I identified that repeated and semantically similar GenAI requests were unnecessarily triggering LLM inference, so I introduced a FinOps-driven optimization approach centered around semantic caching and consumption analysis, which reduced LLM inference costs by approximately 80%."

3. Full STAR answer

S — Situation

"While working on enterprise GenAI solutions, I identified a challenge that is easy to underestimate during the proof-of-concept phase: LLM inference cost can grow rapidly once an AI application starts operating at enterprise scale.

During an initial implementation, the focus was naturally on functionality — making sure users could interact with the AI system and receive useful responses.

But as usage increases, every request can potentially result in model inference, and the cost isn't determined only by the number of requests. It can also be affected by input tokens, output tokens, model selection and repeated requests for similar information.

I recognized that if we simply scaled the original architecture, we could end up paying the model to repeatedly solve essentially the same problem.

So I treated LLM cost as an architectural concern rather than something that should be addressed after the system was already in production."

4. T — Task

"My responsibility was to identify where we were unnecessarily consuming LLM capacity and design an optimization approach that significantly reduced inference cost without materially compromising the usefulness of the GenAI application.

I wanted to avoid a simplistic solution such as immediately switching to a smaller or less capable model.

The objective was to first understand the workload and determine whether we could avoid unnecessary inference altogether.

I therefore approached the problem from a FinOps perspective: measure consumption, identify repeated patterns, determine where inference was unnecessary, and then introduce architectural controls to reduce those calls."

5. A — Action

Step 1 — Analyze the consumption pattern

"I started by looking at the workload from a consumption perspective.

Instead of treating every LLM request as unique, I wanted to understand whether users were asking identical or semantically similar questions.

This distinction was important because traditional exact-match caching would only help when the request was character-for-character identical.

In enterprise AI applications, users often express the same intent in slightly different ways.

For example, two users might ask:

'What is the company's remote-working policy?'

and:

'Can you tell me the policy for working remotely?'

The wording is different, but the underlying information need may be substantially the same."

6. Step 2 — Introduce semantic caching

"Based on that observation, I introduced semantic caching as part of the optimization strategy.

Instead of only checking whether the incoming request exactly matched a previous request, we could represent the request semantically and determine whether a sufficiently similar request had already been processed.

If an appropriate cached response existed and met the required similarity and freshness conditions, we could return that result instead of invoking the LLM again.

This allowed us to reduce unnecessary inference while preserving the existing application experience."

7. Step 3 — Don't cache everything

This is an important Dive Deep point.

If the interviewer asks:

"Would you cache every LLM response?"

Answer:

"No. Semantic caching needs guardrails.

I would only cache responses where the underlying information is sufficiently stable and where returning a previous response doesn't introduce unacceptable correctness or security risk.

Dynamic, personalized, sensitive or rapidly changing requests may need to bypass the cache.

Similarly, the similarity threshold needs to be tuned carefully. A low threshold can produce incorrect matches, while a threshold that's too strict reduces the economic benefit.

So the caching strategy needs to consider semantic similarity, freshness, data sensitivity, personalization and response quality."

This answer shows that you understand the trade-off, rather than thinking caching automatically equals savings.

8. Step 4 — Preserve quality

The interviewer may ask:

"How did you make sure cost optimization didn't hurt quality?"

Answer:

"I treated quality as a constraint rather than optimizing cost in isolation.

The goal wasn't simply to maximize the cache hit rate. A cached response is only valuable if it is appropriate for the current request.

So the optimization needs to balance several metrics: cache hit rate, response relevance, freshness, latency and cost per request.

If a cached response doesn't meet the required quality or freshness criteria, the request should go back to the normal inference path."

9. Step 5 — Make cost observable

This is where your broader GenAI observability experience becomes useful.

Your resume says you designed AI observability covering:

• application telemetry
• model/API latency
• token consumption
• errors
• usage patterns
• cost monitoring.

Use that:

"I also made cost observable at the application level.

Traditional infrastructure monitoring can tell you CPU utilization, memory utilization and service health, but those metrics don't explain why an LLM workload is becoming expensive.

So I looked at AI-specific consumption signals such as token usage, model/API latency, request patterns and cost.

This allowed the team to understand not just whether the system was operational, but whether it was economically sustainable."

10. Step 6 — Think beyond caching

This is important for a Principal Solutions Architect answer.

"I also looked at the problem as a broader architecture optimization exercise rather than assuming semantic caching was the only lever.

Depending on the workload, other optimization dimensions include reducing unnecessary prompt content, controlling output length, choosing an appropriate model for the task, reducing redundant retrieval, batching where appropriate, and avoiding unnecessary agent or model calls.

My principle was to first eliminate unnecessary inference and then optimize the inference that remained."

That last sentence is excellent:

"First eliminate unnecessary inference; then optimize the inference that remains."

11. Step 7 — Make it reusable

"Rather than treating the optimization as a one-off change to one application, I looked at how the pattern could become part of the broader GenAI platform architecture.

The objective was to make FinOps a standard architectural concern so that new AI workloads would consider cost, observability and consumption from the beginning."

This connects to your resume's broader platform-engineering work.

12. R — Result

This is where you give your strongest number.

"The result was approximately an 80% reduction in LLM inference costs.

Beyond the direct financial impact, the work changed how we approached enterprise GenAI architecture. Cost became an explicit design dimension alongside security, scalability, reliability and performance.

The key lesson for me was that GenAI FinOps isn't simply about negotiating model pricing or choosing the cheapest model. A large portion of optimization comes from architecture — understanding when you actually need inference and eliminating unnecessary model calls."

Your resume explicitly states the approximately 80% reduction in LLM inference costs from FinOps and semantic caching.

13. Strong final reflection

End with:

"The biggest lesson I took from this was that frugality and customer experience don't have to conflict.

If you optimize the architecture intelligently, you can reduce cost while potentially improving latency and scalability at the same time.

For me, that is the right way to approach FinOps: don't simply ask, 'How can I spend less?' Ask, 'What work doesn't need to happen in the first place?'"

That's a strong Amazon-style conclusion.

14. 2-minute version

If the interviewer asks for a concise answer:

"One example was an enterprise GenAI workload where I identified that LLM inference costs could become a significant scaling concern.

The problem was that repeated and semantically similar requests were triggering model inference even when the underlying information need was essentially the same.

I took ownership of the optimization from a FinOps perspective. I analyzed consumption patterns and introduced semantic caching so that sufficiently similar requests could reuse appropriate previous responses rather than invoke the LLM again.

I was careful not to treat caching as a blanket solution. We needed to consider semantic similarity, freshness, sensitivity and response quality, because an incorrect cached response can be worse than paying for another inference.

I also incorporated AI-specific observability around token consumption, request patterns, latency and cost so that we could measure the economic behavior of the workload.

More broadly, I looked at other optimization levers such as reducing unnecessary model calls and optimizing the amount of information sent to the model.

The result was approximately an 80% reduction in LLM inference costs.

The key lesson for me was that GenAI FinOps is fundamentally an architecture problem. Rather than simply choosing a cheaper model, I prefer to first eliminate unnecessary inference and then optimize the inference that remains."

15. Technical deep dive: How semantic caching works

If the interviewer suddenly goes technical:

"Explain semantic caching to me."

Answer:

"With traditional caching, the cache key is usually based on an exact request or deterministic parameters.

Semantic caching adds an embedding-based similarity layer.

Conceptually, the flow is:

User request → embedding → similarity search → threshold evaluation → cache hit or LLM inference.

The incoming request is converted into an embedding representation. We then search for semantically similar previous requests.

If the similarity score exceeds the configured threshold and the cached response satisfies freshness and policy requirements, we return the cached response.

Otherwise, we invoke the normal RAG or LLM pipeline and potentially store the resulting response for future requests.

The important part is that the threshold isn't purely a technical parameter. It's a business and quality trade-off. A threshold that's too permissive increases the probability of returning an inappropriate answer, while one that's too restrictive reduces the cache hit rate and therefore the financial benefit."

16. Where would caching sit in a RAG architecture?

A strong answer:

User
  │
  ▼
API / Application
  │
  ▼
Semantic Cache
  │
  ├── Cache Hit ───────────────► Response
  │
  └── Cache Miss
          │
          ▼
       RAG Pipeline
          │
          ├── Retrieve documents
          │
          ├── Context construction
          │
          ▼
       LLM Inference
          │
          ▼
       Response
          │
          ▼
     Cache if eligible

Then say:

"The exact placement depends on what I'm caching. I could cache at different layers, but I would be careful about caching the final answer when the underlying knowledge is dynamic. In some cases, caching embeddings or retrieval results may be safer and still reduce downstream work."

17. Interviewer: "What are the risks of semantic caching?"

Answer:

"There are several.

First, correctness. A semantically similar query isn't necessarily asking the same question.

Second, freshness. The underlying enterprise data may have changed.

Third, security. A cached response must not cross authorization boundaries between users or groups.

Fourth, personalization. Responses containing user-specific context shouldn't be shared across users.

Fifth, cache poisoning. An incorrect or malicious response shouldn't become a reusable cached answer.

So I would design cache keys and access controls around tenant, identity, authorization context, data classification and freshness requirements where appropriate."

This is particularly strong for an enterprise AWS SA interview.

18. Interviewer: "How would you calculate the ROI?"

Answer:

"I'd start with baseline inference economics.

I'd measure:

• requests per day
• average input tokens
• average output tokens
• model pricing
• cache hit rate
• inference latency
• quality impact

Then I could compare:

Baseline inference cost

versus

Post-cache inference cost + cache infrastructure cost + any additional operational cost.

I would also look at the business value of latency improvements and capacity reduction.

Importantly, I wouldn't declare success based purely on cost reduction. I'd want to verify that quality, freshness and security remained within the required thresholds."

19. Interviewer: "Why not just use a smaller model?"

Excellent answer:

"That can absolutely be one of the optimization levers, but I wouldn't make it the first or only lever.

If the application is unnecessarily invoking a model for requests that could have been served from a valid cached result, switching to a smaller model still means we're paying for inference that wasn't necessary.

My approach is hierarchical:

First eliminate unnecessary work → then optimize necessary work.

That could mean semantic caching first, then prompt optimization, retrieval optimization, model selection, token controls and other workload-specific optimizations.

And any model change should be validated through evaluation to make sure we aren't trading away too much quality for cost."

20. Interviewer: "How would you optimize an agentic AI workload?"

This is very relevant to your AWS JD.

Answer:

"Agentic workloads introduce another dimension because a single user request can result in multiple model calls and tool invocations.

I would first establish observability around the complete agent trajectory: number of model calls, tokens per step, tool calls, retries, latency and cost.

Then I'd identify unnecessary reasoning loops, redundant tool calls and repeated retrieval.

I'd introduce caching where appropriate, constrain agent loops, use deterministic tools for deterministic operations, select models according to task complexity and establish maximum budgets for tokens and execution steps.

I would also evaluate whether every decision actually requires an LLM. If a deterministic rule can safely handle a decision, I would prefer the deterministic mechanism.

Finally, I'd combine those optimizations with quality evaluation so that cost reduction doesn't come at the expense of agent reliability."

21. Interviewer: "How would you design this on AWS?"

You can give this answer:

"On AWS, I would start with Amazon Bedrock for model access and use application-level observability to capture request, token, latency and cost metrics.

For semantic caching, I'd use an appropriate vector-capable data store to represent and retrieve semantically similar requests, with cache entries governed by freshness, authorization and data-classification requirements.

For RAG workloads, I'd also look at whether retrieval results or intermediate computations could be reused safely rather than repeatedly performing the same work.

I'd instrument the application so that cost could be attributed by application, customer, model, use case and environment.

For production, I'd combine those controls with IAM, encryption, auditability, infrastructure as code and evaluation pipelines."

Again, unless you actually used a particular AWS service for this project, present it as your AWS design approach, not as something you previously implemented.

22. Interviewer: "What metrics would you monitor?"

Memorize this table:

23. Interviewer: "What if the 80% cost reduction reduced quality?"

Answer:

"Then I wouldn't consider the optimization successful.

The objective is not minimum cost; it's the best economic outcome within defined quality, security and reliability constraints.

I would establish a quality baseline before the optimization and compare it afterward using an evaluation dataset and production metrics.

If quality degraded beyond the acceptable threshold, I would adjust the semantic similarity threshold, restrict which requests could use caching, change freshness policies or remove caching for that use case.

I'd rather achieve a smaller verified cost reduction than claim an 80% reduction while degrading the customer experience."

This demonstrates excellent judgment.

24. Amazon Leadership Principles covered

Frugality

The obvious one.

You achieved an 80% cost reduction.

Dive Deep

You didn't just observe a large bill. You investigated the underlying consumption behavior.

Deliver Results

You have a concrete measurable outcome.

Ownership

You identified the problem and drove the optimization.

Invent & Simplify

You changed the architecture rather than relying solely on operational cost-cutting.

Customer Obsession

You protected quality and user experience while optimizing cost.

Highest Standards

You considered security, freshness and correctness in the caching design.

25. What NOT to say

Avoid:

"We added Redis and reduced the cost by 80%."

Your resume says FinOps + semantic caching, not that Redis was specifically used.

Avoid:

"We reduced tokens by 80%."

Your resume says LLM inference costs were reduced by 80%, which is different.

Avoid:

"Semantic caching alone reduced costs by 80%."

Your resume says:

"Spearheaded FinOps initiatives and semantic caching, reducing LLM inference costs by 80%."

So keep the attribution accurate:

"Through FinOps initiatives including semantic caching, we reduced LLM inference costs by approximately 80%."

26. Your story cheat sheet

Before the interview, remember these 7 points:

Problem: LLM inference becomes expensive at scale.

Insight: Not every request requires new inference.

Action: Analyze consumption + semantic caching + broader FinOps optimization.

Architecture: Request → semantic similarity → cache decision → inference if necessary.

Controls: Quality + freshness + security + authorization.

Result: ~80% reduction in LLM inference costs.

Lesson:

"First eliminate unnecessary inference; then optimize the inference that remains."

Your resume also establishes broader AI observability around token consumption, latency, errors, usage patterns and cost, which makes this story particularly useful for the AWS role's production-AI focus.