Engineering
Decoupling AI Infrastructure
Migrating Shiv from Gemini Developer API to Vertex AI Without Changing the Product
Engineering Case Study • AI Infrastructure • Cloud Architecture
Built while developing Learnzy.
| Project Type | Engineering Case Study |
| Category | AI Infrastructure · Cloud Architecture · Production Migration |
| Built For | Learnzy |
| Author | Himanshu Gupta |
| Status | Production |
| Part of | LearnzyLabs |
Executive Summary
Shiv originally used the Gemini Developer API through a Supabase Edge Function.
The architecture was intentionally simple during the early stages of Learnzy.
flowchart TD
A[Android] --> B[Supabase Edge Function]
B --> C[Gemini Developer API]
This allowed rapid iteration while validating the product.
As usage increased, two new engineering constraints appeared.
The Gemini Developer API had limited quota and billing.
At the same time, Learnzy had Google Cloud credits available through Vertex AI.
The challenge wasn’t simply changing AI providers.
The challenge was migrating the infrastructure without changing the Android application, breaking Shiv, or exposing infrastructure complexity to users.
The final architecture introduced an AI Gateway on Cloud Run that completely separated the product layer from the infrastructure layer.
Students continued talking to Shiv exactly the same way.
Only the infrastructure changed.
Context
During the first versions of Learnzy, speed of iteration mattered more than infrastructure.
The simplest architecture was:
flowchart TD
A[Student] --> B[Android]
B --> C[Supabase Edge Function]
C --> D[Gemini Developer API]
D --> E[Response]
The application worked well.
Shiv already had:
- Persona
- Context
- Guardrails
- Product Identity
The language model itself wasn’t the problem.
Infrastructure became the problem.
The Problem
Two independent systems existed.
System One
Shiv was running through the Gemini Developer API.
flowchart TD
A[Android] --> B[Gemini Developer API]
B --> C[Developer Billing]
System Two
Google Cloud Platform credits were available through Vertex AI.
flowchart TD
A[Google Cloud] --> B[Vertex AI]
B --> C[Startup Credits]
C --> D[Unused]
The product was paying through one billing system while another funding source remained unused.
I wanted to move the infrastructure without changing the product.
Design Constraints
Before changing the architecture I defined several constraints.
Product Constraints
- Shiv’s personality should remain unchanged.
- Android should not require any code changes.
- Existing APIs should remain stable.
Infrastructure Constraints
- Use Google Cloud credits.
- Remove dependence on Gemini Developer billing.
- Keep API credentials off the client.
- Avoid downloaded service account JSON keys.
Engineering Constraints
- Migration should be reversible.
- Minimal downtime.
- Easy future migrations.
- Provider independence.
Existing Architecture
flowchart TD
A[Student] --> B[Android App]
B --> C[Supabase gemini-proxy]
C --> D[Gemini Developer API]
D --> E[Gemini Response]
Advantages:
- Simple.
- Fast to build.
- Minimal infrastructure.
Limitations:
- Developer API quotas.
- Separate billing.
- Limited infrastructure control.
- Difficult future provider changes.
Existing Solutions Considered
| Option | Decision | Reason |
|---|---|---|
| Remain on Gemini Developer API | Rejected | Limited quota and unused Google Cloud credits |
| Move Android directly to Vertex AI | Rejected | Android becomes infrastructure-aware and credential handling becomes harder |
| Introduce an AI Gateway | Selected | Product stays stable while infrastructure becomes replaceable |
Whiteboard Thinking
Instead of asking:
How do I migrate to Vertex?
I asked:
Which component should actually know that Vertex exists?
The answer wasn’t Android.
The answer wasn’t Shiv.
The answer was one infrastructure layer.
Existing flow:
flowchart TD
A[Android] --> B[Supabase]
B --> C[Gemini]
Desired flow:
flowchart TD
A[Android] --> B[Supabase]
B --> C[Infrastructure Layer]
C --> D[Language Model]
The unknown layer became the AI Gateway.
The Insight
Infrastructure should never leak into the product.
The Android application shouldn’t care whether responses come from:
- Gemini Developer API
- Vertex AI
- OpenAI
- Anthropic
- a future local model
Its responsibility is simply:
Talk to Shiv.
Everything else belongs somewhere else.
Final Architecture
flowchart TD
A[Student] --> B[Android Application]
B --> C[Supabase Edge Function]
C --> D[Authentication Layer]
D --> E[Gateway Request Validation]
E --> F[Cloud Run AI Gateway]
F --> G[Infrastructure Abstraction]
G --> H[Vertex AI Gemini]
H --> I[Language Response]
I --> A
Layer Responsibilities
Android
Responsible for:
- Chat UI
- User interaction
- Authentication token
Never responsible for:
- AI provider
- Billing
- Credentials
Supabase
Responsible for:
- User validation
- Session verification
- Product routing
- Gateway authentication
Supabase continues exposing the exact same API to Android.
Nothing changes.
Cloud Run Gateway
The gateway became the new infrastructure layer.
Responsibilities include:
- Provider abstraction
- Vertex authentication
- Secret management
- Billing isolation
- Future provider switching
Cloud Run is now the only component aware of Vertex AI.
Vertex AI
Provides:
- Language generation
- Image understanding
- Model inference
Vertex no longer interacts directly with the application.
Authentication Architecture
The gateway uses a dedicated service account.
flowchart TD
A[Cloud Run] --> B[Attached Service Account]
B --> C[Vertex AI]
No downloaded JSON credentials.
No embedded API keys.
Authentication is handled through Google Cloud IAM.
Request Flow
sequenceDiagram
participant Student
participant Android
participant Supabase as Supabase Edge Function
participant Gateway as Cloud Run AI Gateway
participant Vertex as Vertex AI
Student->>Android: Sends message to Shiv
Android->>Supabase: Existing API request
Supabase->>Supabase: Verify user
Supabase->>Gateway: Authenticated gateway request
Gateway->>Vertex: Model inference
Vertex-->>Gateway: Model response
Gateway-->>Supabase: Normalized response
Supabase-->>Android: Shiv response
Android-->>Student: Display response
Rollback Strategy
One important design goal was reversibility.
If the gateway ever failed, the migration should be undoable within minutes.
Since Android never changed, rollback only requires changing infrastructure routing.
flowchart TD
A[Cloud Run Disabled] --> B[Supabase]
B --> C[Developer API]
C --> D[Application Continues]
This significantly reduced deployment risk.
Engineering Decisions
| Decision | Reason |
|---|---|
| Keep Android unchanged | Applications should not know infrastructure details |
| Keep Supabase as the public API | Stable interface for all clients |
| Introduce Cloud Run | Separate infrastructure from product logic |
| Use attached IAM Service Accounts | Avoid long-lived credentials |
| Authenticate gateway requests | Prevent unauthorized access |
Architecture Comparison
Before
flowchart TD
A[Android] --> B[Supabase]
B --> C[Gemini Developer API]
After
flowchart TD
A[Android] --> B[Supabase]
B --> C[Cloud Run]
C --> D[Vertex AI]
D --> E[Gemini]
Only one new layer was introduced.
Everything above it remained untouched.
Benefits
Product
- No application changes.
- Shiv behaves exactly the same.
Infrastructure
- Google Cloud credits utilized.
- Better provider abstraction.
- Easier monitoring.
Security
- No API keys in Android.
- No downloaded service account JSON.
- Centralized authentication.
Engineering
- Future provider migration becomes easier.
- Stable public interface.
- Reduced coupling.
Trade-offs
Advantages
- Infrastructure abstraction.
- Better security.
- Uses startup credits efficiently.
- Easier future migrations.
- Stable Android client.
Limitations
- Additional Cloud Run hop.
- Slight increase in latency.
- More infrastructure components to monitor.
Lessons Learned
Initially, I believed I was migrating from one AI provider to another.
Looking back, that wasn’t the real engineering problem.
The real problem was coupling product behavior to infrastructure.
Once I introduced a dedicated infrastructure layer, changing providers became an implementation detail rather than an application rewrite.
That realization fundamentally changed how I think about AI systems.
Products should depend on capabilities.
Infrastructure should depend on providers.
Those are not the same thing.
Future Direction
The AI Gateway has become the foundation for future AI capabilities.
Potential future work includes:
- Multi-model routing
- Automatic fallback providers
- Cost-aware model selection
- Request logging and analytics
- A/B testing between models
- Streaming responses
- Unified text, vision, and voice endpoints
The long-term goal is for Shiv to interact with a stable internal gateway while the underlying infrastructure evolves independently.
Project Outcome
The migration successfully moved Shiv from the Gemini Developer API to Vertex AI while preserving the existing product experience.
Students never noticed the migration.
The Android application required no architectural changes.
Google Cloud startup credits became usable.
More importantly, the project introduced an architectural principle that now guides much of Learnzy’s backend design:
Products should remain stable while infrastructure remains replaceable.
The migration wasn’t really about Vertex AI.
It was about designing a system where future infrastructure decisions no longer affect the product.