Adding AI Features to Your Next.js App Without Overengineering It

AI features are everywhere now: chatbots, image analysis, content generation. But integrating AI into a production Next.js app doesn't have to mean building a complex pipeline. In this post, I'll walk through practical patterns I used in my Calorie Counter app, which uses GPT-4o for nutrition analysis and food image recognition.

The goal is simple: add AI that works reliably, without overengineering it.

What We're Building

I built the Calorie Counter as a solo project. Users can type a food item (or upload a photo) and get back structured nutrition data, a health score, and recipe video recommendations.

The AI handles two things:

1. Text input → Extract detailed nutrition information

2. Image input → Identify the food item from a photo, then extract nutrition

Everything else, health scoring, video search, data persistence, is handled by regular code. This is the first principle: use AI only where it adds value.

1. Structured Outputs with Zod

The biggest mistake I see developers make is treating LLM responses as free-form text and then writing fragile parsing logic. OpenAI supports structured outputs: you pass a schema, and the response is guaranteed to match it.

First, define your schema with Zod:

import { z } from "zod";

const FatDetailsSchema = z.object({
  saturated: z.number(),
  unsaturated: z.number(),
  trans: z.number(),
});

const CarbohydrateDetailsSchema = z.object({
  fiber: z.number(),
  sugar: z.number(),
  starch: z.number(),
});

export const NutritionScoresSchema = z.object({
  food_item: z.string(),
  calories: z.number(),
  protein: z.number(),
  fat: z.number(),
  fat_details: FatDetailsSchema,
  carbohydrates: z.number(),
  carbohydrate_details: CarbohydrateDetailsSchema,
  cholesterol: z.number(),
  sodium: z.number(),
  potassium: z.number(),
  vitamin_c: z.number(),
  calcium: z.number(),
  iron: z.number(),
});

Then use zodResponseFormat when calling the API:

import { zodResponseFormat } from "openai/helpers/zod";

const response = await client.chat.completions.parse({
  model: MODEL,
  messages: [
    { role: "system", content: systemPrompt },
    { role: "user", content: userPrompt },
  ],
  response_format: zodResponseFormat(NutritionScoresSchema, "NutritionScores"),
  temperature: 0.3,
});

return response.choices[0].message.parsed!;

The key things here:

• zodResponseFormat tells OpenAI to return JSON matching your Zod schema. No manual parsing needed.

• temperature: 0.3 keeps responses consistent. For factual data like nutrition info, you want low creativity.

• .parsed! gives you a fully typed object. Your TypeScript types and your AI output are the same schema.

This is much cleaner than parsing free-form text with regex or hoping the model returns valid JSON.

2. Handling Rate Limits with Retry Logic

If you're using OpenAI (or Azure OpenAI), you will hit rate limits eventually. A simple retry wrapper with exponential backoff handles this gracefully:

async function withRetry<T>(fn: () => Promise<T>, maxRetries = 3): Promise<T> {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err: unknown) {
      const status = (err as { status?: number }).status;
      if (status === 429 && attempt < maxRetries) {
        const delay = Math.pow(2, attempt) * 1000; // 1s, 2s, 4s
        await new Promise((r) => setTimeout(r, delay));
        continue;
      }
      throw err;
    }
  }
  throw new Error("Unreachable");
}

Then wrap your API calls:

const response = await withRetry(() =>
  client.chat.completions.parse({
    model: MODEL,
    messages: [...],
    response_format: zodResponseFormat(NutritionScoresSchema, "NutritionScores"),
    temperature: 0.3,
  })
);

This only retries on 429 (rate limit) errors. Other errors fail immediately: you don't want to retry on a bad prompt or authentication failure.

3. Vision API: A Two-Pass Approach

For image analysis, I used a two-call pattern:

Call 1: Identify the food — Send the image to GPT-4o's vision endpoint with strict guardrails.

const base64Image = imageBuffer.toString("base64");

const visionResponse = await withRetry(() =>
  client.chat.completions.create({
    model: MODEL,
    messages: [
      {
        role: "system",
        content: `You are a precise food image analyzer. Your rules:
1. Determine if the image contains food or not
2. Never classify non-food items as food
3. If the image contains people, landscapes, or non-food objects,
   respond with is_food: false`,
      },
      {
        role: "user",
        content: [
          { type: "text", text: "Analyze this image and return a JSON response..." },
          {
            type: "image_url",
            image_url: { url: `data:\({mimeType};base64,\){base64Image}` },
          },
        ],
      },
    ],
    response_format: { type: "json_object" },
  })
);

Call 2: Normalize the output — A second, cheaper call formats the vision result into a clean schema.

Why two calls instead of one? The vision model is good at identifying food, but its output format can be inconsistent. A dedicated formatting pass gives you reliable structure. This is cheaper and more predictable than asking the vision model to do both at once.

4. Keep AI Out of What Doesn't Need It

This is the part most developers skip. The Calorie Counter computes a health score for every food item — but this is done algorithmically, not by the LLM:

export function calculateHealthScore(nutrition: NutritionData): HealthScore {
  // Each nutrient has a signed weight: positive = good, negative = bad
  const weights = {
    protein: 0.15,
    fiber: 0.12,
    vitamin_c: 0.08,
    calcium: 0.08,
    // ...
    saturated_fat: -0.12,
    trans_fat: -0.15,
    sugar: -0.10,
    sodium: -0.08,
  };

  // Score each nutrient against its healthy range,
  // multiply by weight, sum to get 0-10 score
}

Why not ask GPT for the health score? Three reasons:

• Cost — Every LLM call costs money. A math function is free.

• Consistency — The same input always produces the same score. LLMs can vary.

• Debuggability — When a score seems wrong, you can trace the algorithm. With an LLM, you'd be guessing.

Rule of thumb: If the logic can be expressed as a deterministic function, don't use AI for it.

5. API Route Structure

Keep your API routes thin. They should validate input, call your AI functions, and return the response:

// app/api/calculate-nutrition/route.ts
export async function POST(request: Request) {
  const body = await request.json();
  const parsed = CalculateNutritionRequestSchema.safeParse(body);

  if (!parsed.success) {
    return NextResponse.json(
      { error: "Invalid request", details: parsed.error.flatten() },
      { status: 400 }
    );
  }

  const nutrition = await getNutritionInfo(parsed.data);
  const healthScore = calculateHealthScore(nutrition);
  const videos = await getRecipeVideos(nutrition.food_item);

  return NextResponse.json({ nutrition, healthScore, videos });
}

Zod validates the input. The AI function returns typed data. The health score is computed. Videos are fetched. No middleware chains, no abstraction layers — just a straightforward pipeline.

6. Client-Side: Don't Overthink It

On the frontend, a simple loading state is often enough:

const [loading, setLoading] = useState(false);
const [result, setResult] = useState<NutritionResult | null>(null);

const handleSubmit = async () => {
  setLoading(true);
  try {
    const response = await fetch("/api/calculate-nutrition", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ food_item, quantity, unit }),
    });
    const data = await response.json();
    setResult(data);
  } catch (err) {
    setError("Something went wrong. Please try again.");
  } finally {
    setLoading(false);
  }
};

I deliberately did not use streaming here. For a nutrition lookup, the user needs the complete table: partial results aren't useful. Streaming adds complexity (managing chunks, partial UI states, error handling mid-stream). It makes sense for chatbots, but not for structured data.

Add streaming when the user benefits from seeing partial output. Skip it when they need the complete result.

7. Cost Control Checklist

A few practical things that keep API costs in check:

• Validate before calling — Client-side validation prevents empty or garbage requests from hitting the API.

• Limit file sizes — The calorie counter enforces a 10MB image limit client-side. Smaller images = fewer tokens.

• Use low temperature — For factual queries, temperature: 0.3 or lower reduces token usage and gives more focused responses.

• Hybrid approach — Use AI for what requires intelligence (food identification, nutrition extraction). Use regular code for everything else (scoring, video search, CRUD).

Wrapping Up

Adding AI to a Next.js app comes down to a few principles:

1. Use structured outputs — Zod schemas with zodResponseFormat eliminate parsing headaches.

2. Handle failures gracefully — Exponential backoff for rate limits, Zod validation for inputs.

3. Keep AI scoped — Not everything needs an LLM. Deterministic logic should stay deterministic.

4. Skip streaming for structured data — Simple request/response is easier to build and debug.

5. Control costs at every layer — Validate early, limit payloads, use low temperature.

If you have questions or want to share how you're integrating AI into your apps, find me on X.