Prompt EngineeringPrivacyLLMs

Prompting Strategies for Multi‑App Contextual Assistants (Lessons from Gemini’s App Context Pull)

UUnknown

2026-03-08

10 min read

Practical patterns to feed email, photos, and docs into LLMs with RAG, compression, and privacy-first orchestration to cut token costs and leaks.

Hook: Your assistant can access everything — but token limits and privacy make it useless if done badly

Developers building contextual assistants in 2026 face a common trap: users expect an assistant that draws from email, docs, photos and other siloed apps, yet naïvely dumping that data into an LLM explodes token costs and creates privacy risk. If your assistant is slow, expensive, or leaks PII, adoption dies fast. This article gives practical, production-ready techniques and prompt patterns — inspired by industry moves like Apple’s partnership with Google’s Gemini (late 2025) and agentic assistants from Alibaba’s Qwen line — to feed multi‑app context into LLMs while preserving token efficiency and privacy.

Executive summary — what you'll learn

How to design a context orchestration pipeline that prioritizes relevance and privacy before tokens hit the model.
Concrete prompt patterns for compressed, structured context blocks that save tokens and improve accuracy.
Multimodal techniques for photos and OCR’ed docs: convert before including.
Privacy-first best practices: local summarizers, redact-first workflows, consent and audit logging.
Code recipes (Python/Node) showing RAG + compressor + orchestration in action.

Why 2025–26 changes matter for multi‑app assistants

Late 2025 and early 2026 saw two notable trends that inform practical design choices today:

Gemini-style assistants that can pull context across apps (email, photos, YouTube history) highlight that multi‑app integration is feasible — but also that raw access is dangerous if unmanaged.
Agentic assistants (see Alibaba's Qwen updates) show the shift from “answering” to “acting” — orchestrators need to control what context is used before actions are taken.

Those advances make orchestration and token efficiency non-negotiable: models can access more data, but your system must decide what to expose and how.

Core challenges we solve

Token bloat: Long email threads and full documents quickly exceed context windows or become costly.
Relevance: Not all app data matters; irrelevant context increases hallucinations.
Multimodal conversion: Photos and PDFs need structured, compact representations.
Privacy: PII/PHI in context must be controlled — consented, redacted, or summarized locally.
Latency and cost: Frequent model calls with bulky prompts are slow and expensive.

High-level solution: Context orchestration pipeline

Implement an orchestration pipeline that enforces policies and token budgets before content reaches the LLM. The pipeline has five stages:

Signal collection — metadata-only fetch (timestamps, sender, last-modified, size, tags).
Prioritization — score items by relevance heuristics (query similarity, recency, user intent, permission).
Privacy filter — client-side redaction/consent check and sensitivity scoring.
Compression / representation — run extractive or abstractive summarizer or convert images to captions/scene graphs.
Prompt assembly — pack prioritized compressed items into a structured, token-aware context block with a budget.

Why this order matters

Score and filter first to avoid unnecessary work (and token waste). Compression is expensive but much cheaper than sending full documents repeatedly. Privacy must be enforced before any network calls that could expose sensitive content.

Technique 1 — Retrieval + RAG with metadata-aware filtering

Retrieval-Augmented Generation (RAG) remains the backbone for multi‑app assistants. Key differences for multi-app contexts:

Store lightweight metadata in the vector DB alongside embeddings: app source, privacy level, owner, last modified, and a short extractive teaser (1–2 lines).
At query time, fetch top-N by vector similarity, then apply a metadata filter (e.g., only items with consent=true or privacy_score < threshold).
Rerank by recency and contextual relevance; discard duplicates across apps (dedupe by hash/signature).

Practical pattern: fetch top 50 embeddings, metadata-filter to 10, then compress each to a 1–3 sentence extract and send the best 3–5 to the LLM.

Technique 2 — Prompt compression patterns

Prompt compression is the most directly cost-saving tactic. Use two complementary approaches:

1) Extractive compression (cheap, deterministic)

Run a lightweight extractor to pull subject lines, action items, people involved, dates, and 1–3 sentence extractive summary of the latest segment.
Store these extracts in the vector DB and prefer them over raw docs during prompt assembly.

2) Abstractive compression (smarter, slightly costlier)

Use a small local or hosted summarization model (e.g., an edge T5/Flan micro-model) to produce concise, intent-aware summaries.
Design summaries to fit a schema so the downstream LLM can reason reliably (e.g., JSON keys: summary, actions, participants, sensitivity).

Compression pattern: JSON schema + size budget

{
  "source": "gmail",
  "teaser": "Product spec discussion about payment API",
  "summary": "Decides to switch to v2 API, pending security review; next step: schedule demo",
  "actions": ["schedule demo", "security review"],
  "sensitivity": "confidential",
  "tokenEstimate": 42
}

Send a fixed number of JSON objects as the context block. The LLM can parse it reliably and token usage is predictable.

Technique 3 — Multimodal: photos and docs

Photos and PDFs are high-value but high-cost if included raw. Convert first:

Photos — run a local captioner + scene graph + face/landmark detection. Keep a short caption (1–2 lines) and a list of detected objects/people with privacy flags.
Scanned docs — OCR then run extractive summarizer on the textual layers. Keep the extracted key facts and redacted snippets only.
Video/YouTube history — prefer transcripts and timestamped highlights instead of raw history.

Prompt pattern for a photo context block:

{
  "photo_id": "img_2025_12_19_01",
  "caption": "Office whiteboard with API diagram: auth, gateway, v2 payment flow",
  "objects": ["whiteboard", "diagram", "people:2"],
  "privacy": "internal",
  "tokenEstimate": 28
}

Privacy is not an afterthought. Implement layered controls:

Client-side prefiltering — run on-device detectors for PII (SSNs, emails, account numbers). Mark or redact before sending anything to server.
Consent-backed metadata — store per-item consents and use them to gate inclusion. Don’t assume global consent covers all items.
Local summarizer pattern — run a small summarizer on-device: only the compressed summary leaves the device.
Encrypted vectors — encrypt embeddings and use private compute / TEEs for vector search when sensitivity is high.
Policy sidecar — an access-control service that enforces privacy policies and logs all decisions for audits.

Example: For health-related messages, the client returns a policy token “health_consent=false” and a 1-line safe summary like “Patient follow-up message (sensitive) — requires explicit consent.”

Technique 5 — Token efficiency operational tricks

Keep the system prompt static on the server and only send minimal context to refer to it — store a “system profile ID” that the model handler resolves.
Use short labels and numeric codes in context blocks to reduce repetition (e.g., participant IDs instead of full names, with a small legend if needed).
Avoid multi-example few-shot prompts; prefer model fine-tuning or instruction tuning, or store examples server-side and reference an example ID.
Use streaming where possible: get an initial answer with compact context, and then progressively reveal more context if the user asks follow-ups.

Pattern: Progressive disclosure + clarifying questions

Instead of including everything up front, ask a short clarifying question with 1–2 context items. If the user confirms, fetch and include more. This reduces unnecessary context passing and improves user control.

Context orchestration algorithm (pseudocode)

function assembleContext(query, user) {
  // 1. quick metadata fetch
  candidates = metadataSearch(query, limit=50)
  // 2. score by similarity, recency, consent
  candidates = filterByConsent(candidates, user)
  candidates = scoreAndRank(candidates, query)
  // 3. dedupe and cap
  candidates = dedupe(candidates)
  candidates = candidates.slice(0, 10)
  // 4. compress locally or with a small model
  compressed = candidates.map(c => compress(c))
  // 5. enforce token budget and assemble JSON blocks
  contextBlocks = packByTokenBudget(compressed, budget=1500)
  return contextBlocks
}

Code recipe: Python RAG + compressor (simplified)

from vectordb import VectorDB
from local_compressor import compress_text
from llm_client import LLM

vdb = VectorDB()
llm = LLM()

def answer(query, user_id, token_budget=1500):
    # metadata-only search
    candidates = vdb.search_metadata(query, top_k=50)
    candidates = [c for c in candidates if check_consent(c, user_id)]
    ranked = rerank_by_context(candidates, query)[:10]
    compressed = [compress_text(r['text']) for r in ranked]
    # pack until token budget
    context, used = pack_until_budget(compressed, token_budget)
    prompt = build_prompt(query, context)
    return llm.generate(prompt)

This recipe explicitly separates retrieval, consent filtering, compression, and prompt assembly.

Prompt templates — real, copy-paste ready

1) Concise context block (JSON)

System: You are an assistant that reads JSON context blocks.

User: {"query":"Summarize next steps for migrating payments","context":[{...},{...}]}

Assistant: Use only the provided context blocks to answer. If you need more details, ask one clarifying question.

2) Redaction-aware prompt

System: Treat fields with "sensitivity":"high" as unusable without explicit consent.

User: Here are candidate items. Provide a short plan using only non-sensitive items.

Context: [ {"summary":"...","sensitivity":"low"}, {"summary":"...","sensitivity":"high"} ]

Case study: Enterprise assistant for Product + Support teams

Scenario: An assistant integrates Gmail (tickets), Drive (specs), and Photos (screenshots). Basic stats from a production pilot:

Average raw context per request: 45,000 tokens (full docs + threads + images OCR).
After metadata prioritization + extractive compression: 3,200 tokens.
After abstractive compression + JSON packing: 560 tokens (84% token reduction vs extractive; ~99% vs raw).
Accuracy (human-rated) rose by 12% because irrelevant content was excluded; cost per request fell 7–10x.

Design notes: the team used a small on-device summarizer to compress email text and image captioning at the client; all PII-containing items required explicit consent for expansion.

Operational checklist: quick wins you can deploy this week

Implement metadata-only search before any heavy retrieval.
Ship a client-side PII detector and redact sensitive fields by default.
Store short extractive teasers with embeddings — use teasers in prompts by default.
Build a token-budget guard that refuses to assemble prompts above a set threshold.
Replace few-shot examples with instruction-tuned behavior (server-side system message).

Future trends and recommendations for 2026

Larger context windows will continue to grow, but orchestration still wins: passing curated context costs less and improves reliability.
On-device summarization will become mainstream for privacy-sensitive apps — invest in small, fast compressors.
Unified multimodal embeddings will reduce friction between photos/docs, enabling more semantic retrieval without sending raw assets.
Standardized privacy manifests (consent, sensitivity labels) will emerge; design your schema to be pluggable.
Agentic orchestration (like Qwen’s agentic features) will require strict policy sidecars: whenever the assistant can act, ensure context governance is enforced programmatically.

Common tradeoffs — when to prefer each technique

Need highest fidelity and low hallucination? Use abstractive compression + more context, accept higher cost.
Privacy-sensitive data? Use client-side compression + redaction and only send metadata unless consented.
Latency-sensitive flows? Prefer extractive summarization and smaller context blocks to keep roundtrip times low.

Final pattern library (cheat sheet)

Metadata-first RAG — metadata search → filter → compress → prompt.
JSON context blocks — structured, small, parseable by LLMs.
Client-side summarization — privacy-preserving default for sensitive apps.
Progressive disclosure — clarify before expanding context.
Policy sidecar — enforce privacy, consent, and actionability consistently.

Conclusion — build for orchestration, not for brute force

Gemini-style multi‑app access and agentic assistants show us what’s possible in 2026, but they also expose the risks of naive integration. The winning assistants won’t be those that shove more tokens at a model; they’ll be the ones that orchestrate context intelligently: prioritize, compress, and enforce privacy before the model ever sees sensitive text or images. Apply the patterns above to reduce cost, improve accuracy, and keep your users’ data safe.

Call to action

Ready to implement a context orchestration pipeline for your assistant? Start with the operational checklist above and run a two-week pilot that measures token consumption, latency, and privacy incidents. If you want a starter kit (sample orchestrator code, compression models, and JSON schemas), sign up for the technique.top developer bundle and get templates you can drop into production.

Unknown

Contributor

Senior editor and content strategist. Writing about technology, design, and the future of digital media. Follow along for deep dives into the industry's moving parts.