Important: modern LLMs are trained with an instruction hierarchy. OpenAI’s IH‑Challenge (Mar 10, 2026) and the Model Spec emphasize a clear priority ordering: System > Developer > User > Tool. Place safety‑critical and policy constraints in the system or developer layer so they remain highest priority and are resilient to lower‑priority inputs (including tool outputs and web content). See OpenAI IH‑Challenge for details: https://openai.com/index/instruction-hierarchy-challenge/ (Mar 10, 2026).

Model updates (note)

• OpenAI released the GPT-5.4 family (including lower-latency mini/nano variants) used in agent deployments and enterprise runtimes in early 2026. When choosing a model, test both the full and compact variants for accuracy/latency/cost tradeoffs and include per-variant regression tests. Source: OpenAI News (Apr 2026): https://openai.com/index/gradient-labs and related announcements.

• Agents SDK and model-native harnesses (Apr 15, 2026): OpenAI introduced a next-generation Agents SDK that includes native sandbox execution and a model-native harness. If you run agents, validate that your runtime supports sandboxing, short-lived credentials, and auditable execution traces. Do not assume behavior parity between an agent runtime and direct API calls—test both paths. Source: OpenAI News (Apr 15, 2026): https://openai.com/index/the-next-evolution-of-the-agents-sdk.

• Chain-of-thought monitoring and misalignment detection (Mar 19, 2026): OpenAI published internal monitoring findings showing chain-of-thought traces can surface misalignment in coding agents. Where you collect thinking traces, make them auditable, subject to retention policies, and optionally detachable from user-facing answers for privacy and compliance. Source: OpenAI News (Mar 19, 2026): https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment.

• Gemini inference tiers (Apr 2, 2026): Google introduced Flex (cost-optimized, higher-latency) and Priority (high-reliability) inference tiers for the Gemini API. Route background or batch-like work to Flex and interactive user-facing requests to Priority; implement graceful downgrade logic and telemetry that records which tier served each request. Source: Google AI Blog (Apr 2, 2026): https://blog.google/innovation-and-ai/technology/developers-tools/introducing-flex-and-priority-inference/.

Prompt strategies

• Zero-shot: simple tasks, low token cost
• Few-shot: consistent format or tricky edge-cases; include diverse examples and order them intentionally (the last example often has outsized influence)
• Chain-of-thought: multi-step reasoning or proofs — isolate thinking traces when possible and avoid leaking sensitive intermediate content
• Self-consistency: sample multiple reasoning traces and take the consensus for high-stakes tasks
• ReAct / tool-guided: when the model must propose actions (tool calls); implement an orchestrator to execute, verify, and sanitize tool outputs
• Structured output: use schema validation (e.g., zod, JSON Schema) rather than relying on free-form parsing

Temperature guide

• 0: deterministic-style behavior for classification/extraction (note: greedy decoding still permits small variance)
• 0.3–0.5: balanced
• 0.7–1.0: creative

Workflow

Step 1: Define the task contract

Before writing any prompt, answer these questions:

type PromptContract = {
  input: string;       // What does the model receive?
  output: string;      // What should it produce?
  format: string;      // JSON, markdown, plain text?
  constraints: string[]; // What must it avoid?
  edgeCases: string[];   // How should it handle ambiguity?
  examples: Array<{ input: string; output: string }>;
};

Step 2: Write the system prompt

Focus system-level text on what must always hold (safety, privacy, legal constraints). Put product-level preferences in developer instructions. Remember: models trained with instruction-hierarchy data are more likely to honor these distinctions.

(Example omitted here for brevity — keep a compact system prompt that lists role, task, output format, rules, and examples.)

Step 3: Implement structured output with OpenAI (Responses API)

• Prefer the Responses API when you need agentic primitives (tools) or an orchestrator; OpenAI’s engineering posts describe using the Responses API together with a shell tool and hosted containers to execute sandboxed actions and keep execution auditable.
• Treat tool outputs as lower-priority content in the instruction hierarchy: do not place tool-provided text into system messages without sanitization and verification. Validate and re-check tool outputs server-side before acting on them.
• Use schema helpers (like zodResponseFormat) to produce machine-parseable responses and validate them server-side to guard against format drift.

Example (Responses + zod):

import OpenAI from "openai";
import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";

const FindingSchema = z.object({
  severity: z.enum(["critical", "warning", "info"]),
  line: z.number(),
  message: z.string(),
  suggestion: z.string(),
});

const ReviewSchema = z.object({
  findings: z.array(FindingSchema),
});

const client = new OpenAI();

async function reviewCode(diff: string) {
  const response = await client.responses.create({
    // Use the current-generation family you validated for this task (example: gpt-5.4)
    model: "gpt-5.4",
    instructions: systemPrompt,
    input: diff,
    text: {
      format: zodResponseFormat(ReviewSchema, "code_review"),
    },
  });

  // Validate and parse on the server to guard against format drift
  return ReviewSchema.parse(JSON.parse(response.output_text));
}

Notes:

• Replace the model name with the specific variant you tested (gpt-5.4, gpt-5.4-mini, etc.). Compact variants reduce latency/cost but can change output fidelity.
• Always run server-side validation and a fallback retry/clarification when format validation fails.

Prompt versioning & rollout (production pattern)

Adopt explicit versioning and staged rollout for system/developer prompts:

• Keep prompts in version control alongside integration tests (prompts/ directory in the repo).
• Use semantic versioning for prompts (e.g., prompt v1.2.0) and include a short changelog entry describing behavioral intent and test coverage.
• Automate regression tests: unit-style prompts that assert final output, format compliance, and performance budgets (tokens, latency).
• Canary and rollback: roll a new prompt to a small % of traffic, monitor key metrics (accuracy, format compliance, error rates), then promote or rollback automatically.
• Runtime feature flags: allow switching between prompt versions at runtime and maintain per-version telemetry so you can compare behavior.

Rationale: treat prompts like code — the OpenAI Model Spec and iterative-deployment guidance encourage incremental rollout and monitoring of model/agent behavior.

Step 4: Implement with Anthropic

• Anthropic’s official prompting guidance recommends explicit roles, clear structure, and — when helpful — XML-like tags to reduce ambiguity in multi-component prompts. Wrap examples and sections in tags such as <instructions>, <examples>, <example>, <context>, and <input> to improve format fidelity for Claude models. Source: Anthropic Prompting Best Practices (platform.claude.com).
• Include 3–5 diverse examples wrapped in <examples> / <example> tags to improve format fidelity and edge-case handling for Claude models.

Step 5: Agents and orchestrators

• When building agents, the model should only propose tool calls — an orchestrator must execute them in a sandbox and return sanitized outputs for the next step. OpenAI’s agent runtime examples demonstrate combining Responses API, a shell tool, and hosted containers to run secure, auditable code and reduce blast radius. Source: OpenAI News (Mar 2026) and engineering posts.

• Newer agent runtimes include model-native harnesses and native sandbox execution (Apr 15, 2026). If you adopt these runtimes, verify sandbox boundaries (filesystem, network, environment variables), short-lived credential behavior, and audit logs. Design tests that exercise escape vectors and prompt-injection paths inside the agent runtime so you can detect behavioral differences between agent-hosted execution and direct API calls. Source: OpenAI News (Apr 15, 2026): https://openai.com/index/the-next-evolution-of-the-agents-sdk.

• Design a tight execution loop: propose → execute → return result → propose next step. This reduces hallucination and gives you a place to enforce policy, rate limits, retries, and identity checks.

• Treat tool outputs as untrusted: validate, sanitize, and re-check them against higher-priority system/developer instructions before using them to make decisions.

Prompt safety checklist (operational)

• Minimize authority: give the agent the least privilege needed (allowlist endpoints, limit API scopes, and avoid broad filesystem or network write permissions).
• Verification steps: require explicit verification for sensitive or irreversible actions (human approval, re-authentication, or second-factor confirmation).
• Sanitize external content: canonicalize and filter inputs from web pages, emails, or tools; avoid passing raw HTML or scraped text into the model without normalization.
• Constrain impact: default to read-only or simulated-run modes and require an explicit intent switch for irreversible operations.
• Audit & short-lived credentials: log proposals and tool outputs, and use time-limited credentials for downstream calls so compromised outputs cannot be reused.

Chain-of-thought and reasoning

• Chain-of-thought remains a powerful technique for multi-step reasoning, but it increases token costs and can reveal intermediate reasoning that might be sensitive.
• OpenAI’s internal monitoring work shows chain-of-thought traces can be useful for detecting misalignment and failure modes. Make thinking traces auditable and optionally separable from final answers so you can store, inspect, or purge them according to privacy policy. (See OpenAI monitoring posts, Mar 2026.)
• Prefer concise, auditable reasoning blocks and consider separating the "thinking" trace from the answer (e.g., <thinking> vs <answer> blocks) so you can store or purge traces depending on privacy needs.

Example template:

<thinking>
1. Restate the problem
2. Break into subproblems
3. Solve each subproblem with short steps
4. Verify constraints
</thinking>

<answer>
[Concise final answer here]
</answer>

Examples

Example 1: Classification with few-shot

(unchanged — keep compact few-shot pattern; enforce return-only rule and server-side validation)

Example 2: Data extraction with structured output

(unchanged — use structured JSON schema, return null for missing fields, parse relative dates against provided reference date)

Example 3: Multi-turn agent instructions

(unchanged — agent system prompt that lists available tools, behavior loop, citation rules, and constraints; ensure orchestrator validates tool output)

Decision tree

(unchanged — retain mapping from problem type to prompt pattern: classification, extraction, reasoning, generation, agent/tool use)

Edge cases and gotchas (UPDATED)

• Prompt injection: instruction-hierarchy training (IH‑Challenge) reduces some classes of prompt-injection attacks, but never assume immunity. Modern attacks increasingly resemble social engineering; architectural mitigations (least privilege, verification, sandboxing, short-lived tokens) remain vital. See OpenAI guidance: https://openai.com/index/designing-agents-to-resist-prompt-injection/ (Mar 11, 2026).
• Agent runtime differences: newer runtimes that offer model-native harnesses or sandboxed execution can change where and how prompts are evaluated. Test for behavioral differences between agent-hosted flows and direct API calls and include these cases in regression tests.
• Chain-of-thought leakage: thinking traces are useful for debugging and monitoring but can reveal sensitive intermediate steps. Separate and protect reasoning traces with explicit retention and access controls.
• Tool outputs are lower-priority: do not treat tool-proposed text as authoritative commands. Validate before using.
• Token budget & context: system prompts count against the context window — measure and compact context using retrieval, summarization, or the Responses API’s hosted workspace when you need file access.
• Model drift and portability: prompts that work for one provider can fail on another. Test across providers and include provider-specific tags or examples when needed.
• Few-shot ordering: the last example often has the most influence — order examples intentionally.
• Negative instructions vs positive rules: prefer "Always do Y" over "Don't do X" where possible.
• Output format compliance: models may drift — always validate parsed outputs server-side and fall back to a retry or clarification flow.
• Temperature 0 is not true determinism: small variance can remain. For strict determinism use programmatic checks and validation.
• Long prompts degrade: distill system prompts and use external context stores for large documents.
• Anthropic XML tips: official Claude guidance recommends consistent descriptive tags (<instructions>, <examples>, <input>) and nesting when content has a natural hierarchy to improve parsing and format fidelity.
• Gemini inference tiers (Flex vs Priority): use Flex for cost-sensitive, latency-tolerant workloads and Priority for interactive, user-facing workloads. Implement graceful-downgrade logic that retries or queues work to Flex when Priority is unavailable and monitor rate limits and cost tradeoffs. Source: Google (Apr 2, 2026): https://blog.google/innovation-and-ai/technology/developers-tools/introducing-flex-and-priority-inference/.

Evaluation criteria

• Accuracy: % of outputs matching ground-truth labels
• Format compliance: % of outputs parseable as the requested format
• Consistency: variance across multiple identical requests (temperature 0)
• Cost efficiency: tokens consumed per successful completion (include inference tier cost in estimates)
• Latency: time-to-first-token and total generation time
• Robustness: accuracy on adversarial / edge-case inputs
• Provider portability: cross-provider test coverage and provider-specific validation

Research-backed changes included in this update

• Reinforced instruction hierarchy priority (System > Developer > User > Tool) and citation to IH‑Challenge (OpenAI, Mar 10, 2026): https://openai.com/index/instruction-hierarchy-challenge/.
• Concrete agent prompt-injection mitigations and architectural controls (OpenAI, Mar 11, 2026): link to OpenAI guidance on designing agents to resist prompt injection.
• Recommendations for agent runtimes that combine the Responses API with sandboxed tools and hosted containers, and new model-native harnesses with native sandbox execution (OpenAI News, Apr 2026).
• Anthropic official prompting guidance added: XML-like tags and example-structuring recommendations (platform.claude.com).
• Gemini API inference-tier guidance (Flex vs Priority) and client-side downgrade strategies (Google, Apr 2, 2026): https://blog.google/innovation-and-ai/technology/developers-tools/introducing-flex-and-priority-inference/.

Sources & references

• OpenAI News: "The next evolution of the Agents SDK" (Apr 15, 2026) — https://openai.com/index/the-next-evolution-of-the-agents-sdk
• OpenAI News: "Designing AI agents to resist prompt injection" (Mar 11, 2026) — https://openai.com/index/designing-agents-to-resist-prompt-injection
• OpenAI News: "How we monitor internal coding agents for misalignment" (Mar 19, 2026) — https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment
• OpenAI Model Spec and iterative deployment guidance — https://model-spec.openai.com/
• Anthropic: Prompting best practices / Developer docs — https://platform.claude.com/docs/en/intro
• Google AI Blog: Gemini API Flex & Priority (Apr 2, 2026) — https://blog.google/innovation-and-ai/technology/developers-tools/introducing-flex-and-priority-inference/