Workshop: Validating Generative AI Models in the Age of E-23

October 28, 5:30–7:30 PM • Downtown Toronto

GenAI is racing into production, and your validation program has to keep pace. Regulators are already signaling higher expectations with the new E-23 guidance explicitly expanding its principles to include emerging AI and ML systems. That means firms must demonstrate effective governance, clear documentation, and robust validation practices—not just for traditional models, but also for GenAI pipelines, prompts, and retrieval-augmented systems.

Join us after work in downtown Toronto for a practical, hands-on session focused on how to validate GenAI systems in ways you can defend to stakeholders, auditors, and regulators while aligning with the expectations outlined in E-23.

Who should attend

• Model Risk & Validation, Compliance, Internal Audit
  
• Data Science & ML Engineering, ML Ops
  
• Product Owners and Risk Champions for GenAI initiatives
  

What you’ll learn (and practice)

Scope the system, not just the model: Map prompts, RAG pipelines, tools, and guardrails; define boundaries and dependencies consistent with E-23’s emphasis on end-to-end system risk.

Design defensible tests: Build acceptance criteria for hallucination, harmful content, bias, robustness, privacy leakage, and IP risk—aligned with regulator expectations for reliability, fairness, and explainability.

Evaluation methods that work: Pairwise judging, rubric-based LLM-as-judge, golden sets, and human review—and guidance on which approaches map most directly to E-23 compliance.

RAG & retrieval validation: Groundedness, citation quality, retrieval recall/precision, and corpus hygiene checks as part of monitoring content pipelines.

Prompt & config change control: Versioning, test harnesses, regression suites, and rollback criteria—reflecting E-23’s control requirements for lifecycle management.

Monitoring in production: Metrics for degradation, drift, jailbreaks, and safety incidents; practical alert→ticket→remediation loops for operational resilience.

Evidence & documentation: Model/system cards, validation reports, and audit-ready artifacts that align directly with supervisory expectations under E-23.

Format (interactive + practical)

Short lightning talks followed by guided mini-exercises with realistic case studies and editable templates. You’ll leave with assets you can adapt immediately.

Agenda (2 hours)

• 5:30 PM — Arrival & networking (light bites & drinks)
  
• 5:45 PM — Welcome & objectives
  
• 5:55 PM — Scoping GenAI Systems (components, risks, and control points)
  
• 6:10 PM — Mini-Exercise: Define acceptance criteria for a GenAI use case
  
• 6:30 PM — Evaluation Techniques (LLM-as-judge, golden sets, human review)
  
• 6:45 PM — Mini-Exercise: Build a small evaluation plan & test harness outline
  
• 7:05 PM — Monitoring & Change Control (from pre-prod tests to on-call playbooks)
  
• 7:20 PM — Debrief & immediate next steps
  
• 7:30 PM — Close
  

What you’ll take away

• A GenAI Validation Plan template (scope, risks, test strategy, acceptance criteria)
  
• Evaluation workbook: examples for groundedness, harmful content, bias, and robustness
  
• A lightweight test harness checklist for prompts, RAG, and guardrails
  
• A 60–90 day playbook to operationalize validation and monitoring
  

Logistics

• Date & Time: October 28, 5:30–7:30 PM
  
• Location: Downtown Toronto (venue details provided upon registration)
  
• Capacity: Limited to keep the session highly interactive
  
• Bring: Laptop recommended (we’ll share templates); optional: a GenAI use case from your org
  

Why attend

• Actionable, not academic: Concrete artifacts, controls, and workflows.
• Cross-functional by design: Risk, data, and product perspectives in one room.
• Audit-ready outcomes: Evidence you can stand behind, aligned with E-23 and SR 11-7 expectations.