Evaluating & Improving Generative & Agentic AI

The AI Literacy Specialization Program is one-of-a-kind hierarchical & cognitive skills based curriculum that teaches artificial intelligence (AI) based on a scientific framework broken down into four levels of cognitive skills.Part 3: Analyze and Evaluate combines the below two cognitive skills -Analyzing (examining model's outputs to identify biases that may have been learned from the training data)Evaluating (assessing the performance, ethics, and overall effectiveness of an AI system)Whether you’re a policymaker, red teamer, LLM safety auditor, risk manager, AI developer, or a certification aspirant (AIGP, RAI, AAIA), this program gives you what theory alone cannot: applied, practice-ready evaluation and mitigation skills.The course is structured into four progressive modules: what to evaluate, how to evaluate, how to mitigate, and real-world case studies:IntroductionWhy Evaluate AI?Who Should Evaluate & When?What to Evaluate?1.1) Standard Safety Evaluation1.2) Frontier Capabilities & Misuse1.3) Misalignment1.4) Structural & Multi-Agent RisksHow to Evaluate?2.1) Overview2.2) Benchmarks & DatasetsBenchmark Repository v0.12.3) Evaluation Metrics2.4) Evaluation Techniques2.5) Red Teaming 101Evaluation Toolkit Repository v0.1How to Mitigate?3.1) How to MitigateMitigation Techniques Repository v0.1Case Studies4.1) Case Study 1: Automated Evaluation of LLMs Using Standard Eval Framework4.2) Case Study 2: Measuring Model Performance and Enhancing Explainability of a Credit Risk Model4.3) Case Study 3: Measuring Group Fairness in Housing Price Predi