Add evaluation questions for education/course content#89
Open
dav-mac wants to merge 1 commit into
Open
Conversation
# Added evaluation questions for the following content: - Machine Learning at the Edge on Arm: A Practical Introduction: https://www.edx.org/learn/machine-learning/arm-education-machine-learning-at-the-edge-on-arm-a-practical-introduction - Optimizing Generative AI on Arm Processors: from Edge to Cloud: https://www.edx.org/learn/computer-science/arm-education-ai-on-arm - Optimizing Generative AI on Arm Processors: https://github.com/arm-education/AI-on-Arm - Advanced AI: Mixture of Experts: https://github.com/arm-education/Advanced-AI-Mixture-of-Experts - Advanced AI: Hardware Software Co-Design: https://github.com/arm-education/Advanced-AI-Hardware-- Software-Co-Design - ExecuTorch on Arm labs: https://github.com/arm-education/executorch_on_arm_labs - Introduction to Cloud Compute on Arm: https://github.com/arm-education/Introduction-to-Cloud-Compute-on-Arm - Arm Helium Technology M-Profile Vector Extension (MVE): https://github.com/arm-education/Arm-Helium-Technology # Question types Added 4 questions for each course: Question 1 = broad discovery Question 2 = specific detail Question 3 = task-oriented Question 4 = learning discovery ("Where can I learn about...", "Where can I find guidance on...", etc.)
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a large set of new evaluation questions for the embedding-generation workflow, mapping each question to one or more expected source URLs.
Changes:
- Expanded
eval_questions.jsonwith many new Q/A retrieval targets across multiple Arm Education resources. - Added expected URL mappings for EdX-hosted videos and multiple GitHub-hosted labs/READMEs/PDFs.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+3
to
+6
| "question": "How are multi-layer artificial neural networks structured, and what roles do layers, weights, biases, and activation functions play?", | ||
| "expected_urls": [ | ||
| "https://courses.edx.org/videos/block-v1:ArmEducationX+EDARMXML.6x+2T2022+type@video+block@dc338f5160934b888cbfb8be2446e2d4" | ||
| ] |
Comment on lines
+51
to
+54
| "question": "How can I get started with hands-on materials for deploying and improving generative AI inference on Arm hardware across Raspberry Pi and Graviton?", | ||
| "expected_urls": [ | ||
| "https://github.com/arm-education/AI-on-Arm/blob/main/README.md" | ||
| ] |
Comment on lines
+146
to
+150
| { | ||
| "question": "I need to decide whether Arm Neoverse cloud instances are a good target for my server workload and understand which architecture, tooling, and platform choices matter before I start porting. What should I review?", | ||
| "expected_urls": [ | ||
| "https://github.com/arm-education/Introduction-to-Cloud-Compute-on-Arm/blob/main/README.md" | ||
| ] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussed with @JoeStech -
Related content
Added evaluation questions for the following content:
Question types
Added 4 questions for each of the above courses:
Question 1 = broad discovery
Question 2 = specific detail
Question 3 = task-oriented
Question 4 = learning discovery ("Where can I learn about...", "Where can I find guidance on...", etc.)