AI Evaluations

How Evaluations Work

When a contribution is ingested, it is queued for evaluation. The AI engine analyzes the contribution across multiple dimensions specific to its type. Evaluations are processed asynchronously — you will be notified when results are available.

The evaluation engine uses versioned models, meaning the criteria and weights can be updated over time. When a new model version is deployed, previous evaluations retain their original scores — they are never retroactively changed.

Code Evaluation Criteria

Code contributions (commits, pull requests) are evaluated across the following dimensions:

Documentation Evaluation Criteria

Non-code contributions like documentation, reports, and proposals are evaluated using rubrics specific to their type. Common dimensions include:

• Clarity & Structure — how well-organized and readable the content is.
• Completeness — whether the document covers all relevant aspects of the topic.
• Accuracy — factual correctness and alignment with platform standards.
• Actionability — whether the document leads to clear next steps or decisions.

Transparency & Explainability

Every evaluation result includes a breakdown of scores across each dimension. You can see exactly how your contribution was assessed and why it received the scores it did. This transparency is a core design principle — there are no black-box evaluations.

You can view your evaluation scores from Dashboard > Evaluations, where each evaluation shows the overall score and the per-dimension breakdown.

Peer Feedback

In addition to AI evaluation, contributions receive peer feedback from other community members. Peer feedback uses a structured rubric with 5 to 7 questions designed to capture human judgment that AI might miss — creativity, collaboration quality, and mentorship.

Reviewers are automatically assigned to ensure balanced feedback distribution. You can view feedback you have received and feedback you need to provide from Dashboard > Feedback.

Evaluation Review Process

If you believe an AI evaluation does not accurately reflect the quality of your contribution, a human review process is available. Evaluations can be flagged for review by administrators or working group leads, who compare the AI assessment against expert human judgment.

The evaluation review process helps calibrate and improve the AI models over time, ensuring that evaluations become more accurate with each iteration.