Organize human feedback on LLM answers using JSON Schemas, example records, Python and JavaScript SDK source, review templates, and quality playbooks. Maintained by Acadify Solution.
This repository provides reference materials and client source. It does not include a hosted evaluation service or a complete server implementation. API examples require a compatible backend that you operate or separately arrange.
-
Read the four-pillar methodology: Good, Bad, Learned, and Remember.
-
Inspect the feedback schema and example payloads.
-
Validate the included examples locally:
git clone https://github.com/AcadifySolution/asr-feedback.git cd asr-feedback python -m pip install jsonschema python scripts/validate_schemas.py -
Adapt the feedback session template to your use case and run an independent review using the quality playbook.
Keep real customer data, credentials, and sensitive prompts out of public issues and sample records. See the security policy.
| Need | Starting point |
|---|---|
| Understand the proposed system | Architecture |
| Structure feedback and session reports | Schemas and schema guide |
| Integrate client source with your backend | SDK source and integration guide |
| Define review criteria | Quality framework and taxonomy |
| Run a review process | Playbooks and templates |
| Explore proposed reporting formats | Sample reports and analytics guidance |
| Review data handling proposals | Governance documentation |
Documentation describes proposed practices and illustrative examples. It is not an independently audited record of client deployments, certifications, service availability, or operational results. Review and validate each practice for your own implementation before adoption.
| Question | Evidence to collect |
|---|---|
| Do reviewers agree? | Independently rated samples, rubric version, and an agreement calculation |
| Are feedback records valid? | Schema version and validation results for the actual dataset |
| Does a model change help? | A held-out dataset, baseline and candidate versions, and comparable evaluation runs |
| How fast is review? | Recorded submission and completion timestamps |
| Which errors recur? | Versioned severity labels and issue counts over comparable periods |
No measured client outcomes, adoption totals, or benchmark improvements are claimed by this README. Examples in linked documents are reference material; validate any quantitative claims against primary evidence before reusing them.
Use AI Evals Framework for automated dataset-based evaluation alongside human feedback. The organisation project directory lists related RAG, fine-tuning, and agent tools.
If this toolkit helps, star it to save it or fork it to adapt the schemas. Contributions should include a reproducible example and validation results; follow CONTRIBUTING.md and the code of conduct.
MIT.