Hi TextAttack team — I maintain EvalPort, an open, framework-agnostic JSON spec (Apache 2.0) for portable LLM/NLP evaluation data: TestCase/Grader/Suite/ResultSet documents with a JSON Schema validator, so a scored dataset or run can move between tools without a one-off converter. Filing this as an issue per CONTRIBUTING.md's guidance on feature requests, before writing any code.
I read the real classes in textattack/attack_results/ rather than guessing:
AttackResult(original_result, perturbed_result) (attack_result.py) — original_result/perturbed_result are each a GoalFunctionResult (model output + succeeded flag), and num_queries is stored directly. The subclasses (SuccessfulAttackResult, FailedAttackResult, SkippedAttackResult, MaximizedAttackResult) each represent one outcome of running an attack against one input.- An
Attacker run over a dataset produces a list[AttackResult] — one entry per input example.
Mapping: each attacked input (original_result.attacked_text) is an EvalPort TestCase, the attack recipe (e.g. TextFoolerJin2019, PWWSRen2019) is the Grader, and each AttackResult — perturbed_result's success/failure plus num_queries — is a GraderResult. A caveat worth surfacing in the adapter itself: EvalPort's passed field is generic ("did this grader accept the output"), so a literal mapping needs a documented convention for which way robustness graders should read it (e.g. "passed" = "the attack failed to fool the model" i.e. the model was robust) rather than assuming "passed" always means "attack succeeded." I'd rather get that convention right in a real adapter than guess at it here.
Two ways this could land, and I don't have a strong preference:
- A standalone
textattack-openeval-adapter package in the EvalPort repo, depending on textattack as a normal dependency — zero footprint on this repo beyond maybe a docs link.
- A small optional export module inside this repo if you'd rather it live here.
Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
— Sahi, independent contributor (not affiliated with QData/TextAttack)
Hi TextAttack team — I maintain EvalPort, an open, framework-agnostic JSON spec (Apache 2.0) for portable LLM/NLP evaluation data:
TestCase/Grader/Suite/ResultSetdocuments with a JSON Schema validator, so a scored dataset or run can move between tools without a one-off converter. Filing this as an issue per CONTRIBUTING.md's guidance on feature requests, before writing any code.I read the real classes in
textattack/attack_results/rather than guessing:AttackResult(original_result, perturbed_result)(attack_result.py) —original_result/perturbed_resultare each aGoalFunctionResult(model output + succeeded flag), andnum_queriesis stored directly. The subclasses (SuccessfulAttackResult,FailedAttackResult,SkippedAttackResult,MaximizedAttackResult) each represent one outcome of running an attack against one input.Attackerrun over a dataset produces alist[AttackResult]— one entry per input example.Mapping: each attacked input (
original_result.attacked_text) is an EvalPortTestCase, the attack recipe (e.g.TextFoolerJin2019,PWWSRen2019) is theGrader, and eachAttackResult—perturbed_result's success/failure plusnum_queries— is aGraderResult. A caveat worth surfacing in the adapter itself: EvalPort'spassedfield is generic ("did this grader accept the output"), so a literal mapping needs a documented convention for which way robustness graders should read it (e.g. "passed" = "the attack failed to fool the model" i.e. the model was robust) rather than assuming "passed" always means "attack succeeded." I'd rather get that convention right in a real adapter than guess at it here.Two ways this could land, and I don't have a strong preference:
textattack-openeval-adapterpackage in the EvalPort repo, depending ontextattackas a normal dependency — zero footprint on this repo beyond maybe a docs link.Either way, real tests would validate against EvalPort's actual JSON Schema, not a mock.
Let me know which direction you'd prefer, or if this isn't a fit for your roadmap right now — no worries either way. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
— Sahi, independent contributor (not affiliated with QData/TextAttack)