Skip to content

CodeTailor: accept ===== before test code, and support subgoal labels - #1599

Merged
bnmnetp merged 3 commits into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-subgoal-labels
Oct 10, 2026
Merged

bnmnetp merged 3 commits into
RunestoneInteractive:mainfrom
aspadiyath:codetailor-subgoal-labels

Conversation

@aspadiyath

Copy link
Copy Markdown
Contributor

Two commits.

1. Accept four or more = before a question's test code.
For book-authored questions, CodeTailor found the unit tests by splitting the activecode source on exactly ====. activecode.js accepts ===== and longer, which older books use, and for those questions the split left a stray = at the top of the tests. Every generated solution then failed with a SyntaxError, and CodeTailor silently fell back to the static example instead of personalizing help. CodeTailor now splits on a line of four or more =, matching activecode.js. This commit stands alone and can be merged separately.

2. Subgoal labels.
Instructors can write subgoal labels (full-line comments naming each step) into a question's backup Parsons problem. Students can write their own in the editor.

  • Help before any code: a student who asks for help before writing any code gets a Parsons problem of just the backup's labels, under the function's def line from the starter code. The def is #settled in the settled-block personalization. Untouched starter code counts as no code. This is logged as solution type subgoal_labels.
  • Student labels: when a student's code contains labels, the LLM is asked to follow that plan and keep them. They're restored into the personalized solution and puzzle, each above the code it names. Labels are also copied onto paired distractors, so they never reveal which block is correct.
  • What isn't a label: commented-out code (# total = 0) isn't treated as a label, comments that ship with the starter code aren't the student's labels, and full-line comments never become draggable blocks.
  • Unchanged for existing questions: questions whose backup problem has no comments behave exactly as before.

Testing:
Tested by hand on a local server with py4e-int's list_writeAvgq:

  • the labels-only puzzle (graded correct),
  • personalized puzzles that keep a student's labels,
  • AI_personalized results once the separator issue was out of the way.

🤖 Generated with Claude Code

aspadiyath and others added 2 commits October 8, 2026 13:45
CodeTailor found a book-authored question's unit tests by splitting its
activecode source on exactly "====". activecode.js accepts "=====" and
longer, which older books use, and for those questions the split left a
stray "=" at the top of the tests. Every generated solution then failed
with a SyntaxError and CodeTailor silently fell back to the static example
instead of personalizing help.

Split on a line of four or more "=", as activecode.js does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Instructors can now write subgoal labels -- full-line comments naming each
step -- into a question's backup Parsons problem, and students can write
their own in the editor.

- A student who asks for help before writing any code gets a puzzle of just
  the backup problem's labels, under the function's def line from the
  starter code (locked in place for the settled-block personalization).
  Untouched starter code counts as no code, so the print call a question
  ships with no longer sends the student to the LLM. Logged as solution
  type "subgoal_labels".
- When a student's code contains labels, the LLM is asked to follow that
  plan and keep them, and they are restored into the personalized solution
  and puzzle, each above the code it names. Labels are copied onto paired
  distractors too, so they never reveal which block is correct.
- Comments never become blocks. Commented-out code ("# total = 0") is not
  a label, and comments that ship with the starter code ("# write code
  here") are not the student's labels.
- The personalized solution no longer has blank lines where comments were
  stripped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Delimiter compatibility, visible-suffix handling, ambiguous question lookup, and commented-code detection have unresolved correctness issues.

4 open findings
What changed in this PR

Adds broader activecode suffix parsing and subgoal-label support for CodeTailor.

Changes:

  • Parses longer test delimiters and starter code.
  • Generates labels-only puzzles for untouched code.
  • Preserves student labels through personalization.
File Description
subgoal_labels.py Implements label detection, anchoring, and puzzle creation.
get_personalized_solution.py Instructs the LLM to preserve labels.
end_to_end.py Restores labels after testing and block generation.
coach.py Extracts starter/tests and serves labels-only puzzles.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread bases/rsptx/book_server_api/routers/coach.py Outdated
Comment thread bases/rsptx/book_server_api/routers/coach.py Outdated
Comment thread bases/rsptx/book_server_api/routers/coach.py Outdated
Review follow-ups:
- Find the test suffix the way activecode.js does: the first run of four
  or more "=", wherever it is, not only on a line of its own.
- Keep a "===!" visible suffix in the starter code. activecode drops only
  the marker, so cutting the starter there made an untouched exercise look
  edited and skip the labels-only puzzle.
- Look up the starter code in the course's base course first, then fall
  back to a global name match, as the main CodeTailor path does. Question
  names are unique only within a base course.
- Treat any commented-out Python statement other than a bare expression
  as code, e.g. "# count: int".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@aspadiyath

Copy link
Copy Markdown
Contributor Author

Fixed Copilot's feedback in 260cb8e

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Java commented code is misclassified as labels, and scoped question misses prevent the intended starter-code fallback.

1 open finding
4 resolved since last review
Previously missed (1)

In code that hasn't changed since last review

Medium severity Java commented code is misclassified as labels

bases/​rsptx/​book_server_api/​routers/​personalized_parsons/​subgoal_labels.py:59

The Java branch treats common commented-out code without a trailing semicolon/brace as a label. For example, // if (x > 0), // int total = 0, and // return total all make is_label_line(..., "java") return true, so backup code can appear as a subgoal and student code can be preserved as a supposed plan. Expand the Java code heuristic to recognize control/declaration/assignment/return syntax, not only line terminators.

🧠 Review effort: Balanced

Comment on lines +310 to +312
question = await fetch_question(problem_id, basecourse=basecourse)
if not question:
question = await fetch_question(problem_id)
@bnmnetp
bnmnetp merged commit 0165133 into RunestoneInteractive:main Oct 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants