Skip to content

Save consumer output and exit codes to log files - #182

Open
pecomyint wants to merge 2 commits into
mainfrom
add/consumer-output-logs
Open

pecomyint wants to merge 2 commits into
mainfrom
add/consumer-output-logs

Conversation

@pecomyint

Copy link
Copy Markdown
Collaborator

Summary

  • Consumer stdout (tracebacks, exits) only reached Workflow's panels and was lost on restart — we could not tell why an analysis consumer stopped.
  • Each consumer now also writes consumer_<name>.log (rotating, next to general.log) with the start command and an exited with code N line.

Test plan

  • pytest tests/unit passes; ruff check src/dashpva/ clean
  • New test test_workflow_consumer_logs.py
  • Start/stop a consumer from Workflow; check logs/consumer_*.log

🤖 Generated with Claude Code

@pecomyint
pecomyint requested a review from Osayi-ANL September 30, 2026 22:15
…r take a log_file next to general.log, replacing the workflow's own handler

remove: CONSUMER_LOG_MAX_BYTES and CONSUMER_LOG_BACKUP_COUNT — consumer logs now share LogManager's rotation

@Osayi-ANL Osayi-ANL left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed b37978c. The consumer logs now use the existing LogMixin and LogManager instead of a separate handler in workflow.py. set_log_manager() and get_logger() take an optional log_file, which writes next to general.log with the same rotation and format. The CONSUMER_LOG_* settings are removed, so consumer logs use the shared 10 MB x 10 rotation.

Requesting changes for two things.

  1. The associator log fills up fast, especially with more than one consumer. Most of it is the pvapy stats block printed every report period. With 2 consumers it wrote 5.3 MB in about 70 minutes, so all the rotation files would fill in 12 to 24 hours. One option is a boolean COMPACT in settings.py. When True, each stats block is written as one short line, and tracebacks, errors and the start and exit lines are still written in full. When False, everything is written as it is now.

  2. A normal Stop loses the last output and the exit code. stop() sets _running = False, so run() stops reading right after the signal is sent. The fix is to keep reading in run() until the output ends, then wait() and log the exit code, and have stop() only send the signal.

Smaller notes: the branch is 6 commits behind main, blank lines get logged as empty timestamp lines, and two Workflows on the same machine write to the same consumer_associator.log, which rotation does not handle well.

@Osayi-ANL

Osayi-ANL commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

@pecomyint could you check out b37978c and see how you like it? It moves the consumer logs onto the existing LogMixin and LogManager instead of a separate handler in workflow.py. Also, for the associator log filling up quickly, would something like the COMPACT setting from the review work, or would you prefer another approach? I don't really mind any, but the associator file can fill up fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants