Skip to content

Add compositional analysis code and example#57

Open
beyazit-y wants to merge 27 commits intomainfrom
compositional-analysis
Open

Add compositional analysis code and example#57
beyazit-y wants to merge 27 commits intomainfrom
compositional-analysis

Conversation

@beyazit-y
Copy link
Copy Markdown
Contributor

No description provided.

@beyazit-y beyazit-y requested a review from kevinchang73 April 8, 2026 18:01
Comment thread examples/compositional_analysis/README.md Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/run_exp.py Outdated
Comment thread examples/compositional_analysis/test.py Outdated
Comment thread examples/compositional_analysis/test.py Outdated
Comment thread examples/compositional_analysis/test.py Outdated
Comment thread examples/compositional_analysis/test.py Outdated
Comment thread examples/compositional_analysis/train.py Outdated
Comment thread examples/compositional_analysis/utils.py Outdated
Comment thread examples/compositional_analysis/utils.py Outdated
Comment thread examples/compositional_analysis/utils.py Outdated
Comment thread examples/compositional_analysis/utils.py
Comment thread src/verifai/compositional_analysis.py Outdated
Comment thread src/verifai/compositional_analysis.py Outdated
Comment thread examples/compositional_analysis/train.py Outdated
Comment thread examples/compositional_analysis/README.md Outdated
Comment thread src/verifai/compositional_analysis.py Outdated
Copy link
Copy Markdown
Collaborator

@dfremont dfremont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Beyazit, thanks for making this PR! With a few changes I was able to run everything except the final analysis (see comments), and the code looks OK overall. A few bigger things to improve:

  • There should be unit tests for the new functionality. They can be basic, not using a neural network or anything (we want the tests to run without optional dependencies such as stable baselines anyway), but should exercise the main code paths and check that the output is reasonable.
  • There should be a new page in the documentation describing the compositional analysis features. It can explain the example in more detail (for example, what do S, X, SXS, etc. actually mean) and also explain how to use the top-level APIs ScenarioBase and CompositionalAnalysisEngine. The docstrings for those are a good starting point (and you can auto-generate documentation from them, as we do in docs/feature_api.rst for example), but they need to be fleshed out: for example, the analysis methods accept a list of scenario names, but the example only ever calls them with a single string.

If you can help me solve the issue I had when running analyze.py I will try to run the example again. Thanks!

Comment thread examples/compositional_analysis/train.py Outdated
@beyazit-y
Copy link
Copy Markdown
Contributor Author

I fixed most of the comments. Specifically: (i) fixed the example (it should run smoothly now -- lmk if it doesn't work), (ii) removed redundant code, (iii) eliminated most of the repetitive code, (iv) added unit tests, and (v) improved README and docstrings. There is still more we can improve esp check and falsify methods, which can be further reimplemented to eliminate code repetition; however, as the falsify method is still experimental (which is mentioned in the README), I think it is good to keep those separate for now. I am working with an undergraduate to improve the code in this regard. Please lmk if you have other comments. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants