111 create yaml template for problem submission

[kvdblom]

Draft yaml template for review

[Dvermetten]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

[CIGbalance]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

[CIGbalance]

I still think it would be best to enter problems separately, but I understand the necessity of streamlining this process. So if we want to allow entering suites, maybe we have different templates? Because some values might only make sense for a suite? Otherwise, for a suite, a lot of the input would be "varies" for number of objectives, variables, constraints, dynamic, noise, etc?

[kvdblom]

Yes, it would be nice to have problems separately, I have also prototyped how it might work to have both the suite and component problems, see here:

OPL/problems.yaml

Line 1107 in 84b34b6

problems:

I am not sure about separate templates for suites or problems, but we can think about this and discuss what makes sense. I would imagine, e.g., the BBOB sphere still has a bunch of "varies", because it is scalable in the number of variables for example. For a suite, I would want an exhaustive list (e.g., noise: yes, no / optional, or something like that) of the options, rather than a vague "varies". I will describe this in a comment in the template.

[kvdblom]

TODO: create a new issue for this.

[bn12]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

• name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  • name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

[kvdblom]

Maybe it would be useful to have some sort of comments to describe what the fields are / what would be valid input, specifically for the cases where there is a difference between quoted and non-quoted input. Example:

  variables: # information about the input variables
    types: continuous # can be one of (continuous, integer, binary, mixed)
    conditional: 'no' # whether there are conditional dependencies between variables, 'yes' or 'no'
    dimensionality: scalable # number of input variables, either as a number (in quotes) or scalable

Yes we should do that.

[kvdblom]

Thanks for this! The comments are mostly easy fixes or not that serious.

But I am mostly wondering of how this is supposed to be used? It is called a template, but it has BBOB-specific entries? I think this increases the risk of biasing the results.

I think I would suggest doing a template with a lot more comments (like @Dvermetten suggested) and then we can use this as real data as well as link to it as an example?

We should split between

• A template
• An example

[Dvermetten]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem):
other info:
name: null
partial evaluations: Not Present
full name: Electric Motor Design Optimization
constraint properties: Hard Constraints, Soft Constraints, Box Constraints
number of constraints: '12'
type of dynamicism: ''
form of noise model: ''
type of noise space: ''
other noise properties: ''
description of multimodality: Constraints are multimodal
key challenges / characteristics: Time-consuming solution evaluation, highly-constrained
problem
scientific motivation: Challenging to find good solutions in a limited time
limitations: 'Unavailability, even if available, it wouldn''t be helpful to use
for benchmarking due taking a long time to evaluate a single solution '
implementation languages: Python
links to implementations: Implementation not freely available
approximate evaluation time: 8 minutes
links to usage examples: ''
general: This is not an available problem, but could be interesting to show to
researchers which difficulties appear in real-world problems

[kvdblom]

To check what fields from the other info (based on the form) should be added to the template (taken from a different problem): other info: name: null partial evaluations: Not Present full name: Electric Motor Design Optimization constraint properties: Hard Constraints, Soft Constraints, Box Constraints number of constraints: '12' type of dynamicism: '' form of noise model: '' type of noise space: '' other noise properties: '' description of multimodality: Constraints are multimodal key challenges / characteristics: Time-consuming solution evaluation, highly-constrained problem scientific motivation: Challenging to find good solutions in a limited time limitations: 'Unavailability, even if available, it wouldn''t be helpful to use for benchmarking due taking a long time to evaluate a single solution ' implementation languages: Python links to implementations: Implementation not freely available approximate evaluation time: 8 minutes links to usage examples: '' general: This is not an available problem, but could be interesting to show to researchers which difficulties appear in real-world problems

I wonder if we should separate this into numbers for soft and hard constraints:

• number of constraints: '12'

These are currently not yet included

• form of noise model: ''
• type of noise space: ''
• description of multimodality: Constraints are multimodal

[kvdblom]

Work in progress ...

`

Please enter the relevant information.

Fields that are not relevant can be left empty.

* name:
  short: BBOB
  full: Real-Parameter Black-Box Optimization Benchmarking
  suite/generator/single: suite
  objectives:
  number: '1'
  types: single
  1 - single
  2 - bi or multi or multiple
  3 - multi or multiple
  4 or more - many
  scalable - scalable
  variables:
  types: continuous
  continuous or discrete or mixed-integer
  conditional: 'no'
  yes or no
  dimensionality: scalable
  scalable or fixed number or interval or ranges (5-11,19,85)
  constraints:
  present: 'no'
  yes or no
  soft: '0'
  number of soft constraints
  hard: '0'
  number of hard constraints
  boundary/box: 'yes'
  yes or no, in case of yes the number should be equal to the dimensionality under variables
  permutation: 'no'
  yes or no, in case of yes the number should be euqal to the length of the permutation
  dynamic:
  present: 'no'
  yes or no
  types: ''
  ??
  noise: 'no'
  yes or no, guess we don't differentiate types of noise (by now?)
  modality:
  types: 'unimodal, multimodal'
  evaluations:
  multi-fidelity: 'no'
  partial possible: 'no'
  independent objectives: 'no'
  reference:
  links:
  - https://doi.org/10.1080/10556788.2020.1808977
  authors: ''
  contact person: ''
  implementations:
  
  * name: COCO
    link: https://github.com/numbbo/coco
    languges: 'C, Python'
    evaluation time: 'less than a second'
    specific requirements: 'no'
    source:
    real-world:
    degree: ''
    open/closed: ''
    artificial: 'yes'
    other: 'no'
    textual description:
    general info: ''
    motivation: 'evaluate algorithm performance for typical difficulties that occur in continuous problems'
    challenage/key characteristics: ''
    limitations: ''
    other info: ''
    `

This should be included now

[kvdblom]

TODO:

• Definitely still needs a consistency check
• When we are happy (to avoid double work), split into two files: (a) template.yaml and (b) example.yaml

[bn12]

I'm unsure whether this information is needed. I would always say "yes" since I can't imagine a situation where there are no dependencies but I guess there surely are. Anyhow, I wonder whether this has any value?!?

[Dvermetten]

Here, conditional refers to some variables only being active when some condition is met (e.g. in a configuration scenario, variable 'restart_threshold' only is used when 'uses_restarts' is True

[bn12]

If the number is a range (or "any"), it's scalable, if not, it's not.
Isn't it that simple?
And if not, is the field really required or just for experts?

[Dvermetten]

Generally yes, but you might have a suite with a problem with fixed dimensionality 5 and one with 10, then number is 5-10, but the dimensionality is not scalable

[bn12]

General comment:
I'd prefer to have the number of fields as small as possible to just not discourage people from filling the template. That's why I question every entry and try to keep the template as easy as possible.
I also think the template should not address optimisation experts but domain experts who might not be aware of our terminology.

[Dvermetten]

For domain experts, we have the simplified interface via the google form. In the template, we want to leave as much flexibility as possible for people with more detailed information about the problem they are submitting

[bn12]

If the number is 0 then we expect a "no", if not, we expect "yes", or?
Do we need the field?

[bn12]

Can fields "soft" and "hard" be merged to "hardness" or the like?
Possible entries would then be "hard", "soft", or maybe "both"?

[bn12]

Again I think the field might be economised ;)
If the type is empty, presence would be "no" if not, "yes".

[Dvermetten]

It would become hard to infer for optional or unknown, so might be easiest to still have the field even if it can become redundant

[bn12]

Same as for "dynamic".

[bn12]

It's complex ...
What do we need this information for? Do we need it?

I agree that modality is important but there is way more to it than just "uni" or "multi" and just this doesn't really help IMO.

[bn12]

Or "some" in case of different groups of objectives who share evaluation properties?

[Dvermetten]

Yes, we will add it

[bn12]

a possible value could be "any" in the sense of "any of the above" but leaving this field empty is confusing. (please don't contact us?)

[Dvermetten]

Contact person does not need to be an author, could also be maintainer,... or noone if something is abandoned, but we still want to have it in the database. But it should not be under reference to avoid confusion, rather be its own top-level field

[bn12]

Plural? Can people leave multiple implementations? Do we want this?

[Dvermetten]

Yes, it can be useful for example if suites are implemented in different software for different languages

[olafmersmann]

Yes, there can be multiple independent implementations and I think that's fine.

[bn12]

Is it clear that we refer to a single fitness/objective function evaluation time here? (not a whole opt. run)

And I guess we expect the maximum in case of "independent objectives" (different evaluation times) above? Maybe better put a field "time" in the group of "evaluations"? I don't know ...

[Dvermetten]

Yes, we should add a description that we want the time to evaluate a full solution

[bn12]

We don't expect any entry here, right?
Can't we just put a comment to the two (sub-) fields following like
" in case of "real-world" or "real-world-like" "
?

[Dvermetten]

Yes, we can remove this subheader and put it directly at the property description

[olafmersmann]

Do we want an enumeration of constraints? Then we can have one or many soft constraints, one or many box/boundary constraints etc.

It would also allow us to add linear or quadratic constraints in a natural way.
F.E.:

constraints:
  - type: box 
    soft: yes # Defaults to no
    number: 2 # Number of variables that are box constraint
  - type: linear
    soft: no # Default but doesn't hurt
    number: 10
  - type: permutation
    number: 1

No entries in the constraints list means the problem is unconstrained.

[Dvermetten]

Sounds like a good idea

[olafmersmann]

Again, I would prefer a definitive entry. I.e. I would have entered 4 or 5 entries for COCO (one for each language). Hiding the languages in a string seems odd. At least make languages a list.

[Dvermetten]

Agreed, will make this a list

[olafmersmann]

Do we really want this? Most of it sounds like it could be debated forever.