Fixing OPENML scenario

[mlindauer]

Hi,

I looked into the "old" OPENML scenario. I already fixed some issues. Right now my tool complains that the order of the columns in algorithm_runs.arff is wrong. Right now it is:

@attribute instance_id STRING
@attribute algorithm STRING
@attribute acc numeric
@attribute repetition numeric
@attribute runstatus {ok, timeout, memout, not_applicable, crash, other}

but it should be

@ATTRIBUTE instance_id STRING
@ATTRIBUTE repetition NUMERIC
@ATTRIBUTE algorithm STRING
@ATTRIBUTE acc NUMERIC
@ATTRIBUTE runstatus {ok, timeout, memout, not_applicable, crash, other}

Furthermore, I wonder why every feature is in its own group. Since there are no feature costs and the runstatus is always ok, we could put all features into one group.
In the end, this is not a real issue.

Best,
Marius

[larskotthoff]

IIRC we put them all into different groups for feature selection -- if everything is in one group, adding one feature is the same as adding all of them.

[mlindauer]

How do we proceed with the other issue? Is it easy for you in R to fix the column order problem? Or who is responsible?

[larskotthoff]

Done. Is everything ok now?

[joaquinvanschoren]

Is there anything I should do?
On Thu, 27 Oct 2016 at 19:16, Lars Kotthoff notifications@github.com
wrote:

Done. Is everything ok now?

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABpQV0dSfY41L8CwLI5mqQ2IzWi3zP-Vks5q4NsVgaJpZM4Kf8DM
.

[mlindauer]

At least I can read the data now (after replacing each ' with ").
To generate plots is not really working because the algorithm names are so long.
Could we shorten them?
e.g., cut the common prefix "weka.classifier"
and move the hyperparameter configuration into a readme?

[larskotthoff]

The hyperparameter configuration is part of the algorithm, otherwise we'd get misleading results when doing selection. Could we instead do something to adjust the plots?

[mlindauer]

Some algorithm names do not even fit in one line on github, e.g.
1160_weka.classifiers.meta.AttributeSelectedClassifier -- -E \weka.attributeSelection.PrincipalComponents -R 0.95 -A 5\ -S \weka.attributeSelection.Ranker -T -1.7976931348623157E308 -N -1 -W weka.classifiers.trees.J48 -- -C 0.25 -M 2

How would you compress it such that it can fit into a plot? All my tries so far lead to plots where the algorithm names were much longer than the actual plot size.
I agree that we should still be able to distinguish between different hyperparameter configurations but the details could go into the readme. For example, the above name could shortened to
PrincipalComponents_Conf1

What do you think?

[larskotthoff]

Hmm, we don't have anything for this specified in the data format specification. I'm hesitant to come up with an ad-hoc version just to fix the plots. If this doesn't break anything else, I'd prefer doing this only in the plots, i.e. having labels "algorithm1", "algorithm2" etc and a legend somewhere that tells you what algorithm1 is.

[joaquinvanschoren]

For plots I would simply cut off the name after the nth characters and add
an ellipsis (...). The main algorithm is always in the beginning of the
string.
On Fri, 28 Oct 2016 at 20:04, Lars Kotthoff notifications@github.com
wrote:

Hmm, we don't have anything for this specified in the data format
specification. I'm hesitant to come up with an ad-hoc version just to fix
the plots. If this doesn't break anything else, I'd prefer doing this only
in the plots, i.e. having labels "algorithm1", "algorithm2" etc and a
legend somewhere that tells you what algorithm1 is.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#10 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABpQVxydyWpvHOwl-9cVvRGhrr67X6ooks5q4jk6gaJpZM4Kf8DM
.

[mlindauer]

For plots I would simply cut off the name after the nth characters and add
an ellipsis (...).

Ok, I will do this.
However, I'm still not so happy with the solution. If I would talk about these algorithms in a paper, I would also need to come up with some meaningful abbreviation (instead of "algorithm1" or "algori(...)"). Therefore I would prefer to consistently solve the problem in this scenario right away.

Maybe we should extend the ASlib format such that parameter configuration can be specified somewhere. For example, in the ASP-POTASSCO scenario it is only 1 solver with 11 different configurations.

[larskotthoff]

That sounds like a good idea. Let's talk more about this in a bigger group.

[hhoos]

On 28 Oct 2016, at 11:04, Lars Kotthoff notifications@github.com wrote:

Hmm, we don't have anything for this specified in the data format specification. I'm hesitant to come up with an ad-hoc version just to fix the plots. If this doesn't break anything else, I'd prefer doing this only in the plots, i.e. having labels "algorithm1", "algorithm2" etc and a legend somewhere that tells you what algorithm1 is.

That sounds like a much better solution than using ellipses, which can get confusing and potentially misleading.

Cheers,

Holger