Defining search space

Next, it’s time to define a search space for hyperparameter. A search space is represented as JSON element. For example, a search space for embedding dimensionality looks like following:

{
  "type": "int",
  "attributes": {
    "name": "embedding_dim",
    "low": 64,
    "high": 128
  }
}

type should be int, float, or categorical. attributes is arguments that Optuna takes. name is a name of hyperparameter. low and high are the range of a parameter. For categorical distribution, choices is available. For more information about attributes, please see the Optuna API reference (suggest_float, suggest_int, and suggest_categorical).

The entire example of AllenNLP configuration for allennlp-optuna is following:

• hparams.json

[
  {
    "type": "int",
    "attributes": {
      "name": "embedding_dim",
      "low": 64,
      "high": 128
    }
  },
  {
    "type": "int",
    "attributes": {
      "name": "max_filter_size",
      "low": 2,
      "high": 5
    }
  },
  {
    "type": "int",
    "attributes": {
      "name": "num_filters",
      "low": 64,
      "high": 256
    }
  },
  {
    "type": "int",
    "attributes": {
      "name": "output_dim",
      "low": 64,
      "high": 256
    }
  },
  {
    "type": "float",
    "attributes": {
      "name": "dropout",
      "low": 0.0,
      "high": 0.5
    }
  },
  {
    "type": "float",
    "attributes": {
      "name": "lr",
      "low": 5e-3,
      "high": 5e-1,
      "log": true
    }
  }
]