# Graph Driver

## Usage

``` sh
    ./benchdnn --graph [benchdnn-knobs] [graph-knobs] [graph-case] ...
```

* [graph-knobs] can have the following attributes:

  - `--mb=INT` -- Override minibatch size specified in the JSON file default
      case. When set to `0`, use batch size as defined by the
      individual test case. The option doesn't take effect for
      operations that don't support the `mb` concept. The default is `0`.

  - `--in-shapes=ID:SHAPE[*TAG+ID:SHAPE*TAG+...]` -- Override a shape and
      stride of a graph input tensor that includes `ID` in a graph with `SHAPE` and
      `TAG` values. `SHAPE` and `TAG` are separated by a `*`. Multiple
      inputs may be specified using the `+` delimiter.

      If both `--mb` and `--in-shapes` are set, `--mb` takes precedence
      over `--in-shapes`.

      The shape of internal tensors and graph output tensors are inferred
      by the graph driver automatically. By default, the option value is empty,
      meaning values are taken from the original graph.

      `TAG` means the memory layout of that tensor, represented by a string starting with `a`.
      The order may differ; a different order means a different memory layout that users may
      provide according to their own needs. Assume, for instance, a tensor with shape `[1,32,4,4]`
      & stride `[512,16,4,1]`, the stride of which can be represented as a Tag `abcd`. If users
      want to modify stride to `[128,1,128,32]`, `TAG` should be provided as `acdb`, and the
      stride values will be calculated within the Benchdnn-graph.

      Below are several use options for `--in-shapes`:
      1. Modify shape only: `--in-shapes=ID:SHAPE[+ID:SHAPE...]`. Users can modify
              rank as well. Modifying shape to 1D tensor with shape `[0]` is also included:
              `--in-shapes=ID:0[+ID:0+...]`.
      2. Modify stride only: `--in-shapes=ID:TAG[+ID:TAG...]`.
      3. Modify shape and stride: `--in-shapes=ID:SHAPE[*TAG+ID:SHAPE*TAG+...]`.
              Users can modify rank as well.
      4. Modify rank to 0; that is, a scalar with shape [], which is represented by `-`
              in cml: `--in-shapes=ID:-[+ID:-+...]`.

      Examples are provided below.

  - `--op-attrs=ID:ATTR_STRING[+ID:ATTR_STRING]` -- Override a series attributes
              value of op with `ID` in the graph with `ATTR_STRING` values.
              `ATTR_STRING` is `ATTR_NAME:ATTR_VALUE[*ATTR_NAME:ATTR_VALUE]`.
              Multiple attributes value changes may be specified using the `*`
              delimeter. Multiple ops modification may be specified using the `+`
              delimeter. By default, the option value is empty, meaning values are taken from original graph.
  - `--expected-n-partitions=INT` -- Specify the number of expected partitions
      returned from the graph. `INT` is a non-negative integer value. When `INT`
      value is `0`, the check is skipped. By default, the value is `1` which means
      the graph should be fused as one partition.
  - `--dt={undef [default], f32, bf16, f16}` -- Specify the data types in the
    input JSON file. Currently, you can define data types for pure floating-point
    input graph only. For example, you can specify `--dt=f16` for an `f32` graph
    and then test it in `f16`. It has the same effect as changing the data type
    field of all logical tensors in the input JSON file from `f32` to `f16`. If
    `--dt` is not specified or specified as `undef`, the original data types
    contained in the input JSON file will be used for testing.
    - `--dt=ID:DT[+ID:DT]` -- Another format to specify the data types in the
      input JSON file. `ID` specifies the input or output tensor of an operation
      in the JSON file. `DT` is the target data type. To specify the data types of
      multiple tensors, use `+` to concatenate the `ID` and `DT` pairs. An error
      will occur if `ID` is not contained in the JSON file. oneDNN operations have
      restrictions for input and output tensor data types. Changing the data type
      of a tensor may lead to graph construction failures, for example, failure to
      perform the `add_op()` operation in the graph.

* [graph-case] is a JSON file which is dumped by a library or created from
  scratch. It must be passed to the graph driver as `--case=JSON_FILE`. Refer to
  the JSON file example at the end of this document.

The oneDNN Graph serialization feature to dump JSON files at runtime may be enabled
by using the `-DONEDNN_ENABLE_GRAPH_DUMP=ON` build time switch. By default, dump is
disabled. When the build option is on, and the `ONEDNN_GRAPH_DUMP=subgraph` environment
variable is specified, the library generates JSON files with partitions
returned.

## Limitations

* Graph driver doesn't support `--mode-modifier=M` or `--mode=F` (which contains
  `--mode-modifier=M` in it).

## Example

Run the demo `conv_post_ops_fusion` partition
[pattern/f32/conv_post_ops_fusion.json](../inputs/graph/pattern/f32/conv_post_ops_fusion.json)
with default input shapes and op attributes:

```shell
./benchdnn --mode=P --graph --case=./tests/benchdnn/inputs/graph/pattern/f32/conv_post_ops_fusion.json
```

If the JSON file is under `tests/benchdnn/inputs/graph`, we can use the relative
path as shown below,

```shell
./benchdnn --mode=P --graph --case=pattern/f32/conv_post_ops_fusion.json
```

Run the demo pattern with new input shapes by using `--in-shapes`:

```shell
# rewrite input shape only
./benchdnn --mode=C --graph --in-shapes=0:2x64x112x112+1:32x64x2x2 --case=pattern/f32/conv_post_ops_fusion.json
# rewrite stride only
./benchdnn --mode=C --graph --in-shapes=0:dcba+1:dcba --case=pattern/f32/conv_post_ops_fusion.json
# rewrite shape and stride
./benchdnn --mode=C --graph --in-shapes=0:2x64x112x112*dcba+1:32x64x2x2*dcba --case=pattern/f32/conv_post_ops_fusion.json
# rewrite rank
./benchdnn --mode=C --graph --in-shapes=0:2x64x112x112x112+1:32x64x2x2x2 --op-attrs=0:strides:1x1x1*pads_begin:0x0x0*pads_end:0x0x0*dilations:1x1x1 --case=pattern/f32/conv_post_ops_fusion.json
# rewrite rank and stride
./benchdnn --mode=C --graph --in-shapes=0:2x64x112x112x112*edcba+1:32x64x2x2x2*edcba --op-attrs=0:strides:1x1x1*pads_begin:0x0x0*pads_end:0x0x0*dilations:1x1x1 --case=pattern/f32/conv_post_ops_fusion.json
# rewrite rank to 0 rank with shape []
./benchdnn --mode=C --graph --in-shapes=0:- --case=op/f32/add.json
# rewrite to 1D tensor with shape [0]
./benchdnn --mode=C --graph --in-shapes=0:0+1:0 --case=op/f32/add.json
```

Run the demo `conv` op
[op/f32/conv_2d.json](../inputs/graph/op/f32/conv_2d.json) with new strides
attribute by using `--op-attrs`:

```shell
./benchdnn --mode=P --graph --op-attrs=,0:strides:4x4 --case=op/f32/conv_2d.json
```

Run a graph demo batch file [test_graph_ci](../inputs/graph/test_graph_ci):

```shell
./benchdnn --mode=P --graph --batch=test_graph_ci
```

Run same demo batch file on the GPU engine:

```shell
./benchdnn --mode=P --engine=gpu --graph --batch=test_graph_ci
```

Use `-v1` to get more test information, such as graph inputs id and shape,
partition numbers, and so on.

```shell
./benchdnn --mode=P -v1 --graph --mb=1,2,3 --case=op/f32/conv_2d.json
```

Use `--mode=C` or `--mode=c` for correctness testing:

```shell
./benchdnn --mode=C --graph --case=op/f32/conv_2d.json
```

## Demo Cases

Demo JSON files are located in [inputs/graph](../inputs/graph), including
partitions (FP32 MLP partition) and single op (Convolution). Different
data type folders for ops and patterns are available. In general, a JSON file is named as
`workload-pattern_name-additional_info.json`. In this scheme, `workload` stands
for workload name, `pattern_name` stands for the fusion pattern returned by the
library, and `additional_info` differentiates cases based on other settings.
A single op JSON file was named with the op name directly.

## JSON File Example
<details>
    <summary>Conv JSON</summary>

~~~json
{
  "version": "0.5.0",
  "engine_kind": "cpu",
  "fpmath_mode": "strict",
  "graph": [
    {
      "id": 0,
      "name": "Convolution",
      "kind": "Convolution",
      "attrs": {
        "strides": {
          "type": "s64[]",
          "value": [
            2,
            2
          ]
        },
        "pads_begin": {
          "type": "s64[]",
          "value": [
            0,
            0
          ]
        },
        "auto_pad": {
          "type": "string",
          "value": "None"
        },
        "data_format": {
          "type": "string",
          "value": "NCX"
        },
        "pads_end": {
          "type": "s64[]",
          "value": [
            -1,
            -1
          ]
        },
        "groups": {
          "type": "s64",
          "value": 1
        },
        "dilations": {
          "type": "s64[]",
          "value": [
            1,
            1
          ]
        },
        "weights_format": {
          "type": "string",
          "value": "OIX"
        }
      },
      "inputs": [
        {
          "id": 0,
          "dtype": "f32",
          "shape": [
            28,
            512,
            28,
            28
          ],
          "stride": [
            401408,
            1,
            14336,
            512
          ],
          "layout_type": "strided",
          "property_type": "undef"
        },
        {
          "id": 1,
          "dtype": "f32",
          "shape": [
            1024,
            512,
            1,
            1
          ],
          "stride": [
            512,
            1,
            1,
            1
          ],
          "layout_type": "strided",
          "property_type": "constant"
        }
      ],
      "outputs": [
        {
          "id": 2,
          "dtype": "f32",
          "shape": [
            28,
            1024,
            14,
            14
          ],
          "stride": [
            200704,
            1,
            14336,
            1024
          ],
          "layout_type": "strided",
          "property_type": "undef"
        }
      ]
    }
  ]
}
~~~
</details>