This PR is to upgrade oneDNN to v3.12.
Improvements
- Improved performance on future Intel Core Ultra processors with Intel AVX10.2 instruction set support (code name Nova Lake). These optimizations are now enabled by default on compatible processors.
- Improved performance on future Intel Xeon processors with Intel AVX10.2 and Intel AMX instruction set support (code name Diamond Rapids). These optimizations are now enabled by default on compatible processors.
- Introduced initial performance optimizations for future integrated GPUs based on Xe3p-LPG architecture.
- Introduced initial performance optimizations for future discrete GPUs based on Xe3p-XPC architecture. This is a preview functionality not recommended for production use.
- Improved f16 matmul performance on Intel Arc Graphics for Intel Core Ultra processor Series 3 (formerly Panther Lake).
- Improved matmul and convolution performance on Arm Neoverse V2 cores.
Validation results on Xeon CPU
- Dynamo benchmarks
The test results are based on the 3 dynamo benchmark suites with three data types.
| Precision | Shape | Wrapper | Thread | Eager Ratio (3.12/3.11.2) | Inductor Ratio (3.12/3.11.2) |
|---|---|---|---|---|---|
| FP32 | Static | cpp | Multiple | 0.9938 | 0.9999 |
| FP32 | Static | cpp | Single | 0.9970 | 0.9967 |
| AMP_BF16 | Static | cpp | Multiple | 1.0098 | 1.0046 |
| AMP_BF16 | Static | cpp | Single | 1.0042 | 1.0068 |
| AMP_FP16 | Static | cpp | Multiple | 0.9998 | 0.9990 |
| AMP_FP16 | Static | cpp | Single | 1.0032 | 1.0027 |
Validation results on Intel B60
Shows NO verified blocker attributable to the oneDNN v3.12 upgrade.
| Area | Total | v3.12 | v3.11.2 | Verdict |
|---|---|---|---|---|
| UT | 296K | 99.7374% pass rate | 99.7360% pass rate | No verified regressions |
| Accuracy | 1903 | 89.86% pass rate | 89.70% pass rate | No verified regressions |
| Performance | 1767 | 0.994x inductor / 1.008x eager | 1.0000x / 1.0000x | No verified sustained drops |
Note: All data are based on release/2.12 with oneDNN change #183248
UT scope owned by https://github.com/intel/torch-xpu-ops
Dynamo benchmark scope
- huggingface (~ 46 models), timm_models (~ 61 models), torchbench (~77 train, ~99 inf)
- float32, float16, bfloat16, amp fp16, amp bf16
- inference, training
Validation results on AArch64
See below for a breakdown of Arm Neoverse-V1 and Arm Neoverse-V2 performance for a set of NLP, Torchbench, and Dynamo model configurations. We see large improvements for most BF16 + Int8 configurations.
Arm Neoverse-V1 - NLP - FP32 - 16 Threads:
| latency geomean (lower is better) |
|---|
| 1.000 |
Arm Neoverse-V2 - NLP - FP32 - 16 Threads:
| latency geomean (lower is better) |
|---|
| 0.997 |
Arm Neoverse-V1 - Torchbench - 16 Threads:
| Mode | Precision | Throughput Geomean (Higher is Better) |
|---|---|---|
| Eager | AMP_BF16 | 1.0058 |
| Eager | FP32 | 1.0017 |
| Eager | INT8 | 1.0519 |
| TorchScript | AMP_BF16 | 1.5880 |
| TorchScript | FP32 | 1.0027 |
Arm Neoverse-V2 - Torchbench - 16 Threads:
| Mode | Precision | Throughput Geomean (Higher is Better) |
|---|---|---|
| Eager | AMP_BF16 | 1.0102 |
| Eager | FP32 | 1.0022 |
| Eager | INT8 | 1.0177 |
| TorchScript | AMP_BF16 | 1.6058 |
| TorchScript | FP32 | 1.0449 |
Arm Neoverse-V1 - Dynamo (16 threads):
| Suite | Compiler | Data Type | Shape | Latency Geomean (Lower is Better) |
|---|---|---|---|---|
| huggingface | eager | AMP_BF16 | dynamic | 1.0021 |
| huggingface | eager | AMP_BF16 | static | 1.0016 |
| huggingface | eager | FP32 | dynamic | 1.0021 |
| huggingface | eager | FP32 | static | 1.0001 |
| huggingface | inductor | AMP_BF16 | dynamic | 1.0007 |
| huggingface | inductor | FP32 | dynamic | 0.9982 |
| huggingface | inductor | FP32 | static | 1.0005 |
| timm | eager | AMP_BF16 | dynamic | 0.9947 |
| timm | eager | AMP_BF16 | static | 0.9954 |
| timm | eager | FP32 | dynamic | 0.9978 |
| timm | eager | FP32 | static | 1.0055 |
| timm | inductor | AMP_BF16 | dynamic | 0.9994 |
| timm | inductor | AMP_BF16 | static | 0.9937 |
| timm | inductor | FP32 | dynamic | 1.0001 |
| timm | inductor | FP32 | static | 1.0002 |
| torchbench | eager | AMP_BF16 | dynamic | 0.9880 |
| torchbench | eager | AMP_BF16 | static | 0.9998 |
| torchbench | eager | FP32 | dynamic | 1.0241 |
| torchbench | eager | FP32 | static | 1.0072 |
| torchbench | inductor | AMP_BF16 | dynamic | 0.9935 |
| torchbench | inductor | AMP_BF16 | static | 0.9758 |
| torchbench | inductor | FP32 | dynamic | 0.9964 |
| torchbench | inductor | FP32 | static | 1.0050 |
Arm Neoverse-V2 - Dynamo: (16 threads)
| Suite | Compiler | Data Type | Shape | Latency Geomean (Lower is Better) |
|---|---|---|---|---|
| huggingface | eager | AMP_BF16 | dynamic | 1.0092 |
| huggingface | eager | AMP_BF16 | static | 1.0024 |
| huggingface | eager | FP32 | dynamic | 1.0013 |
| huggingface | eager | FP32 | static | 1.0004 |
| huggingface | inductor | AMP_BF16 | dynamic | 1.0041 |
| huggingface | inductor | FP32 | dynamic | 1.0017 |
| huggingface | inductor | FP32 | static | 0.9990 |
| timm | eager | AMP_BF16 | dynamic | 1.0002 |
| timm | eager | AMP_BF16 | static | 0.9997 |
| timm | eager | FP32 | dynamic | 0.9897 |
| timm | eager | FP32 | static | 0.9920 |
| timm | inductor | AMP_BF16 | dynamic | 0.9854 |
| timm | inductor | AMP_BF16 | static | 0.9911 |
| timm | inductor | FP32 | dynamic | 0.9861 |
| timm | inductor | FP32 | static | 1.0039 |
| torchbench | eager | AMP_BF16 | dynamic | 1.0025 |
| torchbench | eager | AMP_BF16 | static | 0.9903 |
| torchbench | eager | FP32 | dynamic | 0.9958 |
| torchbench | eager | FP32 | static | 0.9946 |
| torchbench | inductor | AMP_BF16 | dynamic | 0.9903 |
| torchbench | inductor | AMP_BF16 | static | 0.9614 |
| torchbench | inductor | FP32 | dynamic | 0.9953 |
| torchbench | inductor | FP32 | static | 0.9979 |
Pull Request resolved: #181222
Approved by: https://github.com/malfet, https://github.com/atalman