#IntelGPU

[govindhtech]

PyTorch 2.5: Leveraging Intel AMX For Faster FP16 Inference

Intel Advances AI Development through PyTorch 2.5 Contributions

New features broaden support for Intel GPUs and improve the development experience for AI developers across client and data center hardware.

PyTorch 2.5 supports new Intel data center CPUs. Inference capabilities on Intel Xeon 6 processors are improved by Intel Advanced Matrix Extensions(Intel AMX) for eager mode and TorchInductor, which enable and optimize the FP16 datatype. Windows AI developers can use the TorchInductor C++ backend for a better experience.

Intel Advanced Matrix Extensions(Intel AMX)

Overview of Intel Advanced Matrix Extensions (Intel AMX) to fulfill the computational needs of deep learning workloads, Intel Corporation AMX extends and speeds up AI capabilities. The Intel Xeon Scalable CPUs come with this inbuilt accelerator.

Use Intel AMX to Speed Up AI Workloads

A new built-in accelerator called Intel AMX enhances deep learning training and inference performance on the CPU, making it perfect for tasks like image recognition, recommendation systems, and natural language processing.

What is Intel AMX?

Your AI performance is improved and made simpler using Intel AMX. Designed to meet the computational demands of deep learning applications, it is an integrated accelerator on Intel Xeon Scalable CPUs.

AI Inference Performance Enhancement

Improvement of AI Inference Performance Fourth-generation Intel Xeon Scalable processors with Intel AMX and optimization tools were used by Alibaba Cloud‘s machine learning platform (PAI). When compared to the prior generation, this enhanced end-to-end inferencing.

Optimizing Machine Learning (ML) Models

Improving Models for Machine Learning (ML)Throughput increases using the BERT paradigm over the previous generation were shown by Intel and Tencent using Intel AMX. Tencent lowers total cost of ownership (TCO) and provides better services because to the streamlined BERT model.

Accelerate AI with Intel Advanced Matrix Extensions

Use Intel Advanced Matrix Extensions to Speed Up AI. AI applications benefit from Intel AMX’s performance and power efficiency. It is an integrated accelerator specifically designed for Intel Xeon Scalable CPUs.

PyTorch 2.5

PyTorch 2.5, which was recently published with contributions from Intel, offers artificial intelligence (AI) developers enhanced support for Intel GPUs. Supported GPUs include the Intel Data Center GPU Max Series, Intel Arc discrete graphics, and Intel Core Ultra CPUs with integrated Intel Arc graphics.

These new capabilities provide a uniform developer experience and support, and they aid in accelerating machine learning processes inside the PyTorch community. PyTorch with preview and nightly binary releases for Windows, Linux, and Windows Subsystem for Linux 2 may now be installed directly on Intel Core Ultra AI PCs for researchers and application developers looking to refine, infer, and test PyTorch models.

What is PyTorch 2.5?

A version of the well-known PyTorch open-source machine learning framework is called PyTorch 2.5.

New Featuers of PyTorch 2.5

CuDNN Backend for SDPA: SDPA users with H100s or more recent GPUs may benefit from speedups by default with to the CuDNN Backend for SDPA.

Increased GPU Support: PyTorch 2.5 now supports Intel GPUs and has additional tools to enhance AI programming on client and data center hardware.

Torch Compile Improvements: For a variety of deep learning tasks, Torch.compile has been improved to better inference and training performance.

FP16 Datatype Optimization: Intel Advanced Matrix Extensions for TorchInductor and eager mode enable and optimize the FP16 datatype, improving inference capabilities on the newest Intel data center CPU architectures.

TorchInductor C++ Backend: Now accessible on Windows, the TorchInductor C++ backend improves the user experience for AI developers working in Windows settings.

SYCL Kernels: By improving Aten operator coverage and execution on Intel GPUs, SYCL kernels improve PyTorch eager mode performance.

Binary Releases: PyTorch 2.5 makes it simpler for developers to get started by offering preview and nightly binary releases for Windows, Linux, and Windows Subsystem for Linux 2.Python >= 3.9 and C++ <= 14 are supported by PyTorch 2.5.

Read more on govindhtech.com

[mymobilemag]

Intel #Graphics Processing Unit: #ARC Platform Unveiled. What do you know about Intel? What is your take on Graphics Processing Unit? What do you mean by ARC #Platforms? Find out. Link Mentioned In Bio!!!! @intel #mnbile #technology #tech #mymobileindia #arcgpus #arcgpustaff #GPUs #gpuserver #gpushortage #intel #intelligence #intellectualproperty #intelgpu #intelgpu2022 #intelgpugaming #intelgpuforcreators #intelgpusoftwareengineeringinternship https://www.instagram.com/p/CjILRxsvANx/?igshid=NGJjMDIxMWI=

[telepathlc]

@elonmusk

@elonmuskfans intel gpus are already cool and its the only stock not taking a dip lets fucking gooooo meme

good song blunt one march of the elves bandcamp

[gozealouscloudcollection]

Intel：GPU是我們第二重要的產品 2020發布首款獨顯

CPU在未來毫無疑問依然是Intel最重要的產品，但哪些產品是Intel第二重要的呢？ Intel官方的回答有些出人意料。

日前在出席花旗銀行全球技術大會時，Intel雲計算業務副總裁傑森·格雷貝回答了Xeon之後哪款產品會是Intel第二重要的問題，他的答案是“GPU”，也就是說GPU顯卡在地位在Intel心目中是僅次於CPU的。

至於為什麼，原因也很簡單，傑森·格雷貝認為GPU的應用範圍比專業加速器產品更為廣泛。

Intel現在已經是全球最大的GPU廠商，不過這個是說核顯的，而傑森·格雷貝的回答顯然也不是指核顯GPU，而是高性能獨顯GPU，也就是Xe架構為代表的新一代GPU產品。

根據Intel的計劃，該公司將在2020年發布首款獨顯GPU，基於10nm工藝，首發的主要有遊戲GPU廠商，而2021年則會有7nm工藝的高性能GPU，主要用於數據中心，從首發7nm工藝上也可以看出Intel的重視了。

訪問購買頁面:

英特爾旗艦店

.

from Intel：GPU是我們第二重要的產品 2020發布首款獨顯 via KKNEWS

[hacknews]

First Intel-Made Discrete GPU “DG1” Shown Off At CES 2020 #ces2020 #dg1 #intel #inteldiscretegraphics #intelgpu #intelxe #hacking #hacker #cybersecurity #hack #ethicalhacking #hacknews

[govindhtech]

TorchDynamo Method For Improving PyTorch Code Performance

Introduction of Using TorchDynamo to Write PyTorch Programs Faster. Presenters Yuning Qiu and Zaili Wang discuss the new computational graph capture capabilities in PyTorch 2.0 in their webinar, Introduction to Getting Faster PyTorch Programs with TorchDynamo.

TorchDynamo is designed to keep flexibility and usability while speeding up PyTorch scripts with little to no code modifications. It’s important to note that while TorchDynamo was originally used to describe the whole functionality, it is now known by its API name “torch.compile” in the most recent PyTorch documentation. This nomenclature is also used in this lesson.

Principles of Design and Motivation

PyTorch functions mostly in a “imperative mode” (sometimes called eager mode), which is why data scientists and academics have embraced it so enthusiastically due to its Pythonic philosophy and simplicity of use. This mode makes debugging simple and flexible by executing user code step-by-step. For large-scale model deployment, however, imperative execution may not be the best option.

In these cases, performance improvements are often obtained by assembling the model into an efficient computational network. Although they provide graph compilation, traditional PyTorch techniques like FX and TorchScript (JIT) have a number of drawbacks, especially when it comes to managing control flow and backward graph optimization. TorchDynamo was created to solve these issues by offering a more smooth graph capture procedure while maintaining PyTorch’s natural flexibility.

Torch Dynamo: Synopsis and Essential Elements

TorchDynamo works by tying into the frame evaluation process of Python, which is made possible by PEP 523, and examining Python bytecode while it is running. This enables it to execute in eager mode and dynamically capture computational graphs. PyTorch code must be converted by TorchDynamo into an intermediate representation (IR) so that a backend compiler like TorchInductor may optimize it. It functions with a number of important technologies:

AOTAutograd: Enhances training and inference performance by concurrently tracing forward and backward computational graphs in advance. These graphs are divided into manageable chunks by AOTAutograd so that they may be assembled into effective machine code.

PrimTorch: Reduces the original PyTorch operations to a set of around 250 primitive operators, hence simplifying and reducing the number of operators that backend compilers must implement. Thus, PrimTorch improves the built PyTorch models’ extensibility and portability on many hardware platforms.

TorchInductor: The backend compiler that converts the computational graphs that are recorded into machine code that is optimized. Both CPU and GPU optimizations are supported by TorchInductor, including Intel’s contributions to CPU inductor and Triton-based GPU backend optimizations.

Contributions of Intel to TorchInductor

An important factor in improving PyTorch model performance on CPUs and GPUs has been Intel:

CPU Optimizations: For more than 94% of inference and training kernels in PyTorch models, Intel has provided vectorization utilizing the AVX2 and AVX512 instruction sets. Significant gains in performance have resulted from this; depending on the precision utilized (FP32, BF16, or INT8), speedups have ranged from 1.21x to 3.25x.

GPU Support via Triton: OpenAI’s Triton is a domain-specific language (DSL) for Python that is used to write GPU-accelerated machine learning kernels. By using SPIR-V IR to bridge the gap between Triton’s GPU dialect and Intel’s SYCL implementations, Intel has expanded Triton to accommodate their GPU architectures. Triton may be used to optimize PyTorch models on Intel GPUs because to its extensibility.

Guard Systems and Caching

In order to manage dynamic control flow and reduce the need for recompilation, TorchDynamo provides a guard mechanism. Guards monitor the objects that are referred to in every frame and make sure that the graphs that are cached are only utilized again when the calculation has not changed. A guard will recompile the graph, dividing it into subgraphs if needed, if it notices a change. In doing so, the performance overhead is reduced and the accuracy of the compiled graph is guaranteed.

Adaptable Forms and Scalability

Support for dynamic forms is one of TorchDynamo’s primary features. TorchDynamo is capable of handling dynamic input shapes without the need for recompilation, in contrast to earlier graph-compiling techniques that often had trouble with input-dependent control flow or shape fluctuations. This greatly increases PyTorch models’ scalability and adaptability, enabling them to better adjust to changing workloads.

Examples and Use Cases

During the webinar, a number of real-world use cases were shown to show how useful TorchDynamo and TorchInductor are. For example, when optimized with TorchDynamo and TorchInductor, ResNet50 models trained on Intel CPUs using the Intel Extension for PyTorch (IPEX) demonstrated significant increases in performance. Furthermore, comparable performance advantages for models deployed on Intel GPU architectures are promised by Intel’s current efforts to expand Triton for Intel GPUs.

In summary

TorchDynamo and related technologies provide a major step forward in PyTorch’s capacity to effectively aggregate and optimize machine learning models. Compared to older methods like TorchScript and FX, TorchDynamo provides a more adaptable and scalable solution by integrating with Python’s runtime with ease and enabling dynamic shapes.

The contributions from Intel, especially in terms of maximizing performance for both CPUs and GPUs, greatly expand this new framework’s possibilities. As they continue to be developed, researchers and engineers who want to implement high-performance PyTorch models in real-world settings will find that TorchDynamo and TorchInductor are indispensable resources.

Read more on Govindhtech.com

[govindhtech]

SynxFlow Project: A Smooth Migration From CUDA To SYCL

The SynxFlow Project

SynxFlow, an open-source GPU-based hydrodynamic flood modeling software, in CUDA, C++, and Python Data pre-processing and visualization are done in Python while simulations are executed on CUDA. SynxFlow can simulate floods quicker than real-time with hundreds of millions of computational cells and metre-level precision on many GPUs. An open-source software with a simple Python interface, it may be linked into data science workflows for disaster risk assessments. The model has been widely utilized in research and industry, such as to assist flood early warning systems and generate flood maps for (re)insurance firms.

SynxFlow can simulate flooding, landslide runout, and debris flow. Simulations are crucial to emergency service planning and management. A comprehensive prediction of natural disasters can reduce their social and economic costs. In addition to risk assessment and disaster preparedness, SynxFlow flood simulation can help with urban planning, environmental protection, climate change adaptation, insurance and financial planning, infrastructure design and engineering, public awareness, and education.

- Advertisement -

Issue Statement

Several variables make probabilistic flood forecasting computationally difficult:

Large dataset storage, retrieval, and management

Complex real-time data processing requires high-performance computation.

Model calibration and validation needed as real-world conditions change.

Effective integration and data transfer between hydrological, hydraulic, and meteorological models, and more.

For speedier results, a flood forecasting system must process data in parallel and offload compute-intensive operations to hardware accelerators. Thus, the SynxFlow team must use larger supercomputers to increase flood simulation scale and cut simulation time. DAWN, the UK’s newest supercomputer, employs Intel GPUs, which SynxFlow didn’t support.

These issues offered researchers a new goal to make the SynxFlow model performance-portable and scalable on supercomputers with multi-vendor GPUs. They must transition SynxFlow code from CUDA to a cross-vendor programming language in weeks, not years.

Solution Powered by oneAPI

After considering several possibilities, the SynxFlow project team chose the Intel oneAPI Base Toolkit implementation of the Unified Acceleration Foundation-backed oneAPI protocol. All are built on multiarchitecture, multi-vendor SYCL framework. It supports Intel, NVIDIA, and AMD GPUs and includes the Intel DPC++ Compatibility Tool for automated CUDA-to-SYCL code translation.

- Advertisement -

SynxFlow code migration went smoothly. This produced code that automatically translated most CUDA kernels and API calls into SYCL. After auto-translation, some mistakes were found during compilation, but the migration tool’s error-diagnostic indications and warnings made them easy to rectify. It took longer to switch from NVIDIA Collective Communications Library (NCCL)-based inter-GPU communication to GPU-direct enabled Intel MPI library calls because this could not be automated.

To summarize, there has been a promising attempt to transfer a complicated flood simulation code that was built on CUDA to SYCL, achieving both scalability and performance-portability. The conversion has been easy to handle and seamless thanks to the Intel oneAPI Base Toolkit.

Intel hosted a oneAPI Hackfest at the DiRAC HPC Research Facility

DiRAC

The High Performance Super Computer facility in the United Kingdom serving the theoretical communities of Particle Physics, Astrophysics, Cosmology, Solar System and Planetary Science, and Nuclear Physics.

DiRAC’s three HPC services Extreme Scaling, Memory-Intensive, and Data-Intensive are each designed to support the distinct kinds of computational workflows required to carry out their science program. DiRAC places a strong emphasis on innovation, and all of its services are co-designed with vendor partners, technical and software engineering teams, and research community.

Training Series on oneAPI at DiRAC Hackfest

On May 21–23, 2024, the DiRAC community hosted three half-day remote training sessions on the Intel oneAPI Base Toolkit. The training series was designed for developers and/or researchers with varying degrees of experience, ranging from novices to experts.

The cross-platform compatible SYCL programming framework served as the foundation for a variety of concepts that were taught to the attendees. The students were also introduced to a number of Base Kit component tools and libraries that facilitate SYCL. For instance, the Intel DPC++ Compatibility Tool facilitates automated code migration from CUDA to C++ with SYCL; the Intel oneAPI Math Kernel Library (oneMKL) optimizes math operations; the Intel oneAPI Deep Neural Networks (oneDNN) accelerates hackfest and the Intel oneAPI DPC++ Library (oneDPL) expedites SYCL kernels on a variety of hardware. Additionally, the training sessions covered code profiling and the use of Intel Advisor and Intel VTune Profiler, two tools included in the Base Kit for analyzing performance bottlenecks.

DiRAC Hackfest’s oneAPI Hackath on

In order to complete a range of tasks, including parallelizing Fortran code on Intel GPUs, accelerating math operations like the Fast Fourier Transform (FFT) using oneMKL’s SYCL API, and resolving performance bottlenecks with the aid of Intel Advisor and Intel VTune Profiler, the participants improvised their cutting-edge projects using oneAPI tools and libraries.

The participants reported that it was easy to adjust to using oneAPI components and that the code migration process went smoothly. The teams saw a noticeable increase in workload performance with libraries like Intel MPI. Approximately 70% of the teams who took part indicated that they would be open to using oneAPI technologies to further optimize the code for their research projects. Thirty percent of the teams benchmarked their outcomes using SYCL and oneAPI, and they achieved a 100% success rate in code conversion to SYCL.

Start Programming Multiarchitecture Using SYCL and oneAPI

Investigate the SYCL framework and oneAPI toolkits now for multiarchitecture development that is accelerated! Use oneAPI to enable cross-platform parallelism in your apps and move your workloads to SYCL for high-performance heterogeneous computing.

Intel invite you to review the real-world code migration application samples found in the CUDA to SYCL catalog. Investigate the AI, HPC, and rendering solutions available in Intel’s software portfolio driven by oneAPI.

Read more on govindhtech.com

[govindhtech]

Intel Core Ultra 200V Series CPUs Improve AI PC Performance

For the AI PC Age, New Core Ultra Processors Offer Groundbreaking Performance and Efficiency.

Intel Core Ultra

Leading laptop makers may benefit from the exceptional AI performance, interoperability, and power efficiency of Intel Core Ultra 200V series CPUs due to their large size. The Intel Core Ultra 200V series processors are the most efficient x86 CPU family that Intel has ever released. Their performance is outstanding, they provide revolutionary x86 power efficiency, a significant improvement in graphics performance, uncompromised application compatibility, heightened security, and unparalleled AI compute.Image Credit To Intel

With more than 80 consumer designs from more than 20 of the biggest manufacturing partners in the world, including Acer, ASUS, Dell Technologies, HP, Lenovo, LG, MSI, and Samsung, the technology will power the most comprehensive and powerful AI PCs on the market.

Preorders open today, and beginning on September 24, systems will be sold both online and in-store at more than 30 international shops. Beginning in November, all designs with Intel Core Ultra 200V series CPUs and the most recent version of Windows are eligible for a free upgrade that includes Copilot+ PC capabilities.

“Intel’s most recent Core Ultra processors dispel myths about x86 efficiency and set the industry standard for mobile AI and graphics performance. With our relationships with OEMs, ISVs, and the larger tech community, only Intel has the reach to provide customers an AI PC experience that doesn’t compromise.

Customers of today are more and more producing, interacting, playing, and learning while on the road. They need a system with outstanding performance, extended battery life, uncompromised application compatibility, and improved security. It should also be able to use AI hardware via widespread software enablement.

Intel Core Ultra Platform

With up to 50% lower package power and up to 120 total platform TOPS (tera operations per second) across central processing unit (CPU), graphic processing unit (GPU), and neural processing unit (NPU) to deliver the most performant and compatible AI experiences across models and engines, Intel Core Ultra 200V series processors were designed with all of that in mind. With up to four times the power of its predecessor, the fourth-generation NPU is perfect for energy-efficiently performing AI tasks over an extended period of time.

As part of its AI PC Acceleration Program, Intel works with over 100 integrated software suppliers (ISVs) and developers to activate industry-leading platform TOPS in more than 300 AI-accelerated features.

Through carefully calibrated power management and entirely redesigned Performance-cores (P-core) that are optimized for performance per power per area, the new processors provide efficient and remarkable core performance. Additionally, Intel’s most potent Efficient-cores (E-cores) can now handle a greater workload, guaranteeing silent and cool operation.

With a 30% average performance boost, Intel’s new X 2 graphics microarchitecture, which is included in the Intel Core Ultra 200V line of CPUs, represents a considerable improvement in mobile graphics performance. Support for three 4K displays, eight new 2nd Gen Xe-cores, eight upgraded ray tracing units, and new integrated Intel XMX AI. engines with up to 67 TOPS are all included in the integrated Intel Arc GPU. Enhanced XSS kernels allow the AI engines to power creative applications and improve gaming performance.

Intel Core Ultra 200V series processors

A great PC must be a great PC before it can be a great AI PC. With up to three times the performance per thread, an 80% peak performance boost, and up to 20 hours of battery life in productivity use scenarios, Intel Core Ultra 200V series processors are productivity powerhouses. These fantastic PCs are the next step in the AI PC’s progression. With over 500 optimized AI models, extensive ecosystem support, and collaborations with top ISVs, PCs equipped with the newest Intel Core Ultra CPUs enable customers to fully benefit from AI. The new CPUs, with their several powerful AI engines, deliver:

Content Generation: To make video editing simpler and quicker, work more quickly by automatically recognizing changes in the video scene. Use word prompts to unleash your imagination and create beautiful vector and raster art.

Safety: Check whether videos on the internet have been manipulated by using local AI deep-fake detection. AI screening, identification, and safeguarding of important files against dangerous programs and users may protect your PC’s personal data.

Efficiency: One-time video presentation recordings save time, and fresh audio and video including fresh conversation minimize the need for retakes.

Video games: Enhance gaming experiences and increase frames-per-second performance by using AI to provide upscaled, high-quality pictures.

Concerning Intel Evo Edition Utilizing the most recent Intel Core Ultra Processors: The majority of laptop designs with Intel Core Ultra 200V series CPUs will be Intel Evo Edition models, which are rigorously tested and co-engineered with Intel’s partners to provide the best possible AI PC experience.

These laptops are designed to help eliminate latency, limit distractions, and lessen reliance on battery charges by integrating essential platform technologies with system improvements. This ensures amazing experiences from any location. Intel Evo designs, which are new this year, have to achieve improved metrics for quieter and cooler operation.

Features consist of:

Performance and responsiveness in ultra-thin designs that are cooler and quieter.

Extended battery life in practice.

integrated security that reduces vulnerabilities and aids in stopping malware assaults.

Integrated Intel Arc graphics provide faster game development and more fluid gameplay, even while playing on the fly.

Connectivity that is lightning fast thanks to Intel Wi-Fi 7 (5 Gig).

The ability to use Thunderbolt Share to charge a PC, transmit data, and connect it to numerous displays.

Wake up instantly and charge quickly.

The highest accreditation for sustainability, EPEAT Gold.

What’s Next: Starting today, consumers may pre-order consumer devices equipped with Intel Core Ultra 200V series processors. Commercial products based on the Intel vPro platform will be released in the Next year.

IFA 2024 conference

Image Credit To Intel

The next generation of Intel Core Ultra processors, code-named Lunar Lake, was introduced ahead of the IFA 2024 conference by Jim Johnson, senior vice president and general manager of the Client Business Group, and Michelle Johnston Holthaus, executive vice president and general manager of Intel’s Client Computing Group. Partners from Intel joined them in launching a line of processors that redefines mobile AI performance.

The executives of Intel demonstrated how the new processors’ remarkable core performance, remarkable x86 power efficiency, revolutionary advances in graphics performance, and AI processing capacity provide users everything they need to create, connect, play, or study on the move.

Read more on govindhtech.com

[govindhtech]

Utilizing llama.cpp, LLMs can be executed on Intel GPUs

The open-source project known as llama.cpp is a lightweight LLM framework that is gaining greater and greater popularity. Given its performance and customisability, developers, scholars, and fans have formed a strong community around the project. Since its launch, GitHub has over 600 contributors, 52,000 stars, 1,500 releases, and 7,400 forks. More hardware, including Intel GPUs seen in server and consumer products, is now supported by llama.cpp as a result of recent code merges. Hardware support for GPUs from other vendors and CPUs (x86 and ARM) is now combined with Intel’s GPUs.

Georgi Gerganov designed the first implementation. The project is mostly instructional in nature and acts as the primary testing ground for new features being developed for the machine learning tensor library known as ggml library. Intel is making  AI more accessible to a wider range of customers by enabling inference on a greater number of devices with its latest releases. Because Llama.cpp is built in C and has a number of other appealing qualities, it is quick.

16-bit float compatibility

Support for integer quantisation (four-, five-, eight-, etc.)

Absence of reliance on outside parties

There are no runtime memory allocations.

Intel GPU SYCL Backend

GGM offers a number of backends to accommodate and adjust for various hardware. Since oneAPI supports GPUs from multiple vendors, Intel decided to construct the SYCL backend using their direct programming language, SYCL, and high-performance BLAS library, oneMKL. A programming model called SYCL is designed to increase hardware accelerator productivity. It is an embedded, single-source language with a domain focus that is built entirely on C++17.

All Intel GPUs can be used with the SYCL backend. Intel has confirmed with:

Flex Series and Data Centre GPU Max from Intel

Discrete GPU Intel Arc

Intel Arc GPU integrated with the Intel Core Ultra CPU

In Intel Core CPUs from Generations 11 through 13: iGPU

Millions of consumer devices can now conduct inference on Llama since llama.cpp now supports Intel GPUs. The SYCL backend performs noticeably better on Intel GPUs than the OpenCL (CLBlast) backend. Additionally, it supports an increasing number of devices, including CPUs and future processors with  AI accelerators. For information on using the SYCL backend, please refer to the llama.cpp tutorial.

Utilise the SYCL Backend to Run LLM on an Intel GPU

For SYCL, llama.cpp contains a comprehensive manual. Any Intel GPU that supports SYCL and oneAPI can run it. GPUs from the Flex Series and Intel Data Centre GPU Max can be used by server and cloud users. On their Intel Arc GPU or iGPU on Intel Core CPUs, client users can test it out. The 11th generation Core and later iGPUs have been tested by Intel. While it functions, the older iGPU performs poorly.

The memory is the only restriction. Shared memory on the host is used by the iGPU. Its own memory is used by the dGPU. For llama2-7b-Q4 models, Intel advise utilising an iGPU with 80+ EUs (11th Gen Core and above) and shared memory that is greater than 4.5 GB (total host memory is 16 GB and higher, and half memory could be assigned to iGPU).

Put in place the Intel GPU driver

There is support for Windows (WLS2) and Linux. Intel suggests Ubuntu 22.04 for Linux, and this version was utilised for testing and development.

Linux:sudo usermod -aG render username sudo usermod -aG video username sudo apt install clinfo sudo clinfo -l

Output (example):Platform #0: Intel(R) OpenCL Graphics -- Device #0: Intel(R) Arc(TM) A770 Graphics

orPlatform #0: Intel(R) OpenCL HD Graphics -- Device #0: Intel(R) Iris(R) Xe Graphics \[0x9a49\]

Set the oneAPI Runtime to ON

Install the Intel oneAPI Base Toolkit first in order to obtain oneMKL and the SYCL compiler. Turn on the oneAPI runtime next:

First, install the Intel oneAPI Base Toolkit to get the SYCL compiler and oneMKL. Next, enable the oneAPI runtime:

Linux: source /opt/intel/oneapi/setvars.sh

Windows: “C:\Program Files (x86)\Intel\oneAPI\setvars.bat\” intel64

Run sycl-ls to confirm that there are one or more Level Zero devices. Please confirm that at least one GPU is present, like [ext_oneapi_level_zero:gpu:0].

Build by one-click:

Linux: ./examples/sycl/build.sh

Windows: examples\sycl\win-build-sycl.bat

Note, the scripts above include the command to enable the oneAPI runtime.

Run an Example by One-Click

Download llama-2–7b.Q4_0.gguf and save to the models folder:

Linux: ./examples/sycl/run-llama2.sh

Windows: examples\sycl\win-run-llama2.bat

Note that the scripts above include the command to enable the oneAPI runtime. If the ID of your Level Zero GPU is not 0, please change the device ID in the script. To list the device ID:

Linux: ./build/bin/ls-sycl-device or ./build/bin/main

Windows: build\bin\ls-sycl-device.exe or build\bin\main.exe

Synopsis

All Intel GPUs are available to LLM developers and users via the SYCL backend included in llama.cpp. Kindly verify whether the Intel laptop, your gaming PC, or your cloud virtual machine have an iGPU, an Intel Arc GPU, or an Intel Data Centre GPU Max and Flex Series GPU. If so, llama.cpp’s wonderful LLM features on Intel GPUs are yours to enjoy. To add new features and optimise SYCL for Intel GPUs, Intel want developers to experiment and contribute to the backend. The oneAPI programming approach is a useful project to learn for cross-platform development.

Read more on Govindhtech.com

[govindhtech]

PyTorch 2.4 to Speed Up AI Tasks Support for Intel GPUs

PyTorch 2.4 Launches to Speed Up AI Tasks with Initial Support for Intel GPUs. In order to further speed up AI tasks, PyTorch 2.4 now offers initial support for the  Intel Data Centre GPU Max Series, which integrates  Intel GPUs and the SYCL software stack into the standard PyTorch stack.

Advantages

With Intel GPU support, customers have more options for GPUs and can use a consistent front-end and back-end GPU programming model. Workloads can now be deployed and operated on Intel GPUs with little coding required. To support streaming devices, this version generalizes the PyTorch device and runtime (device, stream, event, generator, allocator, and guard). The generalization facilitates not only PyTorch’s deployment on widely available hardware but also the integration of many hardware back ends.

Integrated PyTorch provides continuous software support, standardized software distribution, and consistent product release schedules, all of which will improve the experience for users of Intel GPUs.

An Overview of Support for Intel GPUs

Eager mode and graph mode are supported in the PyTorch built-in front end thanks to Intel GPU support that has been up streamed into the program. The SYCL programming language is now utilized to implement popular Aten operators in the eager mode. OneAPI Math Kernel Library (oneMKL) and oneAPI Deep Neural Network Library (oneDNN) are used to highly optimize the most performance-critical graphs and operators. To perform the optimization for Intel GPUs and to integrate Triton, the graph mode (torch.compile) now has an enabled Intel GPU back end.

PyTorch 2.4 now includes the necessary parts of Intel GPU support: Aten operators, oneDNN, Triton, Intel GPU source build, and integration of Intel GPU tool chains. In the meantime, PyTorch Profiler which is built on an integration between Kineto and oneMKL is being actively worked on in front of the forthcoming PyTorch 2.5 release. The front-end and back-end enhancements for Intel GPUs that are currently being implemented into PyTorch are depicted in Figure 1.Image Credit To Intel

PyTorch 2.4 Features

Apart from offering essential functionalities for training and inference on the Intel Data Centre GPU Max Series, the PyTorch 2.4 release for Linux maintains the same user interface as other supported hardware for PyTorch.

Using an Intel GPU, PyTorch 2.4 features include:

Workflows for inference and training.

The core eager functions as well as torch.compile are supported, and both eager and compile modes can fully run a Dynamo Hugging Face benchmark.

Data types like automated mixed precision (AMP), BF16, FP32, and so on.

operates on the Intel Data Centre GPU Max Series and Linux.

PyTorch 2.5

The first Intel GPU from the Intel Data Centre GPU Max Series is now available in the PyTorch ecosystem for AI workload acceleration thanks to the Intel GPU on PyTorch 2.4 first support (prototype) release.

In order to achieve beta quality in the PyTorch 2.5 release, they are constantly improving the functionality and performance of the Intel GPU support. Intel Client GPUs will be added to the list of GPUs supported for AI PC use cases as the product develops further. They’re also investigating more features for PyTorch 2.5, like:

Eager Mode: Completely execute Dynamo Torchbench and TIMM eager mode, and implement additional Aten operators.

Torch.compile: Optimise performance while running Dynamo Torchbench and TIMM benchmark compile mode in full.

To support Intel GPU, enable torch.profile under the profiler and utilities section.

Distribution of PyPI wheels.

Support for Windows and the  Intel Client GPU Series.

They invite the community to assess these latest additions to PyTorch’s  Intel GPU support.

Intel Extensions For PyTorch

The most recent performance enhancements for Intel devices are added to PyTorch using the Intel Extension. The Intel XeMatrix Extensions (XMX) AI engines on Intel discrete GPUs and Intel Advanced Vector Extensions 512 (Intel AVX-512) Vector Neural Network Instructions (VNNI) and Intel Advanced Matrix Extensions (Intel AMX) on Intel CPUs are utilized in optimizations. Additionally, the PyTorch xpu device, in conjunction with Intel Extension for PyTorch, facilitates simple GPU acceleration for Intel discrete GPUs.

Workloads and models for Generative AI (GenAI) have become increasingly common in today’s technological environment. These GenAI applications are mostly driven by large language models, or LLMs. The Intel Extension for PyTorch has added special optimizations for a few Large Language Models (LLMs) as of version 2.1.0. See Large Language Models (LLMs) section for additional details on LLM optimizations.

For Python programs, the extension can be loaded as a module, and for C++ projects, it can be linked as a library. It can be dynamically enabled in Python programs by importing intel_extension_for_pytorch.

Buildings

Eager Mode: Custom Python modules (including fusion modules), optimum optimizers, and INT8 quantization APIs are added to the PyTorch frontend in the eager mode. Using extended graph fusion passes, eager-mode models can be transformed into graph mode to further increase performance.

Graph Mode: Performance is enhanced by fusions’ reduction of operator/kernel invocation overhead in the graph mode. In PyTorch, the graph mode typically produces better results from optimization techniques like operation fusion than the eager mode does.

They are enhanced by the Intel Extension for PyTorch, which offers more thorough graph optimizations. Supported graph modes are PyTorch Torchscript and TorchDynamo. They advise you to use torch.jit.trace() instead of torch.jit.script() when using Torchscript since it typically supports a larger variety of workloads. The ipex backend can deliver strong performance with TorchDynamo.

CPU Optimization: Based on the detected instruction set architecture (ISA), Intel Extension for PyTorch automatically assigns operators to underlying kernels on the CPU. The addon makes use of the Intel hardware’s vectorization and matrix acceleration units. For enhanced performance, the runtime extension provides weight sharing and more precise thread runtime management.

Intel GPU

GPU Optimisation: The PyTorch dispatching method is used to implement and register optimized operators and kernels on the GPU. The intrinsic vectorization and matrix calculating capabilities of Intel GPU hardware enhance certain operators and kernels. The DPC++ compiler, which supports both the most recent SYCL standard and several extensions to the SYCL standard, is used by the Intel Extension for PyTorch for GPU. These extensions are located in the sycl/doc/extensions directory.

Encouragement

GitHub issues are used by the team to keep track of bugs and enhancement requests. Check to see whether your issue has previously been reported on GitHub before making a proposal or bug report.

Read more on govindhtech.com