#int4

[govindhtech]

INT8 & INT4 Weight Only Quantization WOQ On Intel Extension

Weight Only Quantization(WOQ)

A practical guide to Large Language Models (LLMs) quantization. The capabilities, uses, and complexity of large language models (LLMs) have all significantly risen in recent years. With an ever-increasing amount of parameters, weights, and activations, LLMs have become larger and more intelligent.

However, They usually have to compress LLMs without significantly sacrificing their performance in order to increase the number of possible deployment targets and lower the cost of inference. Large neural networks, including language models, may be made smaller using a variety of methods. Quantization is one such crucial method.

WOQ meaning

In machine learning, especially in deep learning, Weight Only Quantization (WOQ) is a technique that minimizes the size of neural network models without compromising their functionality. It entails quantizing just the neural network’s weights the parameters that define the behavior of the model into a format with less precision (e.g., 8-bit instead of 32-bit).

This article provides an example of code that uses the Intel Extension for Transformers tool to conduct Weight Only Quantization (WOQ) on an LLM (Intel/neural-chat-7b model) for both INT8 and INT4.

How does quantization work?

INT8 Vs INT4

The process of switching to lower precision data types, such as float16, INT8 or INT4, from high-precision representation, such as float32, for weights and/or activations is known as quantization. Lower precision may greatly minimize the amount of memory needed.

While this may seem simple in principle, there are a lot of subtleties to consider, and computing data type is the most crucial warning to know. Certain operations need us to scale the representation back to high precision at runtime since not all operations support or have low-precision implementation. Although there is some additional cost, they may lessen its effects by using tools like Intel Neural Compressor, OpenVINO toolkit, and Neural Speed.

Because these runtimes include optimized implementations of several operators for low-precision data types, upscale values to high-precision is not necessary, resulting in improved speed and reduced memory use. If lower-precision data types are supported by your hardware, the performance improvements are substantial. For instance, support for float16 and bfloat16 is included into Intel Xeon Scalable processors of the 4th generation.

Therefore, quantization only serves to lower the model’s memory footprint; nevertheless, it may also introduce some cost during inference. Using optimized runtimes and the newest hardware is required to obtain both memory and performance improvements.

What Does WOQ Mean?

There are several methods for quantizing models. Model weights and activations the output values produced by every neuron in a layer are often quantized. One of these quantization methods, called Weight Only Quantization(WOQ), preserves the original accuracy of the activations while only quantizing the model weights. Faster inference and a reduced memory footprint are the clear advantages. In actual use, WOQ improves performance without appreciably affecting accuracy.

Code Execution

The Intel/neural-chat-7b-v3-3 language model’s quantization procedure is shown in the provided code sample. The model, which is an improved version of Mistral-7B, is quantized using Weight Only Quantization (WOQ) methods made available by the Intel Extension for Transformers.

With only one line of code, developers can easily use the power of Intel technology for their Generative AI workloads. You import AutoModelForCausualLM from Intel Extension for Transformers rather of the Hugging Face transformers library, and everything else stays the same.

1. From intel_extension_for_transformers.transformers import AutoModelForCausalLM

For INT8 quantization, just set load_in_8bit to True.

1. # INT8 quantization 2. Q8_model = AutoModelForCausalLM.from_pretrained( 3. model_name, load_in_8bit=True)

Similarly, for INT4 quantization set load_in_4bit to True.1. # INT4 quantization 2. q4_model = AutoModelForCausalLM.from_pretrained( 3. model_name, load_in_4bit=True)

The Hugging Face transformers library may be used in the same way for implementation.

If you set device to GPU, the aforementioned code snippets will utilize BitandBytes for quantization. This makes your code run much faster without requiring any code changes, regardless of whether you are utilizing a CPU or GPU.

GGUF model in operation

A binary file format called GGUF was created expressly to store deep learning models like LLMs especially for CPU inference. It has several important benefits, such as quantization, efficiency, and single-file deployment. They will be utilizing the model in GGUF format in order to maximize the performance of their Intel hardware.

Generally, one would need to utilize an extra library like Llama_cpp in order to execute models in GGUF format. Still, you may use it Intel Extension for Transformers library to run GGUF models since Neural Speed is built on top of Llama_cpp.1. model = AutoModelForCausalLM.from_pretrained( 2. model_name=“TheBloke/Llama-2-7B-Chat-GGUF”, 3. model_file=“llama-2-7b-chat.Q4_0.gguf” 4. )

Take a look at the code example. The code example demonstrates how to use Intel’s AI Tools, Intel Extension for Transformers, to quantize an LLM model and how to optimize your Intel hardware for Generative AI applications.

INT4 vs INT8

Quantizing LLMs for Inference in INT4/8

Better quantization approaches are becoming more and more necessary as models become bigger. However, what is quantization exactly? Model parameters are represented with less accuracy by quantization. For example, using float16 to represent model weights instead of the widely used float32 may reduce storage needs by half.

Additionally, it improves performance at lesser precision by lowering computational burden. Nevertheless, a drawback of quantization is a little reduction in model accuracy. This happens when accuracy decreases and parameters have less representation power. In essence, quantization allows us to sacrifice accuracy for better inference performance (in terms of processing and storage).

Although there are many other approaches to quantization, this sample only considers Weight Only Quantization (WOQ) strategies. Model weights and activations the output values produced by every neuron in a layer are often quantized. But only the model weights are quantized by WOQ; activations remain unaltered. In actual use, WOQ improves performance without appreciably affecting accuracy.

The transformers library from HuggingFace makes quantization easier by offering clear choices. To enable quantization, users just need to specify the load_in_4bit or load_in_8bit option to True. But there’s a catch: only CUDA GPUs can use this capability. Unfortunately, only CUDA GPU devices can use the BitsandBytes configuration that is automatically built when these arguments are enabled. For consumers using CPUs or non-CUDA devices, this presents a problem.

The Intel team created Intel Extension for Transformers (ITREX), which improves quantization support and provides further optimizations for Intel CPU/GPU architectures, in order to overcome this constraint. Users may import AutoModelForCasualLM from the ITREX library rather of the transformers library in order to use ITREX. This allows users, irrespective of their hardware setup, to effortlessly use quantization and other improvements.

The from_pretrained function has been expanded to include the quantization_config, which now takes in different settings for CUDA GPUs and CPUs to perform quantization, including RtnConfig, AwqConfig, TeqConfig, GPTQConfig, and AutoroundConfig. How things behave when you set the load_in_4bit or load_in_8bit option to True is dependent on how your device is configured.

BitsAndBytesConfig will be used if the CUDA option is selected for your device. RtnConfig, which is specifically tailored for Intel CPUs and GPUs, will be used, nonetheless, if your device is set to CPU. In essence, this offers a uniform interface for using Intel GPUs, CPUs, and CUDA devices, guaranteeing smooth quantization across various hardware setups.

Read more on govindhtech.com

[newzzwired]

Qualcomm Snapdragon 8 Gen 2 SoC With Wi-Fi 7, INT4, Ray Tracing, and More Launched: All Details

Qualcomm Snapdragon 8 Gen 2 SoC With Wi-Fi 7, INT4, Ray Tracing, and More Launched: All Details

Qualcomm unveiled Snapdragon 8 Gen 2 SoC at its annual Snapdragon Tech Summit on Wednesday. The new mobile 5G platform brings a list of upgrades over last year’s Snapdragon 8 Gen 1 SoC and is claimed to be 40 percent more power efficient than the older model. It offers real-time raytracing for gaming and supports INT4 and Wi-Fi 7. The Snapdragon 8 Gen 2 supports new image sensors like the…

[roadandruingame]

RaR Musings #7: Meaningful Mechanics

I saw a post this week about other people in the ttrpg design space, lamenting their years of work, and being dismissed for their project seeming like "a dnd clone". A fair concern, to be sure, but it would turn out the criticism stemmed from having a fantasy themed roleplaying game, that uses a d20 and adds proficiency, has character creation that involves classes, and spellcasting with multiple levels of spells. Others suggested there might be similarities if you use the standard stats, like STR, DEX, and INT.

So what's a guy with a fantasy themed roleplaying game that uses Xd10, adding proficiency, has a character creation engine that has classes as a minor element, and spellcasting with a mana system allowing you to cast spells at a higher level, using some basic stats, to do?

Firstly: not worry about it. Creativity is iterative, and DND has been the fantasy roleplay standard for nigh on 50 years, having affected pop culture and videogame design alike. It'd be hard NOT to have anything similar to it, and for those who have no experience outside of DND, dipping a toe outside that space can seem daring and adventurous. The writer is probably upset because they don't understand how generally meaningless their reinventing of the wheel was in terms of convincing people to play their game instead; in fact, there wasn't any mention of WHY he made the effort to design his own game in the first place. Was it distaste for existing products? Because he had vision? Or just to prove that he could do it too, a kind of intellectual parroting?

Game mechanics can't be copywritten, so while it's not illegal to copy mechanics, there needs to be certified thought put into what those mechanics are meant to achieve, and why they may fail to do so.

As an example: both d20 games and Road and Ruin involve rolling dice to generate a random value, and then adding your proficiency as a flat number.

DND falls down here because even high proficiency, like +11 or +13, barely crests over half of the value generated by random d20, much less the more regular +1 to +6. This means a specialist, someone who has lifelong expertise at their craft, can still bungle even a basic action, giving other players a chance to perform, but completely botching the class fantasy of being a specialist, and there's no coded mechanics for varying levels of success or failure to even reward being a specialist beyond increased binary success rate. Multiple overlapping proficiencies don't have cumulative value, and outside of house rules, you can't mix and match Attribute to Proficiency, such as using Strength for Intimidation. However, the system is simplistic, and easy to understand. Not having different values for different proficiencies means only having to refer to a single number as a bonus, which makes stat scaling much more predictable, and as mentioned, giving other players the limelight means the skill monkeys won't hog it.

Road and Ruin HAD a much more 'unique' skill check system; roll your attribute (1-10) as Xd10, and your proficiency (two 0-5 proficiencies combined) determined the minimum score any dice could land. Dice were adjusted, totalled, and the sum divided by 10 to find Success Rate, with scores of 1 or higher expected. This ended up being too much adjusting and adding; it produced the ideal values, but was too slow, and not very fun, especially to do repetitively. Worse, it didn't enable 'skill' to exceed 'raw talent'; you needed a high attribute for the guaranteed 'floor' that proficiency provided to matter, and I wanted those with training to potentially exceed those without training. If INT4 rolls 4d10, and Proficiency 3 meant you couldn't get below a 3 on each, for a 'floor' of 12-40, that still meant an average ~22, regardless of if you were trained or not. Specialization 'rolled' an additional 1d10, but set it aside as an automatic 10, thus improving skill checks beyond what was possible via random dice rolls, raising both floor and ceiling by 10, but not solving the issue of speed or reliability.

So now, Road and Ruin has a Roll + Proficiency system too, except you roll Xd10 (1-5), and Proficiency is two scores (0-5 each), combined, and multiplied by Specialization, with a cheat-sheet of the most common Proficiency results for your character. Adding the dice, and Proficiency, before finding successes, is still slow, but faster now, and due to the multiplication of scores and specialization, your character may even automatically succeed basic tasks, without the need for a roll at all. Such skillmonkeying requires utmost devotion though, and is far better suited to an NPC assistant; but, said NPC will still be built using the same mechanics as what goes into a character, making it easier to understand and appreciate their service.

More importantly: I like it. I understand that others might not; it doesn't have the hallmarks of DND's 'gamble' economy, getting high rolls and confetti when you hit a 20, but frankly, I'm building this game for me, not for people who are satisfied with DND. Even my nine attributes are inspired by World of Darkness, though slightly redefined to suit the needs of my setting instead, and the proficiency skill list is entirely my own, designed to offer as many cases of two overlapping skills as possible. Using any attribute in the skill check, based on what you aim to affect rather than what the proficiency is most known for (using DEX and herbalism to get plant clippings, or INT and herbalism to recall plant facts, for example) is a much more direct and diverse way to handle skill checks, rather than trying to remember whether Nature in DND is Intelligence or Wisdom, and why. Rolling multiple dice instead of 1d20 helps protect against fringe rolls, making the rare cases truly rare, as well as creating a market for spells, equipment, and abilities that affect your skill checks to have meaningful use, rather than simply adding a +1.

But I'm having fun doing all this. Road and Ruin began because I was upset with DND, and over the years, I've done a lot of work, first to intentionally distance it from DND, and only later to begin to paint it in my own colors, doing what I want, not in rebellion of what I don't. Anybody looking to design their own systems should be more preoccupied with how their mechanics feel; if people think that it's too similar to an existing product, one that you intentionally avoided? Then that's tough beans for them. They don't get to define how you have fun, and at the end of the day, that's what playing, and designing, a game is all about.

[slunch]

Hi!! Some questions for you: what are some things you’ve learned recently? What’s your favorite smell? What’s your dream vacation?

hmm, let's see...

learned recently: how to make a UML diagram, when to water a jacaranda, the fact that plant pots need drains (but don't put in rocks because it pushes up the saturation zone), existence of int4 quantization to accelerate neural nets, and the fact that my cat may have started drinking out of the toilet this week

favorite smell: easy, burning the fuck out of a corn tortilla no pan raw on the stovetop

dream vacation: this one's harder to say, i think something crazy like a writer's or artists's retreat in some remote place would be pretty fun. just disconnect from life and work for a week or month or two, do nothing but work on the thing and go for walks and eat simple meals. like a firewatch tower or something.

[smoqueen]

negan’s int4 is so amazing its one of my favorite moves ever he like teleports two feet forward on the startup animation its so stupid

[forlinx]

Introduction to RK3588

What is RK3588?

RK3588 is a universal SoC with ARM architecture, which integrates quad-core Cortex-A76 (large core) and quad-core Cortex-A55(small core). Equipped with G610 MP4 GPU, which can run complex graphics processing smoothly. Embedded 3D GPU makes RK3588 fully compatible with OpenGLES 1.1, 2.0 and 3.2, OpenCL up to 2.2 and Vulkan1.2. A special 2D hardware engine with MMU will maximize display performance and provide smooth operation. And a 6 TOPs NPU empowers various AI scenarios, providing possibilities for local offline AI computing in complex scenarios, complex video stream analysis, and other applications. Built-in a variety of powerful embedded hardware engines, support 8K@60fps H.265 and VP9 decoders, 8K@30fps H.264 decoders and 4K@60fps AV1 decoders; support 8K@30fps H.264 and H.265 encoder, high-quality JPEG encoder/decoder, dedicated image pre-processor and post-processor.

RK3588 also introduces a new generation of fully hardware-based ISP (Image Signal Processor) with a maximum of 48 million pixels, implementing many algorithm accelerators, such as HDR, 3A, LSC, 3DNR, 2DNR, sharpening, dehaze, fisheye correction, gamma Correction, etc., have a wide range of applications in graphics post-processing. RK3588 integrates Rockchip's new generation NPU, which can support INT4/INT8/INT16/FP16 hybrid computing. Its strong compatibility can easily convert network models based on a series of frameworks such as TensorFlow / MXNet / PyTorch / Caffe. RK3588 has a high-performance 4-channel external memory interface (LPDDR4/LPDDR4X/LPDDR5), capable of supporting demanding memory bandwidth.

RK3588 Block Diagram

Advantages of RK3588?

Computing: RK3588 integrates quad-core Cortex-A76 and quad-core Cortex-A55, G610 MP4 graphics processor, and a separate NEON coprocessor. Integrating the third-generation NPU self-developed by Rockchip, computing power 6TOPS, which can meet the computing power requirements of most artificial intelligence models.

Vision: support multi-camera input, ISP3.0, high-quality audio;

Display: support multi-screen display, 8K high-quality, 3D display, etc.;

Video processing: support 8k video and multiple 4k codecs;

Communication: support multiple high-speed interfaces such as PCIe2.0 and PCIe3.0, USB3.0, and Gigabit Ethernet;

Operating system: Android 12 is supported. Linux and Ubuntu will be developed in succession;

FET3588-C SoM based on Rockchip RK3588

Forlinx FET3588-C SoM inherits all advantages of RK3588. The following introduces it from structure and hardware design.

1. Structure:

The SoM size is 50mm x 68mm, smaller than most RK3588 SoMs on market;

100pin ultra-thin connector is used to connect SoM and carrier board. The combined height of connectors is 1.5mm, which greatly reduces the thickness of SoM; four mounting holes with a diameter of 2.2mm are reserved at the four corners of SoM. The product is used in a vibration environment can install fixing screws to improve the reliability of product connections.

2. Hardware Design:

FET3568-C SoM uses 12V power supply. A higher power supply voltage can increase the upper limit of power supply and reduce line loss. Ensure that the Forlinx’s SoM can run stably for a long time at full load. The power supply adopts RK single PMIC solution, which supports dynamic frequency modulation.

FET3568-C SoM uses 4 pieces of 100pin connectors, with a total of 400 pins; all the functions that can be extracted from processor are all extracted, and ground loop pins of high-speed signal are sufficient, and power supply and loop pins are sufficient to ensure signal integrity and power integrity.

The default memory configuration of FET3568-C SoM supports 4GB/8GB (up to 32GB) LPDDR4/LPDDR4X-4266; default storage configuration supports 32GB/64GB (larger storage is optional) eMMC; Each interface signal and power supply of SoM and carrier board have been strictly tested to ensure that the signal quality is good and the power wave is within specified range.

PCB layout: Forlinx uses top layer-GND-POWER-bottom layer to ensure the continuity and stability of signals.

RK3588 SoM hardware design Guide

FET3588-C SoM has integrated power supply and storage circuit in a small module. The required external circuit is very simple. A minimal system only needs power supply and startup configuration to run, as shown in the figure below:

The minimum system includes SoM power supply, system flashing circuit, and debugging serial port circuit. The minimum system schematic diagram can be found in "OK3588-C_Hardware Manual". However, in general, it is recommended to connect some external devices, such as debugging serial port, otherwise user cannot judge whether system is started. After completing these, on this basis, add the functions required by user according to default interface definition of RK3588 SoM provided by Forlinx.

RK3588 Carrier Board Hardware Design Guide

The interface resources derived from Forlinx embedded OK3588-C development board are very rich, which provides great convenience for customers' development and testing. Moreover, OK3588-C development board has passed rigorous tests and can provide stable performance support for customers' high-end applications.

In order to facilitate user's secondary development, Forlinx provides RK3588 hardware design guidelines to annotate the problems that may be encountered during design process of RK3588. We want to help users make the research and development process simpler and more efficient, and make customers' products smarter and more stable. Due to the large amount of content, only a few guidelines for interface design are listed here. For details, you can contact us online to obtain "OK3588-C_Hardware Manual" (Click to Inquiry)

[pc7ooo]

Banana Pi BPI-CM5 Pro: представлена альтернатива Raspberry Pi CM4 с ИИ-процессором Rockchip

В ассортименте Banana Pi, по сообщению ресурса CNX Software, появился вычислительный модуль BPI-CM5 Pro, предназначенный для построения устройств с ИИ-функциями. Новинка, выполненная на аппаратной платформе Rockchip, представляет собой альтернативу Raspberry Pi CM4. Изделие имеет размеры 55×40 мм. Применён процессор RK3576, который содержит по четыре ядра Cortex-A72 (2,2 ГГц) и Cortex-A53 (1,8 ГГц), а также графический блок Arm Mali-G52 MC3 с поддержкой OpenGL ES 1.1/2.0/3.2, OpenCL 2.0 и Vulkan 1.1. Встроенный нейропроцессорный узел облает ИИ-производительностью до 6 TOPS (INT8) и поддержкой INT4/INT8/INT16/BF16/TF32. Объём оперативной памяти LPDDR5 может составлять 8 или 16 Гбайт.

Подробнее на https://7ooo.ru/group/2024/09/09/424-banana-pi-bpi-cm5-pro-predstavlena-alternativa-raspberry-pi-cm4-s-ii-processorom-rockchip-grss-339657823.html

[onedirectdeals]

Waveshare Luckfox Pico Max RV1106 Linux Micro Development Board, Integrates ARM Cortex-A7/RISC-V MCU/NPU/ISP Processors 256MB Memory

Price: Buy Now Last Updated: Single-core ARM Cortex-A7 32-bit core with integrated NEON and FPU. Integrated with built-in POR, audio codec and MAC PHYBuilt-in Rockchip self-developed 4th generation NPU, features high computing precision and supports int4, int8, and int16 hybrid quantization. The computing power of int8 is 0.5 TOPS, and up to 1.0 TOPS with int4Built-in self-developed…

[jcmarchi]

Accelerating Large Language Model Inference: Techniques for Efficient Deployment

New Post has been published on https://thedigitalinsider.com/accelerating-large-language-model-inference-techniques-for-efficient-deployment/

Accelerating Large Language Model Inference: Techniques for Efficient Deployment

Large language models (LLMs) like GPT-4, LLaMA, and PaLM are pushing the boundaries of what’s possible with natural language processing. However, deploying these massive models to production environments presents significant challenges in terms of computational requirements, memory usage, latency, and cost. As LLMs continue to grow larger and more capable, optimizing their inference performance is critical for real-world applications.

In this technical deep dive, we’ll explore cutting-edge techniques for accelerating LLM inference, enabling faster response times, higher throughput, and more efficient utilization of hardware resources. We’ll cover methods ranging from numerical precision techniques and novel attention mechanisms to architectural innovations tailored explicitly for efficient text generation.

Let’s start by understanding why LLM inference is so challenging compared to traditional NLP models.

The Inference Challenge with Large Language Models

Before the advent of LLMs, natural language processing relied on smaller models focused on specific tasks like text classification, named entity recognition, and sentiment analysis. While still computationally intensive, these models could be deployed on modest hardware and followed relatively straightforward inference processes.

LLMs, on the other hand, represent a paradigm shift. These models are trained on vast datasets using billions of parameters, enabling them to perform a wide range of language tasks with remarkable proficiency. However, this power comes at a cost – dramatically increased computational demands during both training and inference.

One key challenge is the autoregressive nature of text generation with LLMs. To produce human-like text, these models predict one token (word or subword) at a time, with each new token depending on the previously generated output. This sequential dependency prevents efficient parallelization and results in computational requirements that scale polynomially with sequence length.

Additionally, LLMs often require long input sequences (prompts) to establish the necessary context for high-quality text generation. Longer input lengths demand more memory to store intermediate states and attention matrices, further straining hardware resources.

With these unique challenges, traditional optimization techniques like quantization and static computation graphs can fall short, struggling to maintain LLM performance while delivering meaningful speedups. Let’s dive into some of the key strategies tailored explicitly for accelerating LLM inference.

Numerical Precision Techniques

From 32-Bit to 16-Bit Precision

One avenue for accelerating LLM inference is to leverage reduced numerical precision for model weights and activations. Modern deep learning frameworks like PyTorch and TensorFlow typically employ 32-bit floating-point (FP32) precision by default. However, research has shown that LLMs can often maintain high accuracy even when operating at lower precisions, such as 16-bit (FP16), 8-bit integers (INT8), or even 4-bit integers (INT4).

Reducing numerical precision offers several benefits:

Reduced Memory Footprint: Lower precision representations require less memory, allowing larger models or batch sizes to fit within the same hardware constraints.

Faster Computation: Many modern CPUs and GPUs provide specialized instructions and hardware acceleration for lower precision arithmetic, enabling significant speedups.

Improved Energy Efficiency: With smaller memory requirements and faster computations, lower precision inference can translate into reduced energy consumption – a crucial advantage for edge and mobile deployments.

While powerful, numerical precision techniques do introduce some accuracy loss compared to FP32 operation. The key is carefully evaluating this trade-off between computational gains and potential performance degradation for your specific use case.

There are two main approaches to quantization with LLMs:

Post-Training Quantization (PTQ): In this method, an LLM is first trained using standard FP32 precision. After training, the model weights are quantized (converted) to a lower precision format like INT8 or INT4. PTQ is straightforward to implement but can lead to greater accuracy drops.

Quantization-Aware Training (QAT): With QAT, the quantization process is simulated during the training phase itself. This allows the model to learn to compensate for quantization errors, minimizing accuracy degradation when the final quantized model is deployed. QAT is more involved but often yields better results compared to PTQ.

For practical application, one might leverage pre-quantized models available on platforms like Hugging Face, which hosts a variety of models optimized through different quantization methods. For instance, if a model quantized using Auto-GPTQ is desired, users can easily load it using Hugging Face’s transformers library. Additionally, to quantize a model, tools like AutoGPTQ can be utilized, which integrate seamlessly with existing libraries to compress the model efficiently.

Here is an example of loading a pre-quantized Llama-2-7b model using the Hugging Face transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "TheBloke/Llama-2-7b-Chat-GPTQ" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) And for custom quantization, one might follow these steps using the AutoGPTQ toolkit: from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig model_id = "llama-2-7b-original" tokenizer = AutoTokenizer.from_pretrained(model_id) quantization_config = GPTQConfig(bits=4, dataset="your-dataset", tokenizer=tokenizer) model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=quantization_config)

Remember that quantization might necessitate post-quantization fine-tuning or prompt engineering to maintain model quality. For new quantization, you can contribute back to the community by pushing your quantized models to platforms like Hugging Face.

Always ensure to balance between model size, computational requirements, and performance when selecting the quantization strategy for your specific use case.

The Flash Attention Algorithm

The multi-head attention mechanism is a core component of transformer-based LLMs, enabling the model to capture long-range dependencies and contextualized representations. However, this attention operation is computationally inefficient for autoregressive text generation, as it requires recomputing many of the same values for each new token.

The Flash Attention algorithm, introduced in the FlashAttention paper, provides a more memory-efficient and parallelization-friendly approach to the attention operation. Instead of recomputing attention values for each token, Flash Attention caches and reuses intermediate key/value matrices, avoiding redundant calculations.

This optimization not only reduces computational overhead but also improves memory access patterns, leading to better utilization of GPU memory bandwidth and parallelism.

While the details of Flash Attention are quite involved, the high-level idea is to decompose the attention operation into two phases:

Prefix Sum Embedding: This phase computes and caches key/value embeddings for all input tokens, enabling efficient reuse during generation.

Causal Attention: The actual attention operation, now optimized to leverage the cached key/value embeddings from the first phase.

By separating these phases, Flash Attention can take advantage of highly parallel GPU operations, significantly accelerating the attention bottleneck in LLM inference.

Here’s a brief, conceptual illustration of implementing Flash Attention with an LLM:

from transformers import AutoModelForCausalLM import torch from flash_attention import flash_attention # Load an LLM like OctoCoder model = AutoModelForCausalLM.from_pretrained("bigcode/octocoder") # Sample system prompt that guides the model towards being a better coding assistant system_prompt = """... (system prompt details) ...""" # Preparing a longer input with the system prompt long_prompt = system_prompt + "Question: Please write a function in Python that transforms bytes to Gigabytes." # Converting the model for Flash Attention optimization model.to_bettertransformer() # Running the model with Flash Attention start_time = time.time() with torch.backends.cuda.sdp_kernel(enable_flash=True): result = model.generate(long_prompt, max_new_tokens=60) print(f"Generated in time.time() - start_time seconds.")

While Flash Attention offers impressive performance gains, it works within the existing transformer architecture. To fully unleash the potential of accelerated LLM inference, we need to explore architectural innovations tailored specifically for this task.

Pruning LLMs

Pruning LLMs is a technique to reduce model size while maintaining functionality. It uses a data-dependent estimator for weight importance based on Hessian matrix approximations. In pruning, less important weight groups are removed, then the model is fine-tuned to recover accuracy. The LLM-Pruner package offers scripts for pruning with various strategies supported. Pruning includes discovering dependencies, estimating group contributions, and a recovery stage involving brief post-training.

Here’s a simplified Python code example demonstrating the use of LLM-Pruner for a LLaMa model:

from transformers import AutoModelForSequenceClassification from pruning import LLMPruner # Load pre-trained LLaMa model model = AutoModelForSequenceClassification.from_pretrained("llama-base") # Initialize the pruner with desired configuration pruner = LLMPruner( model, pruning_ratio=0.25, block_mlp_layers=(4, 30), block_attention_layers=(4, 30), pruner_type='taylor' ) # Execute pruning pruned_model = pruner.prune() # Fine-tune the pruned model pruned_model.fine_tune(training_data)

This code sketch represents loading a pre-trained LLaMa model, setting up the pruner with specific configurations (like which layers to prune and the type of pruner), executing the pruning process, and finally, fine-tuning the pruned model.

Note that for an actual implementation, you would need to fill in details like the specific model name, paths to the data, and additional parameters for the fine-tuning process. Also, be aware that this code is a conceptual representation, and actual syntax may vary depending on the library and versions used.

Architectural Innovations for Efficient Text Generation

The transformer architecture, while highly effective for language modeling tasks, was designed as a general-purpose sequence-to-sequence model. When deploying LLMs for text generation tasks with long input contexts, researchers have found that more specialized architectures can significantly improve inference efficiency without sacrificing quality.

Here are some of the key architectural innovations enabling faster LLM inference:

Alibi: The Alibi architecture, introduced in the PAL-Instruction paper, separates the modeling of long input context from the text generation process itself. It uses a compressed representation of the input context (the “alibi”) to initialize the generation process, avoiding the need to process the full input sequence repeatedly during autoregressive generation.

Rotary Embeddings: Instead of using standard positional embeddings, the rotary embedding technique employs rotation matrices to encode positional information more efficiently. This approach has been shown to improve performance and enable processing of longer input sequences.

Multi-Query Attention (MQA): In traditional attention, each output token attends to the entire input sequence, resulting in redundant computation. MQA reformulates the attention operation to share computations across multiple output tokens, reducing overall complexity.

Multiquery attention

Grouped-Query-Attention (GQA): Building upon MQA, GQA groups output tokens into clusters and computes attention jointly for each cluster. This approach further reduces computational requirements while maintaining high-quality text generation.

While still in active research and development, these architectural innovations have demonstrated impressive speedups for LLM inference tasks, especially when combined with techniques like Flash Attention and numerical precision optimization.

Real-World Deployment Considerations

Beyond the core algorithms and architectures, there are several practical considerations and trade-offs to navigate when deploying LLMs to production environments:

Hardware Acceleration: While CPUs can handle LLM inference, GPUs and other accelerators like Google’s TPUs are essential for achieving high throughput and low latency. Choosing the right hardware and optimizing memory usage is crucial.

Batching and Parallelism: To fully leverage hardware parallelism, strategies like batched inference (processing multiple inputs simultaneously) and model parallelism (distributing an LLM across multiple devices) can significantly boost throughput.

Quantization vs. Quality Trade-Off: The degree of quantization (8-bit, 4-bit, etc.) will directly impact inference speed and memory usage, but also affects output quality. This trade-off must be carefully evaluated for each use case.

Model Distillation: An alternative to quantization, model distillation techniques can compress large LLMs into smaller, more efficient student models while retaining high accuracy.

Caching and Optimized Runtimes: Optimized deep learning runtimes like NVIDIA’s TensorRT and frameworks designed for LLM serving (e.g., MosaicML’s Composable Inference Suite) can provide significant performance boosts through techniques like operator fusion, kernel optimization, and intelligent caching strategies.

The path to optimal LLM deployment often involves combining multiple techniques while carefully considering the specific requirements of your application, infrastructure constraints, and performance targets.

Conclusion

As large language models continue their rapid evolution, accelerating their inference performance is becoming increasingly crucial for enabling real-world applications and democratizing access to these powerful AI capabilities.

In this technical guide, we explored cutting-edge techniques spanning numerical precision optimization, novel attention algorithms like Flash Attention, and architectural innovations tailored for efficient text generation. While each approach offers its own advantages, the true power often lies in combining multiple strategies while navigating the intricate trade-offs between speed, memory usage, and output quality.

Looking ahead, we can expect continued research and development in this domain, fueled by the insatiable demand for more capable and accessible LLMs. From hardware acceleration and model compression to entirely new architectures, the quest for efficient LLM inference remains an exciting frontier in the world of natural language processing and artificial intelligence.

[wikimediauncommons]

file: Mcely, kostel sv. Václava int4.jpg

[govindhtech]

Intel Extension for Transformers & PyTorch LLM Optimisation

Enhancing deep learning model performance is essential for scalability and efficiency in the rapidly changing field of  artificial intelligence. Intel has been in the forefront of creating frameworks and tools to improve  AI models’ memory efficiency and speed of execution, especially with Intel Extension for PyTorch and Intel Extension for Transformers.

Comprehending the AI Stack

There are several layers in the  AI stack, and each is essential to optimizing LLMs. The hardware layer, which consists of Intel Xeon CPUs, Intel Data Centre GPUs, Intel Arc GPUs, and Intel Gaudi AI accelerators, is fundamental.

The acceleration libraries, such as Intel oneAPI Collective Communications Library (oneCCL) and Intel oneAPI Deep Neural Network Library (oneDNN), sit above this layer and offer optimized kernels with Intel optimized instruction sets for effective processing. The highest layer is made up of resource-efficient frameworks such as PyTorch that interface with the hardware and libraries underneath to optimize model performance.

Important Optimization Methods

Optimizing operators is essential to improving LLM performance. Using enhanced instruction sets such as Intel enhanced Vector Extensions (Intel AVX), Intel Advanced Matrix Extensions (Intel AMX), and Intel Xe Matrix Extensions (Intel XMX), Intel replaces the default operation kernels with highly-optimized Intel oneDNN kernels. The accuracy-flexible design of this optimization ensures that applications can operate at maximum speed and precision by supporting a variety of data types, from FP32 to INT4.

Graph optimizations reduce the amount of memory accesses needed during computation, which further enhances efficiency. For example, memory access times can be reduced by combining layers (e.g., Conv+ReLU+Sum) with bandwidth-limited operations (e.g., activation functions, ReLU, or Tanh).

This method works especially well for models such as ResNet-50, where a large amount of processing time is dedicated to bandwidth-constrained tasks. Specific fusion methods, including as linear post-ops fusion and multi-head attention fusion, are used in the context of LLMs with Intel Extension for PyTorch in JIT/Torch script mode to improve performance.

Memory management is essential for maximizing LLM performance because they frequently require large amounts of memory. By pre-filling key/value pairs before to the onset of autoregressive decoding and utilising pre-allocated buffers throughout the decoding stage, the Segment KV Cache approach maximizes memory use.

This technique increases efficiency by lowering the requirement for in-the-moment memory changes. Similar to this, the Indirect Access KV Cache efficiently manages memory by utilising beam index history and pre-allocated buffers, which lowers the overhead related to memory access during inference.

Model compression uses quantization algorithms, which successively decrease weight and activation precision from FP32 to lower precision forms like INT8 or INT4. This reduction minimizes the size of the model, increases inference speed, and lowers the required for memory bandwidth. Smooth Quant is a post-training quantization technique that shifts the quantization difficulty from activations to weights. This allows for the preservation of model accuracy while mitigating activation outliers and optimizing hardware utilization.

A big part of optimization is also played by custom operators. The goal of weight-only quantization is to increase input and output activation precision by quantizing the model’s weights alone. With minimal influence on accuracy, this technique maximizes computational performance by utilising weight-only quantization-optimized bespoke GEMM (General Matrix Multiply) kernels. Performance can be further optimized by using Explicit SIMD (ESIMD) extensions, which provide more precise control over hardware features.

Intel Extension for PyTorch

APIs for implementing these optimizations on CPU and GPU based training and inference are provided by the Intel Extension for PyTorch. You may make sure that your models are optimized to operate well on Intel hardware by making use of these APIs. To make it easier for developers to execute these optimizations, the extension comes with environment configurations and scripts that are intended to maximize hardware utilization.

Another essential element of Intel’s optimization approach are the Intel Gaudi  AI accelerators. Deep learning applications perform better because to the integration of PyTorch with the Intel Gaudi software suite, which effectively transfers neural network topologies onto Gaudi hardware. This integration also supports important kernel libraries and optimizations.

Intel Extension for Transformers

https://community.intel.com/t5/image/serverpage/image-id/56748i59BB048F0E369A11/image-size/large?v=v2&px=999&whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright

Several plugins for widely used pipelines, like audio processing and retrieval-augmented generation (RAG), can be integrated with Neural Chat. By integrating the required optimizations straight into the pipeline setup, it makes the deployment of optimized chatbots easier.

Neural Velocity and Dispersed Interpretation

https://community.intel.com/t5/image/serverpage/image-id/56751iE6BB93D0A520220B/image-size/large?v=v2&px=999&whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright

DeepSpeed

These optimizations are further expanded across numerous nodes or GPUs via Intel’s support for distributed inference via DeepSpeed. DeepSpeed now supports Intel GPUs thanks to the Intel Extension for DeepSpeed. It includes the following parts:

Implementation of the DeepSpeed Accelerator Interface

Implementation of DeepSpeed op builder for XPU

Code for DeepSpeed op builder kernel

With the help of oneCCL, this Intel-optimized extension distributes compute jobs well, lowering memory footprint and increasing throughput overall. Scaling  AI applications across heterogeneous computer systems requires this capacity.

Utilising Optimizations in Real-World Applications

It’s actually very easy to implement these optimizations using Intel’s tools, as you can use the extensions for the PyTorch and Transformers frameworks. For example, Intel Extension for Transformers improves model compression methods such as weight-only and smooth quantization right inside the well-known Transformers API. By setting the quantization parameters and using the integrated APIs, you may optimize models with ease.

In a similar vein, the Intel Extension for Transformers and PyTorch offers an adaptable framework for optimizing deep learning models other than LLMs. This update provides GPU-centric capabilities like tensor parallelism and  CPU optimizations like NUMA management and graph optimization’s to enable fine-tuning and deployment across a variety of hardware configurations.

In summary

You may significantly increase the effectiveness and performance of your AI models by utilising Intel’s extensive hardware stack, accelerated libraries, and optimized frameworks. These optimizations cut the energy and operating expenses associated with running large-scale  AI applications in addition to improving computational performance and reducing latency.

Using the getting started samples from the Intel Extension for PyTorch and Intel Extension for Transformers, you can investigate these optimizations on the Intel Tiber Developer Cloud. You can make sure your LLMs are operating at optimal performance on Intel hardware by incorporating these strategies.

Read more on govindhtech.com

[newslobster]

Qualcomm Snapdragon 8 Gen 2 SoC With Wi-Fi 7, Ray Tracing Launched: Details

Qualcomm Snapdragon 8 Gen 2 SoC With Wi-Fi 7, Ray Tracing Launched: Details

Qualcomm unveiled Snapdragon 8 Gen 2 SoC at its annual Snapdragon Tech Summit on Wednesday. The new mobile 5G platform brings a list of upgrades over last year’s Snapdragon 8 Gen 1 SoC and is claimed to be 40 percent more power efficient than the older model. It offers real-time raytracing for gaming and supports INT4 and Wi-Fi 7. The Snapdragon 8 Gen 2 supports new image sensors like the…

[scottfromappdesign]

#4

21 y/o - he/him

1. How often do you listen to music?

Everyday. Funny enough, I don’t really know anyone who doesn’t have the same answer as me. Music is a really big part of my life and it’s been that way for years.

2. Do you use any music streaming apps? 

Just Apple Music. I’m not a big fan of the other music streaming apps.

3. Do you like to discover new music? (i.e. artists, playlists, genres) If so, what are your methods for finding new music?

No I don’t. I have a hard time discovering new music/artists so I just gave up on doing discovery on my own.

4. Some people have friends that connect over a lot of different topics and some have friends that connect on very few topics. When it comes to music, do you find it easy or hard to connect with your friends over it, and why? 

I find it easy because it’s definitely something that you can have an entire conversation about with a person when a new song or album drops, especially how it makes each other feel. Plus it helps that my friends 9 times out of 10 have the same music taste as me. 

5. Have you ever or do you currently use social media to make new friends and talk about interests you have in common? 

Yes but to an extent.

6. If you answered yes to the previous question, can you share one or more of your experiences? If you answered no to the previous question, can you please elaborate on why you don’t use social media to meet new people?

I don’t really make friends with complete strangers on social media but I will make connections if you’re a friend of a friend or met you very briefly somewhere.

7. Do you find that like counts/follower counts/leaderboards discourage you from using certain apps and/or making connections with people online or do you feel the opposite and why?

Yes and no. It does get intimidating since social media has become such a big part of daily life and “social status” but I’m trying to train myself to think that none of that matters as much as you think it is. I find my self doing social media cleanses often and it has been helping me a lot.

8. On a scale of 1-10 how likely would you be to use an app that allows you that connects to your music streaming apps, shows you randomized playlists (based on your preferences) in order to discover new music, and connect with people who have similar music tastes?

I think a 7. I think there’s a lot of potential in an app like this and if it can be fun and interesting then I would most likely give it a try.

9. With the app idea presented in the previous question, do you have any concerns about the app or features you would implemented in the app?

Having a section or a space for people to share or add  “feelings/mood” so curated playlists for people feeling any specific way could be a really fun feature to the app instead of just generic music taste categories. These sections/playlists should be highly specific like “they didn’t have my drink in Starbucks now I have no will to live” or “my boyfriend just lied to me for the third time and I need a song/playlist to motivate me to break up with him”

[lazotoys]

INT-4 1981 STAR WARS KENNER MADE IN MACAO THE EMPIRE STRIKES BACK MINI-RIG Mira mis otras figuras vintage y modernas. 22 EUROS Se vende el juguete de la foto. Envio gratuito en España (península) Islas Canarias, Ceuta y Melilla, Baleares, consultar. Mira mis otros artículos Envio certificado. Pago por transferencia bancaria, Bizum o PayPal como amigo. Cualquier duda consultar. #starwars #int4 #esb #starwarsvariations #vintage #vintagetoys #lazotoys #lazotoysstarwars #seconhandstarwars #starwarstoys #starwarsforsale https://www.instagram.com/p/CLjIMOvADg4/?igshid=1a1acf9kaxdi0

[kerink]

wtse act 2 is gonna end up looking like homsetuck act 6 with all the shit benny and i keep cramming in there

[forlinx]

Four Advantages Detailed Analysis of Forlinx Embedded FET3576-C System on Module

In order to fully meet the growing demand in the AIoT market for high-performance, high-computing-power, and low-power main controllers, Forlinx Embedded has recently launched the FET3576-C System on Module, designed based on the Rockchip RK3576 processor. It features excellent image and video processing capabilities, a rich array of interfaces and expansion options, low power consumption, and a wide range of application scenarios. This article delves into the distinctive benefits of the Forlinx Embedded FET3576-C SoM from four key aspects.

Advantages: 6TOPS computing power NPU, enabling AI applications

Forlinx Embedded FET3576-C SoM has a built-in 6TOPS super arithmetic NPU with excellent deep learning processing capability. It supports INT4/ INT8/ INT16/ FP16/ BF16/ TF32 operation. It supports dual-core working together or independently so that it can flexibly allocate computational resources according to the needs when dealing with complex deep learning tasks. It can also maintain high efficiency and stability when dealing with multiple deep-learning tasks.

FET3576-C SoM also supports TensorFlow, Caffe, Tflite, Pytorch, Onnx NN, Android NN and other deep learning frameworks. Developers can easily deploy existing deep learning models to the SoM and conduct rapid development and optimization. This broad compatibility not only lowers the development threshold, but also accelerates the promotion and adoption of deep learning applications.

Advantages: Firewall achieves true hardware resource isolation

The FET3576-C SoM with RK3576 processor supports RK Firewall technology, ensuring hardware resource isolation for access management between host devices, peripherals, and memory areas.

Access Control Policy - RK Firewall allows configuring policies to control which devices or system components access hardware resources. It includes IP address filtering, port control, and specific application access permissions. Combined with the AMP system, it efficiently manages access policies for diverse systems.

Hardware Resource Mapping and Monitoring - RK Firewall maps the hardware resources in the system, including memory areas, I/O devices, and network interfaces. By monitoring access to these resources, RK Firewall can track in real-time which devices or components are attempting to access specific resources.

Access Control Decision - When a device or component attempts to access hardware resources, RK Firewall will evaluate the access against predefined access control policies. If the access request complies with the policy requirements, access will be granted; otherwise, it will be denied.

Isolation Enforcement - For hardware resources identified as requiring isolation, RK Firewall will implement isolation measures to ensure that they can only be accessed by authorized devices or components.

In summary, RK Firewall achieves effective isolation and management of hardware resources by setting access control policies, monitoring hardware resource access, performing permission checks, and implementing isolation measures. These measures not only enhance system security but also ensure system stability and reliability.

Advantages: Ultra clear display + AI intelligent repair

With its powerful multimedia processing capability, FET3576-C SoM provides users with excellent visual experience. It supports H.264/H.265 codecs for smooth HD video playback in various scenarios, while offering five display interfaces (HDMI/eDP, MIPI DSI, Parallel, EBC, DP) to ensure compatibility with diverse devices.

FET3576-C SoM notably supports triple-screen display functionality, enabling simultaneous display of different content on three screens, significantly enhancing multitasking efficiency.

In addition, its 4K @ 120Hz ultra-clear display and super-resolution function not only brings excellent picture quality enjoyment, but also intelligently repairs blurred images, improves video frame rate, and brings users a clearer and smoother visual experience.

Advantage: FlexBus new parallel bus interface

FET3576-C of Forlinx Embedded offers a wide range of connectivity and transmission options with its excellent interface design and flexible parallel bus technology. The FlexBus interface on the SoM is particularly noteworthy due to its high flexibility and scalability, allowing it to emulate irregular or standard protocols to accommodate a variety of complex communication needs.

FlexBus supports parallel transmission of 2/4/8/16bits of data, enabling a significant increase in the data transfer rate, while the clock frequency of up to 100MHz further ensures the high efficiency and stability of data transmission.

In addition to the FlexBus interface, the FET3576-C SoM integrates a variety of bus transfer interfaces, including DSMC, CAN-FD, PCIe2.1, SATA3.0, USB3.2, SAI, I2C, I3C and UART. These interfaces not only enriches the SoM's application scenarios but also enhances its compatibility with other devices and systems.

It is easy to see that with the excellent advantages of high computing power NPU, RK Firewall, powerful multimedia processing capability and FlexBus interface, Forlinx Embedded FET3576-C SoM will become a strong player in the field of embedded hardware. Whether you are developing edge AI applications or in pursuit of high-performance, high-quality hardware devices, the Folinx Embedded FET3576-C SoM is an unmissable choice for you.

Originally published at www.forlinx.net.