CTransformers is a python binding for some popular transformer models implemented in C/C++ using the GGML library. CTransformers provides support across CUDA, CPU and Metal. This library also provides very similar huggingface-transformers like interface, which makes it easier to use and integrate in various applications.
We can run the CTransformers benchmark for two models: Llama2-chat and Mistral-7B v0.1-instruct Here is how we run benchmark for CTransformers.
./bench_ctransformers/bench.sh \
--prompt <value> \ # Enter a prompt string
--max_tokens <value> \ # Maximum number of tokens to output
--repetitions <value> \ # Number of repititions to be made for the prompt.
--device <cpu/cuda/metal> \ # The device in which we want to benchmark.
--model_name <name-of-the-model> # The name of the model. (options: 'llama' for Llama2 and 'mistral' for Mistral-7B-v0.1)To get started quickly you can simply run:
./bench_ctransformers/bench.sh -d cuda -n llamaThis will take all the default values (see in the bench.sh file) and do the benchmarks for Llama 2 and Mistral models. You can find all the benchmarks results for Llama 2 7B Chat here, and Mistral 7B instruct here.
Please note, we consider the outputs from the raw (float32 precision) from HuggingFace Transformers (PyTorch backend) weights as our ground truth. We will be comparing this ground truth with model's output with int4 of AutoAWQ. This is our system prompt.
You answers should always be to the point, precise and not more than 2 sentences strictly
| Question | int4 | int8 | Ground Truth |
|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason for the terrible pancakes could be that the batter was too thick, resulting in dense and dry pancakes. | The main reason for the terrible pancakes could be that the batter was too thick, resulting in dense and dry pancakes. | he main reason for the terrible pancakes could be that the batter was too thick, which resulted in a dry and tough texture. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | One bird is left on the branch. | One bird is left on the branch. | One bird is left on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are two grammatical errors in the sentence. | There are two grammatical errors in the sentence. | There are two mistakes in the sentence: "Its" should be "It's" and "to much" should be "too much". |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Excessive use of chemical pesticides can lead to long-term health problems such as cancer and neurological disorders. | Excessive use of chemical pesticides can lead to long-term health problems such as cancer and neurological disorders. | Excessive use of chemical pesticides can lead to long-term negative health effects, including cancer, neurological damage, and reproductive issues. |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann proposed using a spherical configuration of bombs to increase their destructive potential. | John von Neumann proposed using a spherical configuration of bombs to create a "super-bomb" that would maximize the destructive force of the explosion. | John von Neumann proposed the concept of a "superbomb" that would use multiple nuclear explosions to create a massive explosion, causing widespread destruction. |
| Question | int4 | int8 | Ground Truth |
|---|---|---|---|
| I'm making pancakes for breakfast. I added a cup of flour, a teaspoon of salt, and a few tablespoons of sugar to a bowl. I stirred it together, then added a cup of milk, a beaten egg, and a few tablespoons of oil, and stirred until just mixed. Then I put 1/4 a cup on a hot frying pan, and flipped it when brown. But they're terrible! Why? List the main reason. Answer as much precise as possible with one sentence. | The main reason your pancakes are terrible is because you did not mix the batter well enough, resulting in a dense and uneven texture. | The main reason your pancakes are turning out terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. | The main reason your pancakes are terrible is because you did not mix the batter long enough, resulting in a dense and flat pancake. |
| 42 birds are sitting on a tree branch. A hunter passes, shoots one dead, and misses two. How many birds are left on the branch? Answer as much precise as possible with one sentence. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. | After the hunter shoots and kills one bird, 41 birds remain on the branch. |
| How many mistakes can you find in the following sentence? Sentence: Its to much to loose if your talking about hundred’s of dollars. Answer as much precise as possible with one sentence. | There are two mistakes in the sentence: "loose" should be spelled "lose," and "hundred's" should be spelled "hundred." | There are two mistakes in the sentence: "loose" should be spelled "lose," and "hundred's" is incorrect, it should be "hundreds." | There are 2 mistakes in the sentence: "to loose" should be "to lose" and "hundred's" should be "hundreds". |
| You are an expert on agricultural chemical compounds. Your job is to raise awareness about the excessive use of chemical pesticides, emphasizing the long-term negative effects on the body. Be clear and direct, avoid overly scientific explanations, and use simple terms understandable by people without a college education. Answer as much precise as possible with one sentence. | Sure, I'd be happy to help raise awareness about the excessive use of chemical pesticides! Here's a key point to keep in mind: exposure to these chemicals can lead to serious health problems over time, including cancer, hormone disruption, and neurological damage. It's important to consider alternative, safer methods for controlling pests and protecting crops. | Sure, I'd be happy to help raise awareness about the excessive use of chemical pesticides! Here are some key points to consider: * Excessive exposure to chemical pesticides can lead to long-term negative effects on the body, including cancer, hormone disruption, and neurological problems. (1 sentence) * The use of chemical pesticides in agriculture has increased significantly over the past few decades, despite the availability of safer alternatives. (1 sentence) |
Excessive use of chemical pesticides can have long-term negative effects on the body, including cancer, hormone disruption, and neurological problems, due to their toxic and persistent nature, which can accumulate in the body over time and cause irreversible damage. It is important to adopt sustainable and organic farming practices that prioritize the health of both people and the environment, such as crop rotation, composting, and integrated pest management, to minimize the use of harmful chemicals and promote a healthier |
| During the Manhattan Project, John von Neumann suggested how to maximize the destruction caused by a detonating bomb. Which was his idea? Answer as much precise as possible with one sentence. | John von Neumann proposed using a multi-stage thermonuclear weapon design, known as the "boosted fission" or "thermonuclear" concept, which would increase the explosive yield by using a secondary fusion reaction to amplify the energy released in the initial fission reaction. This idea was later adopted and developed into the Teller-Ulam design, which became the basis for modern thermonuclear weapons. | John von Neumann proposed using a multi-stage thermonuclear weapon design, known as the "Teller-Ulam design," which would achieve maximum destruction through a chain reaction of nuclear fission and fusion reactions. This design was later used in the development of the hydrogen bomb. | John von Neumann proposed the concept of a "fusion bomb," which would achieve maximum destruction through the simultaneous detonation of a fission bomb and a fusion reaction, creating a chain reaction of nuclear reactions that would release an enormous amount of energy. This idea was later developed into the hydrogen bomb. |
- Since, ctransformers only supports quantized model. So it does not have benchmarking for float32/16.
- You can not specify GPU device, rather you have to specify the number of GPU Layers.
- The device support for CTransformer is model specific. For example, Llama 2 7B chat, Mistral v0.1 instruct supports CUDA/Metal/CPU, but models like Falcon or MPT only runs on CUDA. You can check out the model support here.