From 0d39f7b5b66c33ea5eaa88bb469d0ce382bb24eb Mon Sep 17 00:00:00 2001 From: Brian Aker Date: Tue, 5 Jan 2010 14:55:54 -0800 Subject: [PATCH] Update on memslap docs. --- docs/memslap.pod | 880 ++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 878 insertions(+), 2 deletions(-) diff --git a/docs/memslap.pod b/docs/memslap.pod index 9afad31f..23220d30 100644 --- a/docs/memslap.pod +++ b/docs/memslap.pod @@ -9,11 +9,885 @@ memslap - Load testing and benchmarking tool for memcached =head1 DESCRIPTION B is a load generation and benchmark tool for memcached(1) -servers. It simulates loads on memcached server clusters. +servers. It generates configurable workload such as threads, concurrencies, connections, +run time, overwrite, miss rate, key size, value size, get/set proportion, +expected throughput, and so on. Furthermore, it also supports data +verification, expire-time verification, UDP, binary protocol, facebook test, +replication test, multi-get and reconnection, etc. + +Memslap manages network connections like memcached with +libevent. Each thread of memslap is bound with a CPU core, all +the threads don't communicate with each other, and there are several socket +connections in each thread. Each connection keeps key size distribution, +value size distribution, and command distribution by itself. You can specify servers via the B<--servers> option or via the environment variable C. + +=head1 FEATURES + +Memslap is developed to for the following purposes: + +=over + +=item Manages network connections with libevent asynchronously. + +=item Set both TCP and UDP up to use non-blocking IO. + +=item Improves parallelism: higher performance in multi-threads environments. + +=item Improves time efficiency: faster processing speed. + +=item Generates key and value more efficiently; key size distribution and value size distribution are configurable. + +=item Supports get, multi-get, and set commands; command distribution is configurable. + +=item Supports controllable miss rate and overwrite rate. + +=item Supports data and expire-time verification. + +=item Supports dumping statistic information periodically. + +=item Supports thousands of TCP connections. + +=item Supports binary protocol. + +=item Supports facebook test (set with TCP and multi-get with UDP) and replication test. + +=back + +=head1 DETAILS + +=head2 Effective implementation of network. + +For memslap, both TCP and UDP use non-blocking network IO. All +the network events are managed by libevent as memcached. The network module +of memslap is similar to memcached. Libevent can ensure +memslap can handle network very efficiently. + +=head2 Effective implementation of multi-threads and concurrency + +Memslap has the similar implementation of multi-threads to +memcached. Memslap creates one or more self-governed threads; +each thread is bound with one CPU core if the system supports setting CPU +core affinity. + +In addition, each thread has a libevent to manage the events of the network; +each thread has one or more self-governed concurrencies; and each +concurrency has one or more socket connections. All the concurrencies don’t +communicate with each other even though they are in the same thread. + +Memslap can create thousands of socket connections, and each +concurrency has tens of socket connections. Each concurrency randomly or +sequentially selects one socket connection from its socket connection pool +to run, so memslap can ensure each concurrency handles one +socket connection at any given time. Users can specify the number of +concurrency and socket connections of each concurrency according to their +expected workload. + +=head2 Effective implementation of generating key and value + +In order to improve time efficiency and space efficiency, asynchronous +memslap creates a random characters table with 10M characters. All the +suffixes of keys and values are generated from this random characters table. + +Memslap uses the offset in the character table and the length +of the string to identify a string. It can save much memory. +Each key contains two parts, a prefix and a suffix. The prefix is an +uint64_t, 8 bytes. In order to verify the data set before, asynchronous +memslap need to ensure each key is unique, so it uses the prefix to identify +a key. The prefix cannot include illegal characters, such as ‘\r’, ‘\n’, +‘\0’ and ‘ ‘. And memslap has an algorithm to ensure that. + +Memslap doesn’t generate all the objects (key-value pairs) at +the beginning. It only generates enough objects to fill the task window +(default 10K objects) of each concurrency. Each object has the following +basic information, key prefix, key suffix offset in the character table, key +length, value offset in the character table, and value length. + +In the work process, each concurrency sequentially or randomly selects an +object from the window to do set operation or get operation. At the same +time, each concurrency kicks objects out of its window and adds new object +into it. + +=head2 Simple but useful task scheduling + +Memslap uses libevent to schedule all the concurrencies of +threads, and each concurrency schedules tasks based on the local task +window. Memslap assumes that if each concurrency keeps the same +key distribution, value distribution and commands distribution, from +outside, memslap keeps all the distribution as a whole. +Each task window includes a lot of objects, each object stores its basic +information, such as key, value, expire time, and so on. At any time, all +the objects in the window keep the same and fixed key and value +distribution. If an object is overwritten, the value of the object will be +updated. Memslap verifies the data or expire-time according to +the object information stored in the task window. + +Libevent selects which concurrency to handle based on a specific network +event. Then the concurrency selects which command (get or set) to operate +based on the command distribution. If it needs to kick out an old object and +add a new object, in order to keep the same key and value distribution, the +new object must have the same key length and value length. + +If memcached server has two cache layers (memory and SSD), running +memslap with different window sizes can get different cache +miss rates. If memslap adds enough objects into the windows at +the beginning, and the cache of memcached cannot store all the objects +initialized, then memslap will get some objects from the second +cache layer. It causes the first cache layer to miss. So the user can +specify the window size to get the expected miss rate of the first cache +layer. + +=head2 Useful implementation of multi-servers , UDP, TCP, multi-get and binary protocol + +Because each thread is self-governed, memslap can assign +different threads to handle different memcached servers. This is just one of +the ways in which memslap supports multiple servers. The only +limitation is that the number of servers cannot be greater than the number +of threads. The other way to support multiple servers is for replication +test. Each concurrency has one socket connection to each memcached server. +For the implementation, memslap can set some objects to one +memcached server, and get these objects from the other servers. + +By default, Memslap does single get. If the user specifies +multi-get option, memslap will collect enough get commands and +pack and send the commands together. + +Memslap supports both the ASCII protocol and binary protocol, +but it runs on the ASCII protocol by default. +Memslap by default runs on the TCP protocol, but it also +supports UDP. Because UDP is unreliable, dropped packages and out-of-order +packages may occur. Memslap creates a memory buffer to handle +these problems. Memslap tries to read all the response data of +one command from the server and reorders the response data. If some packages +get lost, the waiting timeout mechanism can ensure half-baked packages will +be discarded and the next command will be sent. + + +=head1 USAGE + +Below are some usage samples: + +=over 4 + +=item memslap -s 127.0.0.1:11211 -S 5s + +=item memslap -s 127.0.0.1:11211 -t 2m -v 0.2 -e 0.05 -b + +=item memslap -s 127.0.0.1:11211 -F config -t 2m -w 40k -S 20s -o 0.2 + +=item memslap -s 127.0.0.1:11211 -F config -t 2m -T 4 -c 128 -d 20 -P 40k + +=item memslap -s 127.0.0.1:11211 -F config -t 2m -d 50 -a -n 40 + +=item memslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m + +=item memslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m -p 2 + +=back + +The user must specify one server at least to run asynchronous memslap. The +rest of the parameters have default values, as shown below: + +Thread number = 1 Concurrency = 16 + +Run time = 600 seconds Configuration file = NULL + +Key size = 64 Value size = 1024 + +Get/set = 9:1 Window size = 10k + +Execute number = 0 Single get = true + +Multi-get = false Number of sockets of each concurrency = 1 + +Reconnect = false Data verification = false + +Expire-time verification = false ASCII protocol = true + +Binary protocol = false Dumping statistic information + +periodically = false + +Overwrite proportion = 0% UDP = false + +TCP = true Limit throughput = false + +Facebook test = false Replication test = false + +=head2 Key size, value size and command distribution. + +All the distributions are read from the configuration file specified by user +with “—cfg_cmd” option. If the user does not specify a configuration file, +asynchronous memslap will run with the default distribution (key size = 64, +value size = 1024, get/set = 9:1). For information on how to edit the +configuration file, refer to the “Configuration File” section. + +The minimum key size is 16 bytes; the maximum key size is 250 bytes. The +precision of proportion is 0.001. The proportion of distribution will be +rounded to 3 decimal places. + +The minimum value size is 1 bytes; the maximum value size is 1M bytes. The +precision of proportion is 0.001. The proportion of distribution will be +rounded to 3 decimal places. +Currently, asynchronous memslap only supports set and get commands. And it +supports 100% set and 100% get. For 100% get, it will preset some objects to +the server. + +=head2 Multi-thread and concurrency + +The high performance of asynchronous memslap benefits from the special +schedule of thread and concurrency. It’s important to specify the proper +number of them. The default number of threads is 1; the default number of +concurrency is 16. The user can use “—threads” and “--concurrency” to +specify these variables. + +If the system supports setting CPU affinity and the number of threads +specified by the user is greater than 1, asynchronous memslap will try to +bind each thread to a different CPU core. So if you want to get the best +performance asynchronous memslap, it is better to specify the number of +thread equal to the number of CPU cores. The number of threads specified by +the user can also be less or greater than the number of CPU cores. Because +of the limitation of implementation, the number of concurrencies could be +the multiple of the number of threads. + +1. For 8 CPU cores system + +For example: + +--threads=2 --concurrency=128 + +--threads=8 --concurrency=128 + +--threads=8 --concurrency=256 + +--threads=12 --concurrency=144 + +2. For 16 CPU cores system + +For example: + +--threads=8 --concurrency=128 + +--threads=16 --concurrency=256 + +--threads=16 --concurrency=512 + +--threads=24 --concurrency=288 + +The asynchronous memslap performs very well, when +used to test the performance of memcached servers. +Most of the time, the bottleneck is the network or +the server. If for some reason the user wants to +limit the performance of asynchronous memslap, there +are two ways to do this: + +Decrease the number of threads and concurrencies. +Use the option “--tps” that Asynchronous memslap +provides to limit the throughput. This option allows +the user to get the expected throughput. For +example, assume that the maximum throughput is 50 +kops/s for a specific configuration, you can specify +the throughput equal to or less than the maximum +throughput using “--tps” option. + +=head2 Window size + +Most of the time, the user does not need to specify the window size. The +default window size is 10k. For Schooner Memcached, the user can specify +different window sizes to get different cache miss rates based on the test +case. Asynchronous memslap supports cache miss rate between 0% and 100%. +If you use this utility to test the performance of Schooner Memcached, you +can specify a proper window size to get the expected cache miss rate. The +formula for calculating window size is as follows: + +Assume that the key size is 128 bytes, and the value size is 2048 bytes, and +concurrency=128. + +1. Small cache cache_size=1M, 100% cache miss (all data get from SSD). +win_size=10k + +2. cache_size=4G + +(1). cache miss rate 0% + +win_size=8k + +(2). cache miss rate 5% + +win_size=11k + +3. cache_size=16G + +(1). cache miss rate 0% + +win_size=32k + +(2). cache miss + +rate 5% + +win_size=46k + +The formula for calculating window size for cache miss rate 0%: + +cache_size / concurrency / (key_size + value_size) * 0.5 + +The formula for calculating window size for cache miss rate 5%: + +cache_size / concurrency / (key_size + value_size) * 0.7 + +=head2 Verification + +Asynchronous memslap supports both data verification and expire-time +verification. The user can use "--verify=" or "-v" to specify the proportion +of data verification. In theory, it supports 100% data verification. The +user can use "--exp_verify=" or "-e" to specify the proportion of +expire-time verification. In theory, it supports 100% expire-time +verification. Specify the "--verbose" options to get more detailed error +information. + +For example: --exp_verify=0.01 –verify=0.1 , it means that 1% of the objects +set with expire-time, 10% of the objects gotten will be verified. If the +objects are gotten, asynchronous memslap will verify the expire-time and +value. + +=head2 multi-servers and multi-clients + +Asynchronous memslap supports multi-servers based on self-governed thread. +There is a limitation that the number of servers cannot be greater than the +number of threads. Asynchronous memslap assigns one thread to handle one +server at least. The user can use the "--servers=" or "-s" option to specify +multi-servers. + +For example: + +--servers=10.1.1.1:11211,10.1.1.2:11212,10.1.1.3:11213 --threads=6 --concurrency=36 + +The above command means that there are 6 threads, with each thread having 6 +concurrencies and that threads 0 and 3 handle server 0 (10.1.1.1); threads 1 +and 4 handle server 1 (10.1.1.2); and thread 2 and 5 handle server 2 +(10.1.1.3). + +All the threads and concurrencies in asynchronous memslap are self-governed. + +So is asynchronous memslap. The user can start up several asynchronous +memslap instances. The user can run asynchronous memslap on different client +machines to communicate with the same memcached server at the same. It is +recommended that the user start different asynchronous memslap on different +machines using the same configuration. + +=head2 Run with execute number mode or time mode + +The default asynchronous memslap runs with time mode. The default run time +is 10 minutes. If it times out, asynchronous memslap will exit. Do not +specify both execute number mode and time mode at the same time; just +specify one instead. + +For example: + +--time=30s (It means the test will run 30 seconds.) + +--execute_number=100000 (It means that after running 100000 commands, the test will exit.) + +=head2 Dump statistic information periodically. + +The user can use "--stat_freq=" or "-S" to specify the frequency. + +For example: + +--stat_freq=20s + +asynchronous memslap will dump the statistics of the commands (get and set) at the frequency of every 20 +seconds. + +For more information on the format of dumping statistic information, refer to “Format of Output” section. + +=head2 Multi-get + +The user can use "--division=" or "-d" to specify multi-get keys count. +Asynchronous memslap by default does single get with TCP. Asynchronous +memslap also supports data verification and expire-time verification for +multi-get. + +Asynchronous memslap supports multi-get with both TCP and UDP. Because of +the different implementation of the ASCII protocol and binary protocol, +there are some differences between the two. For the ASCII protocol, +asynchronous memslap sends one “multi-get” to the server once. For the +binary protocol, asynchronous memslap sends several single get commands +together as “multi-get” to the server. + +=head2 UDP and TCP + +Asynchronous memslap supports both UDP and TCP. For TCP, asynchronous +memslap does not reconnect the memcached server if socket connections are +lost. If all the socket connections are lost or memcached server crashes, +asynchronous memslap will exit. If the user specifies the “--reconnect” +option when socket connections are lost, it will reconnect them. + +User can use “--udp” to enable the UDP feature, but UDP comes with some +limitations: + +UDP cannot set data more than 1400 bytes. + +UDP is not supported by the binary protocol because the binary protocol of +memcached does not support that. + +UDP doesn’t support reconnection. + +=head2 Facebook test + +Set data with TCP and multi-get with UDP. Specify the following options: + +"--facebook --division=50" + +If you want to create thousands of TCP connections, specify the + +"--conn_sock=" option. + +For example: --facebook --division=50 --conn_sock=200 + +The above command means that asynchronous memslap will do facebook test, +each concurrency has 200 socket TCP connections and one UDP socket. + +Asynchronous memslap sets objects with the TCP socket, and multi-gets 50 +objects once with the UDP socket. + +If you specify "--division=50", the key size must be less that 25 bytes +because the UDP packet size is 1400 bytes. + +=head2 Replication test + +For replication test, the user must specify at least two memcached servers. +The user can use “—rep_write=” option to enable feature. + +For example: + +--servers=10.1.1.1:11211,10.1.1.2:11212 –rep_write=2 + +The above command means that there are 2 replication memcached servers, +asynchronous memslap will set objects to both server 0 and server 1, get +objects which are set to server 0 before from server 1, and also get objects +which are set to server 1 before from server 0. If server 0 crashes, +asynchronous memslap will only get objects from server 1. If server 0 comes +back to life again, asynchronous memslap will reconnect server 0. If both +server 0 and server 1 crash, asynchronous memslap will exit. + +=head2 Supports thousands of TCP connections + +Start asynchronous memslap with "--conn_sock=" or "-n" to enable this +feature. Make sure that your system can support opening thousands of files +and creating thousands of sockets. However, this feature does not support +reconnection if sockets disconnect. + +For example: + +--threads=8 --concurrency=128 --conn_sock=128 + +The above command means that asynchronous memslap starts up 8 threads, each +thread has 16 concurrencies, each concurrency has 128 TCP socket +connections, and the total number of TCP socket connections is 128 * 128 = +16384. + +=head2 Supports binary protocol + +Start asynchronous memslap with "--binary" or "-B" options to enable this +feature. It supports all the above features except UDP, because the latest +memcached 1.3.3 does not implement binary UDP protocol. + +For example: + +--binary + +Since memcached 1.3.3 doesn't implement binary UDP protocol, asynchronous +memslap does not support UDP. In addition, memcached 1.3.3 does not support +multi-get. If you specify "--division=50" option, it just sends 50 get +commands together as “mulit-get” to the server. + +=head1 Configuration file + +This section describes the format of the configuration file. Below is a +sample configuration file: + + *************************************************************************** + #comments should start with '#' + #key + #start_len end_len proportion + # + #key length range from start_len to end_len + #start_len must be equal to or greater than 16 + #end_len must be equal to or less than 250 + #start_len must be equal to or greater than end_len + #asynchronous memslap will generate keys according to the key range + #proportion: indicates keys generated from one range accounts for the total + generated keys + # + #example1: key range 16~100 accounts for 80% + # key range 101~200 accounts for 10% + # key range 201~250 accounts for 10% + # total should be 1 (0.8+0.1+0.1 = 1) + # + # 16 100 0.8 + # 101 200 0.1 + # 201 249 0.1 + # + #example2: all keys length are 128 bytes + # + # 128 128 1 + key + 128 128 1 + #value + #start_len end_len proportion + # + #value length range from start_len to end_len + #start_len must be equal to or greater than 1 + #end_len must be equal to or less than 1M + #start_len must be equal to or greater than end_len + #asynchronous memslap will generate values according to the value range + #proportion: indicates values generated from one range accounts for the + total generated values + # + #example1: value range 1~1000 accounts for 80% + # value range 1001~10000 accounts for 10% + # value range 10001~100000 accounts for 10% + # total should be 1 (0.8+0.1+0.1 = 1) + # + # 1 1000 0.8 + # 1001 10000 0.1 + # 10001 100000 0.1 + # + #example2: all value length are 128 bytes + # + # 128 128 1 + value + 2048 2048 1 + #cmd + #cmd_type cmd_proportion + # + #currently asynchronous memslap only supports get and set command. + # + #cmd_type + #set 0 + #get 1 + # + #example: set command accounts for 50% + # get command accounts for 50% + # total should be 1 (0.5+0.5 = 1) + # + # cmd + # 0 0.5 + # 1 0.5 + cmd + 0 0.1 + 1.0 0.9 + + + +=head1 Format of output + +At the beginning, asynchronous memslap displays some configuration information as follows: + +=over 4 + +=item threads count: 1 + +=item concurrency: 16 + +=item run time: 20s + +=item windows size: 10k + +=item set proportion: set_prop=0.10 + +=item get proportion: get_prop=0.90 + +=back + +=head4 Where + +=over 4 + +=item threads count + +The number of threads asynchronous memslap runs with. + +=item concurrency + +The number of concurrencies asynchronous memslap runs with. + +=item run time + +How long to run asynchronous memslap. + +=item windows size + +The task window size of each concurrency. + +=item set proportion + +The proportion of set command. + +=item get proportion + +The proportion of get command. + +=back + +The output of dynamic statistics is something like this: + + --------------------------------------------------------------------------------------------------------------------------------- + Get Statistics + Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) + Avg(us) Std_dev Geo_dist + Period 5 345826 69165 65.3 0 27 2198 203 + 95.43 177.29 + Global 20 1257935 62896 71.8 0 26 3791 224 + 117.79 192.60 + + + Set Statistics + Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) + Avg(us) Std_dev Geo_dist + Period 5 38425 7685 7.3 0 42 628 240 + 88.05 220.21 + Global 20 139780 6989 8.0 0 37 3790 253 + 117.93 224.83 + + + Total Statistics + Type Time(s) Ops TPS(ops/s) Net(M/s) Get_miss Min(us) Max(us) + Avg(us) Std_dev Geo_dist + Period 5 384252 76850 72.5 0 27 2198 207 + 94.72 181.18 + Global 20 1397720 69886 79.7 0 26 3791 227 + 117.93 195.60 + --------------------------------------------------------------------------------------------------------------------------------- + +=head4 Where + +=over 4 + +=item Get Statistics + +Statistics information of get command + +=item Set Statistics + +Statistics information of set command + +=item Total Statistics + +Statistics information of both get and set command + +=item Period + +Result within a period + +=item Global + +Accumulated results + +=item Ops + +Total operations + +=item TPS + +Throughput, operations/second + +=item Net + +The rate of network + +=item Get_miss + +How many objects can’t be gotten + +=item Min + +The minimum response time + +=item Max + +The maximum response time + +=item Avg: + +The average response time + +=item Std_dev + +Standard deviation of response time + +=item Geo_dist + +Geometric distribution based on natural exponential function + +=back + +At the end, asynchronous memslap will output something like this: + + --------------------------------------------------------------------------------------------------------------------------------- + Get Statistics (1257956 events) + Min: 26 + Max: 3791 + Avg: 224 + Geo: 192.60 + Std: 116.23 + Log2 Dist: + 4: 0 10 84490 215345 + 8: 484890 459823 12543 824 + 12: 31 + + Set Statistics (139782 events) + Min: 37 + Max: 3790 + Avg: 253 + Geo: 224.84 + Std: 116.83 + Log2 Dist: + 4: 0 0 4200 16988 + 8: 50784 65574 2064 167 + 12: 5 + + Total Statistics (1397738 events) + Min: 26 + Max: 3791 + Avg: 227 + Geo: 195.60 + Std: 116.60 + Log2 Dist: + 4: 0 10 88690 232333 + 8: 535674 525397 14607 991 + 12: 36 + + cmd_get: 1257969 + cmd_set: 139785 + get_misses: 0 + verify_misses: 0 + verify_failed: 0 + expired_get: 0 + unexpired_unget: 0 + written_bytes: 242516030 + read_bytes: 1003702556 + object_bytes: 152086080 + packet_disorder: 0 + packet_drop: 0 + udp_timeout: 0 + + Run time: 20.0s Ops: 1397754 TPS: 69817 Net_rate: 59.4M/s + --------------------------------------------------------------------------------------------------------------------------------- + +=head4 Where + +=over 4 + +=item Get Statistics + +Get statistics of response time + +=item Set Statistics + +Set statistics of response time + +=item Total Statistics + +Both get and set statistics of response time + +=item Min + +The accumulated and minimum response time + +=item Max + +The accumulated and maximum response time + +=item Avg + +The accumulated and average response time + +=item Std + +Standard deviation of response time + +=item Log2 Dist + +Geometric distribution based on logarithm 2 + +=item cmd_get + +Total get commands done + +=item cmd_set + +Total set commands done + +=item get_misses + +How many objects can’t be gotten from server + +=item verify_misses + +How many objects need to verify but can’t get them + +=item verify_failed + +How many objects with insistent value + +=item expired_get + +How many objects are expired but we get them + +=item unexpired_unget + +How many objects are unexpired but we can’t get them + +=item written_bytes + +Total written bytes + +=item read_bytes + +Total read bytes + +=item object_bytes + +Total object bytes + +=item packet_disorder + +How many UDP packages are disorder + +=item packet_drop + +How many UDP packages are lost + +=item udp_timeout + +How many times UDP time out happen + +=item Run time + +Total run time + +=item Ops + +Total operations + +=item TPS + +Throughput, operations/second + +=item Net_rate + +The average rate of network + +=back + +=head1 OPTIONS + -s, --servers= List one or more servers to connect. Servers count must be less than threads count. e.g.: --servers=localhost:1234,localhost:11211 @@ -112,10 +986,12 @@ memslap -s 127.0.0.1:11211,127.0.0.1:11212 -F config -t 2m -p 2 To find out more information please check: L +L + =head1 AUTHORS +Mingqiang Zhuang Emingqiangzhuang@hengtiansoft.comE (Schooner Technolgy) Brian Aker, Ebrian@tangent.orgE -Mingqiang Zhuang Emingqiangzhuang@hengtiansoft.comE =head1 SEE ALSO -- 2.30.2