How to perform performance test both for throughput and latencies for my MDEX engine
Note: The below information is for testing MDEX engine performance only.
Additional tools is required to test performance running through your front end application.
Additional tools is required to test performance running through your front end application.
To test MDEX engine performance, we use a program called eneperf. The first is a command line tool included with Endeca. Essentially send requests at a given host and port where you can ramp up concurrent connections to evaluate load and throughput.
The great thing about eneperf is that it add almost no overhead to the query at all. It do nothing with the response object so very little application overhead is added.
Other tools e.g. loadrunner can add latency to the round trip times that can skew your results. These times will show up in an admin stats as “send time” and in the request log as the delta between engine time and response time minus queue time. For testing the performance of the MDEX engine only using a test harness with no overhead is optimal.
Other tools e.g. loadrunner can add latency to the round trip times that can skew your results. These times will show up in an admin stats as “send time” and in the request log as the delta between engine time and response time minus queue time. For testing the performance of the MDEX engine only using a test harness with no overhead is optimal.
Use your production traffic to generate the queries you are going to send through your MDEX engine. Take a production request log and run it through the Request Log Parser utility which strips out all of the non-request information from each line. This gets you a log to use with eneperf or mdex perf.
For example, when you apply request log parser to a request log;
1322705649 127.0.0.1 – 178456 346.67 249.47 200 5 0 10 /graph?node=196&group=0&offset=0&nbins=50&offset=0&sort=price_demo|desc
Becomes
/graph?node=196&allgroups=1&group=0&offset=0&&nbins=50&offset=0&sort=price_demo|desc
We then send these queries at the MDEX engines in different configurations of threading, mdex flags, Endeca configuration and options using eneperf.
Your request log should have enough queries as to fill up the Endeca cache.
If not, and you loop your log using <numiterations> you will get all of your responses after the initial run back from the Endeca cache and the engine will be not be under its normal load.
Depending on your site and type of queries a good starting point is to have at least 10,000 unique queries and preferably 20-40,000 queries in your log for a good test.
If not, and you loop your log using <numiterations> you will get all of your responses after the initial run back from the Endeca cache and the engine will be not be under its normal load.
Depending on your site and type of queries a good starting point is to have at least 10,000 unique queries and preferably 20-40,000 queries in your log for a good test.
Eneperf uses the flags of <num connections> and <num iterations> which do the following
<num connections> Maximum number of outstanding (unreplied) requests to allow before waiting for replies, i.e., the number of simultaneous HTTP connection streams to keep open at all times.
This number emulates multiple clients for the target server. For example, using <num connections> of 16 emulates 16 concurrent clients querying the target server at all times. <num iterations> Number of times to replay the URL log.
<num connections> Maximum number of outstanding (unreplied) requests to allow before waiting for replies, i.e., the number of simultaneous HTTP connection streams to keep open at all times.
This number emulates multiple clients for the target server. For example, using <num connections> of 16 emulates 16 concurrent clients querying the target server at all times. <num iterations> Number of times to replay the URL log.
Example
You have a dgraph you want to test that is configured for 4 threads.You run eneperf with the following command line:
eneperf <mdexHost> <mdexPort> parsed.log 8 1
Here you see numconnections 8. This will mean all 4 threads of your dgraph will always be busy and you will always have 4 queries in queue. (actually in practice you will have just under 4 queries in queue as there will be a lag between when the eneperf tool sends a response and when it actually hits the mdex queue where there will be 3 queries in queue. Typically this will be only a few ms per query depending on network speed etc.) That said, the way we typically test *throughput* performance (queries/second) is to make num connections = num MDEX threads +1 which will create a slight queuing environment.
To test for *latency* (engine processing time OR engine processing time plus travel time to and from the MDEX engine) performance we make num connections = num MDEX threads so we don’t create a queue and therefore skew latency numbers.
Once you have run the test, you can then analyze the results:
Eneperf will give some excellent statistics on the speed of the test.
Example
Running iteration 1… Done:
29422 sent, 29421 received, 1 errors.
51 minutes, 50.62 seconds for iteration 1, ****9.45823 req/sec.***** average throughput for the iteration (from the num iterations flag)
51 minutes, 50.62 seconds elapsed (user: 1.54 seconds, system: 5.43 seconds).
Net: 2.15397e+06 KB (692.455 KB/sec).
Page Size: avg=73.21 KB, std dev=80.51 KB, max=259.27 KB, min=0.20 KB.
Eneperf completed:
29422 sent, 29421 received, *****1 errors.***** Number of errors see the err output for details on errors.
51 minutes, 50.62 seconds elapsed (user: 1.54 seconds, system: 5.43 seconds).
Net: 2.15397e+06 KB (692.455 KB/sec).
Page Size: avg=73.21 KB, std dev=80.51 KB, max=259.27 KB, min=0.20 KB.
Best iteration time: 51 minutes, 50.62 seconds.
Peak rate: 9.45823 req/sec
Avg iteration time: 51 minutes, 50.62 seconds.
Avg rate: 9.45823 req/sec.
*****Total rate: 9.45823 req/sec.***** Total rate for the entire run.
Second, you can use cheetah and run it against the request log you produce in your eneperf test. This, especially combined with the –showAll flag, will break your traffic down into some sub categories and show you your worst engine times, average round trip times queuing information etc.
Thirdly we look at an admin?op=stats page on the mdex engine after the test to see if we can understand where there are possible performance bottlenecks. Especially in the hotspots section of the “Details” tab.
Fourth: Frequently we will monitor things like ram usage, virtual memory statistics, disk I/O and cpu usage to determine if there is not some other bottleneck as well. In windows you can use perfmon to set up counters to analyze various things. In *nix you can output top and iostat and vmstat etc. to files for analysis.
A typical performance testing procedure will go something like this.
· Get a baseline of current performance
· Make one or two changes
· Test performance changes and compare against baseline
· Make second changes
· Test performance changes and compare against baseline and test 2
· Iterate
Example
1. Set up an environment that is a hardware mirror to production. (or as close as you can get)
2. Set up system monitoring software e.g. perfmon in windows or top, iostat or vmstat in Linux etc. record the output of these tests.
3. Stop your MDEX engines and clear all request and error logs.
4. Start your MDEX engines with regular settings
5. Run eneperf using the parsed logs from your production environment.
6. Save the output from eneperf with a descriptive label of this particular test
7. Save off a copy of the request AND error logs also with the same descriptive label of the test
8. go to http://<dgraphhost>:<dgrapport>/admin?op=stats and save the resulting file as <descriptive_test_name.xml >
9. Stop your MDEX engines
10. Make the first changes you are implementing. Example addition of a new thesaurus, a removal of a significant query option, or a change in MDEX flags.
11. Clear your request and error logs again.
12. Start your MDEX engines with new configuration and settings
13. Run eneperf using the parsed logs from your production environment.
14. Save the output from eneperf with a NEW descriptive label of this particular test
15. Save off a copy of the request AND error logs also with the same descriptive label of the test
16. go to http://<dgraphhost>:<dgrapport>/admin?op=stats and save the resulting file as <NEWdescriptive_test_name.xml >
17. Repeat making small changes each time to be able to easily determine what changes had an impact.
Some Examples of things you would want to run performance analysis:
to test removing “allgroups=1” (setNavAllRefinements=true) from your application you can remove this query option from all of the queries in the perf log before using eneperf to send traffic at the engine.
You can test adding a new thesaurus with 10000 new entries
You can test the performance impact of a new server or vs. your current hardware.
You can test the impact of a new data source or new dimensions added to your project
Connect kubernetes pod to a GCS bucket using JS
To connect from a Kubernetes pod to a Google Cloud Storage (GCS)…Easiest way to run an LLM locally on your Mac
I recently sought an efficient method for local experimentation with Language Model…EKS cluster using an existing VPC
The eksctl command line tool can create a cluster by either command-line options or…kubectl Unable to connect to the server
When working with Kubernetes if you are getting Unable to conntect to…
