High cpu or memory usage issues

High CPU usage

Logs investigation

1. Identify which service is using more service. Go through the Access logs from nginx for indentifing the max time taking services. status=200; req_time=346924; rdbms_time=267; rdbms_count=6; authz_time=91; authz_count=3; depsolver_time=146; depsolver_count=1
2. Find the different response types from the access logs using the below command. awk '{print $9}' /var/log/opscode/nginx/access.log | sort | uniq -c | sort -rn sample output

    424886 200
    106221 404
         2 499
   

3. The count of requests per second over the life of the log: cat access.log | awk '{print $4}' | uniq -c sample output

   280 rps
   

4. For example considering depsolver is taking more time for response.
5. Run the fprof for finding which function is the rootcause for longer response in the erchef console. redbug:start("chef_wm_depsolver:make_json_list", [{print_file, "/tmp/redbug.out"}, {file_size, 150}, {msgs,1}]).
6. This captures one execution of make_json_list and prints the function args to file.Next, I had to edit the file (redbug.out) to make it a valid erlang term - so removing the function call, and basically just leaving behind the argument I cared about -- the single long list of cookbook versions. After that:

   {ok, [Content|_]} = file:consult("/tmp/redbug.out").`
   % Run fprof to profile the function in question. We'll use the argument data we just captured in redbug as the input.
   fprof:apply(chef_wm_depsolver, make_json_list, [Content, "https://[2600:1f1c:f24:ad01:b300:cfe6:5f15:b905]", 1], 
    [{file, "/tmp/fprof.trace"}]).
   

7. This handy little escript converts the trace to callgrind format: https://github.com/isacssouza/erlgrind

Load testing setup

1. Setup chef-server & 4 load servers in the AWS console using below AMI's

   chef-server-load-test-03122021(ami-0e2ad9ec5256c7b4c)
   Load generator backup
   load-gen-backup-03122021(ami-0e2ad9ec5256c7b4c)
   

2. Upgrade chef-server to specific version using https://docs.chef.io/server/upgrades/
3. Create user's and organization in the chef-server using the below commands.

chef-server-ctl org-create test1 test1 > test1_validator.pem
chef-server-ctl user-create testuser1 test test [test@example.com](mailto:test@example.com) password > /home/ubuntu/testuser1.pem
chef-server-ctl org-user-add -a test1 testuser1

3. Use specific branch of chef-load (https://github.com/chef/chef-load/tree/mp/working)
4. Copy all the users/client keys from chef-server to chef-load for generating load

Copy the pem's to local and then to load servers
scp -i ~/.ssh/aws-shared-chef-infra-server.pem ubuntu@52.53.176.180:/home/ubuntu/*.pem .

scp -i ~/.ssh/aws-shared-chef-infra-server.pem *.pem ubuntu@54.241.71.101:/home/ubuntu/testing-12.3.1
scp -i ~/.ssh/aws-shared-chef-infra-server.pem *.pem ubuntu@184.169.252.217:/home/ubuntu/testing-12.3.1
scp -i ~/.ssh/aws-shared-chef-infra-server.pem *.pem ubuntu@54.177.170.21:/home/ubuntu/testing-12.3.1
scp -i ~/.ssh/aws-shared-chef-infra-server.pem *.pem ubuntu@3.101.121.52:/home/ubuntu/testing-12.3.1