Debug School: Joshica S

Datadog Training Day 4 - Assignment (14-09-2023)

Joshica S — Thu, 14 Sep 2023 07:15:42 +0000

1 Top 10 metrics/indicators for APM
i. APM Trace Metrics - Tracing application metrics are collected after enabling trace collection and instrumenting your application
ii. Request Count- trace.<SPAN_NAME>.hits - Represent the count of hits for a given span.
trace.<SPAN_NAME>.hits.by_http_status - Represent the count of hits for a given span break down by HTTP status code.
iii. Latency distribution - trace.<SPAN_NAME> - Represent the latency distribution for all services, resources, and versions across different environments and second primary tags.
iv. Errors - trace..errors - Represent the count of errors for a given span.
trace.<SPAN_NAME>.errors.by_http_status - Represent the count of errors for a given span.
v. Span count - trace..span_count - Represent the amount of spans collected on a given interval.

trace.<SPAN_NAME>.span_count.by_http_status- Represent the amount of spans collected on a given interval break down by HTTP status.
vi. Duration - trace.<SPAN_NAME>.duration - Measure the total time for a collection of spans within a time interval
vii. Duration by - trace.<SPAN_NAME>.duration.by_http_status - Measure the total time for a collection of spans for each HTTP status
trace.<SPAN_NAME>.duration.by_service - Measure the total time spent actually processing for each service
trace.<SPAN_NAME>.duration.by_type - Measure the total time spent actually processing for each Service type.
trace.<SPAN_NAME>.duration.by_type.by_http_status - Measure the total time spent actually processing for each Service type and HTTP status.
trace.<SPAN_NAME>.duration.by_service.by_http_status - Measure the total time spent actually processing for each Service and HTTP status.
viii. Apdex - trace.<SPAN_NAME>.apdex - Measures the Apdex score for each web service.
trace.<SPAN_NAME>.apdex.by.resource_<2ND_PRIM_TAG>_service - Represents the Apdex score for all combination of resources, 2nd primary tags and services.
trace.<SPAN_NAME>.apdex.by.resource_service - Measures the Apdex score for each combination of resources and web services.

trace.<SPAN_NAME>.apdex.by.<2ND_PRIM_TAG>_service - Measures the Apdex score for each combination of 2nd primary tag and web services.
trace.<SPAN_NAME>.apdex.by.service - Measures the Apdex score for each web service.
ix. Runtime metrics - Runtime metrics collection can be enabled with the DD_RUNTIME_METRICS_ENABLED=true environment parameter when running with ddtrace-run.
x. Generate custom metrics from spans - Monitor Service Metrics - For send requests, Errors, Latency percentiles, Analyse database queries or endpoints

Top 10 metrics/indicators for Synthetic monitoring The following metrics are generated by Synthetic Monitoring tests and Continuous Testing settings. a.General Metrics - synthetics.test_runs - The number of Synthetic test runs. b.API tests - synthetics.api.response - The count of API responses we receive. c.HTTP tests - synthetics.http.* come from your API HTTP tests d.SSL tests - synthetics.ssl.* come from your API SSL tests e.DNS tests - synthetics.dns.* come from your API DNS tests f.WebSocket tests - synthetics.websocket.* come from your API WebSocket tests g.TCP tests - synthetics.tcp.* come from your API TCP tests h.UDP tests - synthetics.udp.* come from your API UDP tests i.ICMP tests - synthetics.icmp.* come from your API ICMP tests j.Multistep API tests - synthetics.multi.* come from your multistep API tests k.Browser tests - synthetics.browser.* come from your browser tests l.Private locations - synthetics.pl.come from your private locations Metrics starting with: synthetics.test_runs come from all your Synthetic tests datadog.estimated_usage.synthetics. return relevant usage data from your Synthetic tests synthetics.on_demand return relevant usage data for Continuous Testing

3 Top 10 metrics/indicators for RUM
Datadog Real User Monitoring (RUM) provides end-to-end visibility into the user experience and performance of your browser and mobile applications. RUM allows you to capture and retain complete user sessions for 30 days. This means you can pinpoint bugs, prioritize issues, and determine fixes with data collected across an entire quarter.

User satisfaction and Apdex
FP/ FCP How to Improve FP and FCP
Time to interactive How to Improve Your TTI score
Page speed and load time
Time to first byte How to Improve TTFB
DNS Lookup time
Error Rate
Peak Response Time
Hardware utilization
Uptime +++++++++++++++++++++++++++++END++++++++++++++++++++++++++++++

Datadog Training - Day 2 Assignment (12-9-2023)

Joshica S — Tue, 12 Sep 2023 12:26:30 +0000

I. Write it down a step to collect apache metrices to datadog
1. Install Datadog Agent in Ubuntu
Register at https://www.datadoghq.com/
Select the UBUNTU from the Datadog supported agent list
Run the commands in ubuntu -

             $ D_AGENT_MAJOR_VERSION=7 DD_API_KEY=XXX DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"

systemctl start datadog-agent systemctl status datadog-agent datadog-agent status You would get output at the end of Installation
Adding you api key to Agent config: /etc/datadog-agent/datadog.yml

2. Install Apache HTTPD server in Ubuntu

$ sudo apt update $ sudo apt install apache2 # Setup webiste in default virtual host
$ rm /var/www/html/index.html $ vi /var/www/html/index.html

    <html>
    <head>
        <title> Welcome to DevOpsSchool.com </title>
    </head>
    <body>
        <p> I'm running this website on an Ubuntu Server server!
    </body>
    </html>

# Restart Apache

$ service apache2 reload $ service apache2 restart # Validate
$ watch curl http://localhost
3. Go to datadog console, https://app.datadoghq.com/integrations -> Integrations -> Search For Apache and Install the integration
4. Enable mod_status & ExtendedStatus in Apache
By default, Apache ships with the mod_status module already enabled. Verify by running, $ ls /etc/apache2/mods-enabled | grep status*
Output:
status.conf
status.load
Enable mod_status module by invoking the command:
$ sudo /usr/sbin/a2enmod status Configure mod_status in Apache Ubuntu
` sudo vi /etc/apache2/mods-enabled/status.conf

Require all granted

`

Replace ip <>

$ sudo systemctl restart apache2 $ sudo systemctl status apache2 $ curl http://localhost/server-status 5. Enable Datadog Apache Integration configuration to start collecting your Apache metrics.
` $ cd /etc/datadog-agent/conf.d/apache.d/
$ sudo cp conf.yaml.example conf.yaml
$ sudo vi conf.yaml

`              instances:
                    - apache_status_url: http:localhost/server-status?auto        #localhost can be added with IP if it is a remote agent

$ sudo systemctl restart datadog-agentwatch curl http://localhost/ 6. Validate if datadog agent is monitoring Apache
$ datadog-agent configcheck | grep apache -A 5 -B 5 Output:
=== apache check ===
Configuration provider: file
Configuration source: file:/etc/datadog-agent/conf.d/apache.d/conf.yaml
$ datadog-agent status | grep apache -B 5 -A 15 ==Running Checks apache====

Create a dummy traffic on Apache

        $ while true; do curl -s -o /dev/null http://localhost & done

Verify the Apache Metrix on Datadog Console

II. Write it down a step to collect tomcat metrices to datadog
Step A: Loginto Datadog console, Integrations -> Search for Tomcat and Click Install
Step 1: Install Open JDK 11
$ sudo apt-get update $ sudo apt-get install openjdk-11-jdk -y $ apt install openjdk-11-jdk-headless -yStep 2: Download Apache Tomcat 9 in ubuntu:
$ sudo apt-get install wget unzip -y $ sudo -s $ cd /opt/ $ wget https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.80/bin/apache-tomcat-9.0.80.zip $ unzip apache-tomcat-9.0.80.zip $ cd apache-tomcat-9.0.73 $ cd bin $ chmod -R 755 . $ ls $ ./shutdown.sh $ ./startup.shStep 3: Enable JMX Remote in Tomcat 8 to Monitor & Administer $ vi /opt/apache-tomcat-9.0.73/bin/catalina.sh

Add these below lines

CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

$ ./shutdown.sh $ ./startup.sh
Step 4: Configure and enable Tomcat intergration in Datadog agent
$ cd /etc/datadog-agent/conf.d/tomcat.d/ $ cp conf.yaml.example conf.yaml $ datadog-agent configcheck $ systemctl restart datadog-agent $ systemctl status datadog-agent $ datadog-agent configcheck | grep tomcat -A 1 B 1 Output:
==tomcat check====
Step 5: In datadog console, Go to Metrics tab, search for tomcat metrices and also in Infrastructure list under Ubuntu VM check for tomcat apps

III. Write it down a step to collect docker metrices to datadog
Step 1 – Enable Datadog Process Monitoring
Step 2 – Install Docker Server
Step 3 – Enable Docker integration with Datadog

These commands to configure Agent for Docker Intergration.

Add the user running the Agent to Docker's group.

$ usermod -a -G docker dd-agent

`$ cd /etc/datadog-agent/conf.d/docker.d/
$ cp conf.yaml.default conf.yaml
$ vi conf.yaml

`
ad_identifiers:

_docker init_config: instances:
- url: "unix://var/run/docker.sock" new_tag_names: true ` $ systemctl restart datadog-agent $ systemctl status datadog-agent $ datadog-agent configcheck $ datadog-agent check docker $ datadog-agent status | grep docker -B 5 -A 5 ========= Collector ========= Step 4 – Create few containers and Verify$ docker run -itd ubuntu $ docker run -itd ubuntu $ docker run -itd ubuntu $ docker ps `Step 5 - Verify Containers at Datadog console In console -> Integrations -> Docker Install -> Metrics -> Explorer -> Search for command:containerd-shim-runc-v2

IV. - Write it down a step to collect mysql metrices to datadog Step 1 – Install Datadog Agent in Ubuntu
Register at https://www.datadoghq.com/
Select the UBUNTU from the Datadog supported agent list
Run the commands in ubuntu -

            $ D_AGENT_MAJOR_VERSION=7 DD_API_KEY=XXX DD_SITE="datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_script.sh)"
                $ systemctl start datadog-agent
                $ systemctl status datadog-agent
        $ datadog-agent status

          You would get output at the end of Installation
              Adding you api key to Agent config: /etc/datadog-agent/datadog.yml
    (Mysql by default comes with Dataagent package)

Step 2 – Prepare MySQL. On each MySQL server, create a database user for the Datadog Agent:
Go to datadog console, Integrations, Mysql -> Install
Step 3 - Login Mysql in Ubuntu. Run the below commands

        $ sudo su -
        $ mysql -h localhost -u root -p
        mysql> CREATE USER 'datadog'@'localhost' IDENTIFIED BY '<UNIQUEPASSWORD>';
        mysql> CREATE USER 'datadog'@'localhost' IDENTIFIED WITH mysql_native_password by '<UNIQUEPASSWORD>';
        mysql -u datadog --password='<UNIQUEPASSWORD>' -e "show status" | \
grep Uptime && echo -e "3[0;32mMySQL user - OK3[0m" || \
echo -e "3[0;31mCannot connect to MySQL3[0m"
        mysql -u datadog --password='<UNIQUEPASSWORD>' -e "show slave status" && \
echo -e "3[0;32mMySQL grant - OK3[0m" || \
echo -e "3[0;31mMissing REPLICATION CLIENT grant3[0m"

Step 4 - The Agent needs a few privileges to collect metrics. Grant the user the following limited privileges ONLY:

        mysql> GRANT REPLICATION CLIENT ON *.* TO 'datadog'@'localhost' WITH MAX_USER_CONNECTIONS 5;
        mysql> GRANT PROCESS ON *.* TO 'datadog'@'localhost';
        mysql> ALTER USER 'datadog'@'localhost' WITH MAX_USER_CONNECTIONS 5;

If enabled, metrics can be collected from the performance_schema database by granting an additional privilege:

        mysql> show databases like 'performance_schema';
Output:     
            | Database (performance_schema) |
                +-------------------------------+
                | performance_schema            |
        mysql> GRANT SELECT ON performance_schema.* TO 'datadog'@'localhost';

Step 5 - Configure Datadog agent to start collection mysql metrics.
Edit the mysql.d/conf.yaml file, in the conf.d/ folder at the root of your Agent’s configuration directory to start collecting your MySQL metrics and logs

        $ cd /etc/datadog-agent/conf.d/mysql.d/
        $ sudo cp conf.yaml.example conf.yaml
        $ sudo vi conf.yaml
        init_config:
        instances:
          - server: 127.0.0.1
            user: datadog
            pass: "<YOUR_CHOSEN_PASSWORD>" # from the CREATE USER step earlier
            port: "<YOUR_MYSQL_PORT>" # e.g. 3306
            options:
                  replication: false
                  galera_cluster: true
                  extra_status_metrics: true
                  extra_innodb_metrics: true
                  extra_performance_metrics: true
                  schema_size_metrics: false
                  disable_innodb_metrics: false

Note: Wrap your password in single quotes in case a special character is present.
Step 6: Validate mysql metrices
$ systemctl restart datadog-agent $ datadog-agent statusIn metrics explorer UI console, You would be able to see the mysql data reporting

+++++++++++++++++++++++++++++++++++END+++++++++++++++++++++++++

DataDog Training -Assignment Day 1 - 11-09-2023 Set 2

Joshica S — Mon, 11 Sep 2023 12:21:18 +0000

1 Top 10 commands of Datadog Agent

The 10 significant commands that we use for Datadog Agent (for Linux) are:

datadog-agent status
systemctl start datadog-agent
systemctl stopdatadog-agent
systemctl status datadog-agent
systemctl restart datadog-agent
datadog-agent health
datadog-agent configcheck
datadog-agent version
datadog-agent
more /etc/datadog-agent/datadog.yaml

2 Locate how to enable Process monitoring in datadog.yaml?

Step 1. Edit the datadog config file by logging into your server and running
sudo vi /etc/datadog-agent/datadog.yaml
Step 2: Add the following lines to the bottom of the file

process_config:
    enabled: true

Step 3 Restart the datadog agent
sudo systemctl restart datadog-agent sudo systemctl status datadog-agent
Step 4 . You can view all of your processes by going to Datadog Console >> Infrastructure >> Processes.

3 Top 5 Techniques for troubleshooting Datadog Agent

Firstly, use the command "sudo systemctl status datadog-agent" to determine whether the Datadog Agent is running. In the event that the agent isn't running, do a restart by running this command sudo systemctl restart datadog-agent
Check the logs and configuration: Check the logs for any error messages that might indicate issues with the agent. The directory /var/log/datadog contains the agent logs. Try to resolve the issue based on the error message by looking for any stack traces or error messages. Make sure the configuration of the agent is correct by verifying it. Assuming the programming interface key and different subtleties are accurately arranged. Assuming the API key,region and other details are correct. The agent configuration file is located in /etc/datadog-agent/datadog.yaml.
Verify the Connections: Verify whether the agent can connect to the Datadog backend. Run the command "sudo datadog-agent status". In the event that the status is "not running", there might be an issue with the agent configuration or connectivity.
Check for Firewalls: In the event that you are running a firewall on the machine, ensure that the agent is permitted to communicate with the Datadog backend.The agent communicates over ports 443 and 8125. If these ports are blocked by a firewall, the agent may not be able to communicate with the backend.
Check for Resource Constraints: Check if the agent is running into resource constraints, such as running out of disk space or memory. If the agent is running into resource constraints, try freeing up resources or increasing the resource limits for the agent.

DataDog Training Day -1 Assigment Q/A Set 1

Joshica S — Mon, 11 Sep 2023 08:16:21 +0000

What is Observability and its advantage? Observability is the evolution of monitoring where we understand the actual system's state/behaviour. It helps us to build the system better since we get to know the internal condition of the complex system. It collects system data to identify a system's state and performance and is built on top of monitoring. It requires the knowledge of 3 support points - metrics,logs and traces.

It has different advantages and benefits:

It aids in recognizing the system's complexity.
It assists with recognizing the difficult issues
It aids in the troubleshooting of issues at their source.
It increased the automation process
It aids in processes of self-healing.
It makes the user's experience better.
1. Difference between Observability Monitoring? Observability:
  1. A system has to be designed to be observed
  2. It assists in understand how system is performing
  3. It is passive where we can push/publish
  4. It incorporates metrics,logs and traces
  5. It helps isolate issue at scale
  6. It reduces duration of outages
  7. Generate metrics
  8. It helps to detect why my system has failed
  9. It helps to create resilience Monitoring:
  10. Almost anything can be monitored
  11. It helps to understand if the system is working
  12. It is active where we can pull/collect
  13. It incorporates metrics,events and logs
  14. It helps in challenging at scale
  15. It provides rapid response to outages
  16. Collect metrics
  17. It helps to understand the state of my system
  18. It helps to fix based on past scenarios
2. What is Datadog? Datadog is a monitoring and analytics tool for large scale applications. It is useful for determining performance metrics and event monitoring for cloud and infrastructure services. It offers metrics, visualization, and alerting for applications' maintenance and optimization. The primary four signals for datadog are latency, traffic, errors, and saturation. It upholds Windows,Linux and Macintosh operating system and it is accessible and it upholds cloud suppliers like AWS,GCP,Azure and so forth.
3. What is Datadog Agent and How it works?
4. The Datadog Agent is software that runs on hosts.
5. It sends events and metrics from hosts to Datadog, where monitoring and performance data can be analyzed.
6. The Datadog Agent's source code can be found on GitHub at DataDog/datadog-agent because it is open source. Working Mechanism: The Agent should be installed completely. On the other hand, Amazon Linux, CentOS, Debian, Fedora, Red Hat, SUSE, and Ubuntu all come with a separate DogStatsD package. The standalone DogStatsD package is installed with the Agent one-line install command except every occurrence of datadog-agent should be replaced with datadog-dogstatsd. A Docker image is available in the DogStatsD6 Docker image repo. In containerized environments where DogStatsD runs as a sidecar or in environments with a DogStatsD server but no full Agent functionality, this package is used.
7. Component of Datadog Agent and short intro for each Datadog Agent is composed of a main process responsible for collecting infrastructure metrics, logs, and receiving DogStatsD metrics. The main components to this process are: The Collector is in charge of running checks and collecting metrics. The Forwarder sends payloads to Datadog. Two optional processes are spawned by the Agent if enabled in the datadog.yaml configuration file: Datadog Agent is composed of a main process responsible for collecting infrastructure metrics, logs, and receiving DogStatsD metrics. The primary elements of this procedure are: The Collector is in charge of running checks and collecting metrics. The Forwarder sends payloads to Datadog. If the datadog.yaml configuration file is set to "enabled," the Agent will launch two optional processes:

A process for collecting traces is the APM Agent (enabled by default).
The Process Agent is a method for obtaining live process data. In the absence of available containers, it is disabled by default.

Collector
The collector collects all standard metrics every 15 seconds. Agent v6 embeds a Python 2.7 interpreter to run integrations and custom checks.

Forwarder
The Agent forwarder send metrics over HTTPS to Datadog. Buffering prevents network splits from affecting metric reporting. Metrics are buffered in memory until a limit in size or number of outstanding send requests are reached. Afterwards, the oldest metrics are discarded to keep the forwarder’s memory footprint manageable. Logs are sent over an SSL-encrypted TCP connection to Datadog.

DogStatsD
In v6, DogStatsD is a Golang execution of Etsy's StatsD metric total daemon. It is utilized to get and turn up inconsistent measurements over UDP or Unix attachment, accordingly permitting custom code to be instrumented without adding inertness.

GUI
In the datadog.yaml file, you can set the port on which the GUI runs. Set the value of the port to -1 to disable the GUI. For Windows and macOS, the GUI is empowered of course and runs on port 5002. For Linux, the GUI is handicapped as a matter of course.
When the Agent is running, use the datadog-agent launch-gui command to open the GUI in your default web browser.