Introduction to Datadog - Pavana Kumar N

What is Observability and its advantage?

Observabiity is a superset of monitoring, where the monitored events are correlated and enriched to provide context of the issues as well as the cause and analysis of the underlying issues that contributed to degradation of performance and availability.
Advantages of observability includes:
i. Capable of handling rapid changes to the systems, architecture and technonlogies in monitoring the entites.
ii. Provides a deeper understanding and correlation of an issue, the reason for the issue.

What is the difference between monitoring and observability?

Monitoring is the passive, reactive method of ensuring application and infrastructure availability to provide current status of the application availability. It is static in nature where the metrics collected are related to application / infrastructure - up/down, application performance is degraded with minimal insights on the causes.
Observability is the active and predictive method of ensuring application and infrastructure availabiltiy to ensure application health. It is dynamic in nature where the metrics collected are correlated to potential underlying causes that could contribute to issues with availabilty and performance of applications and infrastructure.

What is Datadog?

Datadog is a service that provides observability solutions for a wide range (more than 600+) of application technologies, infrastructure, network, security, availability and end user monitoring. It is an enterprise SaaS product that is designed to work on VMs as well as containerized infrasatructures.

What is Datadog agent and how does it work?

The Datadog Agent collects data from various sources, including:
Metrics: It collects system-level and application-level metrics, such as CPU usage, memory usage, disk space, and custom application metrics.
Traces: It can collect distributed traces from applications instrumented with APM (Application Performance Monitoring) libraries.
Logs: It can collect logs from log files, Docker containers, Kubernetes pods, and more.
Integrations: Datadog provides integrations for a wide range of technologies and services, including databases, web servers, cloud providers, and more. The agent collects data from these integrations.
The collected metrics are converted to Datadog compatible format and transmits it securely to Datadog backend.

Component of Datadog Agent and short intro for each.

The 3 main components of the Datadog agent are Collector, DogstatsD and Forwarder.
Collector: The Collector is the core component of the Datadog Agent responsible for gathering metrics, traces, and logs. It collects, aggregates and processes data from various sources, including system-level metrics, application-level metrics, distributed traces, and log files.
DogstatsD: DogstatsD is a StatsD based daemon service that is used for collecting custom application metrics such as throughput, response times and application server side errors.
Forwarder: Forwarder is the component responsbile for collecting the metrics from Collector and DogstatsD to the Datadog backend in a secured manner.

Top 10 commands of Datadog Agent

1. datadog-agent status
2. systemctl stop datadog-agent
3. systemctl start datadog-agent
4. systemctl status datadog-agent
5. datadog-agent configcheck
6. datadog-agent config
7. datadog-agent health
8. datadog-agent version
9. more /var/log/datadog/agent.log
10. more /etc/datadog-agent/datadog.yaml

Locate how to enable Process monitoring in datadog.yaml?

Edit the Live processes section as follows:
process_config:
process_collection:
enabled: "true"

Top 5 Techniques for troubleshooting Datadog Agent

1. Windows - use the Logs section of the Datadog agent manager
2. Linux - command: more /var/log/datadog/agent.log
3. Linux - command: more /var/log/datadog/process-agent.log
4. Linux - command: datadog-agent status
5. Linux - command: datadog-agent configcheck

Write down the steps to collect apache metrices to datadog

1. Install Datadog host agent for ubuntu (OS of choice)
2. Install Apache HTTPD using aptget
3. Modify the index.html under /var/www/html/ and reload and restart apache service.
4. In Datadog SaaS, search for Apache under integrations, and click on Install Integration.
5. In the Ubuntu host, go to /etc/datadog-agent/conf.d/apache.d/ and create a new file or copy the example yaml with the name conf.yaml.
6. Restart datadog agent.
7. Simulate traffic with command - hile true; do curl -s -o /dev/null http://localhost & done

Write down the steps to collect tomcat metrices to datadog

1. Install Datadog host agent for Ubuntu (OS of choice).
2. Install JDK on the host with apt-get and apt install ofjdk.
3. Install Apache tomcat by downloading the ZIP file of the version required and unzipping in /opt.
4. Change permissions to all the scripts under /opt/apache-tomcat/bin to execute files.
5. Modify the catalina.sh file to enable JMX by adding the code snippet -

CATALINA_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9012 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

1. Go to tomacat.d folder in /etc/datadog-agent/conf.d/ and copy / create a yaml file of the name conf.yaml.
2. Restart datadog agent with systemctl commands.
3. In Datadog SaaS, search for Tomcat under integrations, and click on Install Integration.
4. Open the tomcat UI on the browser with port 8080 and click on the webpage to simulate traffic.

Write down the steps to collect docker metrices to datadog

Install Datadog agent on Ubuntu (OS of choice).
Configure the datadog-agent.yaml to allow live process monitoring under the section process_config.
Restart the datadog agent.
Configure apt to connect to and get packages over https.
Add docker's GPG key.
Set up the repository and install docker and required packages with apt-get.
In Datadog SaaS, search for Docker under integrations, and click on Install Integration.
Create or copy the yaml file under docker.d in the datadog agent conf and update the instances to: - url: "unix://var/run/docker.sock"
Restart datadog agent and check datadog config.
Create new docker containers docker run -itd ubuntu.
Container metrics will be collected and can be viewed under metrics explorer.

Top 10 metrics/indicators for APM

1. Requests - no. of requests the application has received in a given time period.
2. P50 Latency -
3. P95 Latency -
4. Error rate -
5. Status Code -
6. Logs
7. Transactions
8. Traces
9. Spans
10. Resource

Top 10 metrics/indicators for Synthetic monitoring

1. Request Method
2. Response Status
3. Response Time
4. Response Time by location
5. Events
6. Response body
7. Assertions
8. Trace
9. Test ID
10. Tags

Top 10 metrics/indicators for RUM

1. User sessions
2. User views
3. Response Status for actions
4. Response Time for actions
5. Long tasks
6. Session error count
7. Session time spent
8. Durations
9. Trace
10. Error type, message & stack
11. Frustration signals