Write it down a step to collect apache metrices to datadog

To start gathering your Apache metrics and logs, you need to:

Install the Agent on your Apache servers.

Install mod_status on your Apache servers and enable ExtendedStatus.

Configuration
Host

To configure this check for an Agent running on a host:
Metric collection

Edit the apache.d/conf.yaml file in the conf.d/ folder at the root of your Agent's configuration directory to start collecting your Apache metrics. See the sample apache.d/conf.yaml for all available configuration options.

init_config:

instances:
  ## @param apache_status_url - string - required
  ## Status url of your Apache server.
  #
  - apache_status_url: http://localhost/server-status?auto

Restart the Agent.

Log collection

Available for Agent versions >6.0

Collecting logs is disabled by default in the Datadog Agent. Enable it in datadog.yaml:

logs_enabled: true

Add this configuration block to your apache.d/conf.yaml file to start collecting your Apache logs, adjusting the path and service values to configure them for your environment:

logs:
  - type: file
    path: /path/to/your/apache/access.log
    source: apache
    service: apache
    sourcecategory: http_web_access

  - type: file
    path: /path/to/your/apache/error.log
    source: apache
    service: apache
    sourcecategory: http_web_error

See the sample apache.d/conf.yaml for all available configuration options.

Restart the Agent.

Docker

To configure this check for an Agent running on a container:
Metric collection

Set Autodiscovery Integrations Templates as Docker labels on your application container:

LABEL "com.datadoghq.ad.check_names"='["apache"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='[{"apache_status_url": "http://%%host%%/server-status?auto"}]'

Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Docker Log Collection.

Then, set Log Integrations as Docker labels:

LABEL "com.datadoghq.ad.logs"='[{"source": "apache", "service": ""}]'

Kubernetes

To configure this check for an Agent running on Kubernetes:
Metric collection

Set Autodiscovery Integrations Templates as pod annotations on your application container. Aside from this, templates can also be configured with a file, a configmap, or a key-value store.

Annotations v1 (for Datadog Agent < v7.36)

apiVersion: v1
kind: Pod
metadata:
name: apache
annotations:
ad.datadoghq.com/apache.check_names: '["apache"]'
ad.datadoghq.com/apache.init_configs: '[{}]'
ad.datadoghq.com/apache.instances: |
[
{
"apache_status_url": "http://%%host%%/server-status?auto"
}
]
spec:
containers:
- name: apache

Annotations v2 (for Datadog Agent v7.36+)

apiVersion: v1
kind: Pod
metadata:
name: apache
annotations:
ad.datadoghq.com/apache.checks: |
{
"apache": {
"init_config": {},
"instances": [
{
"apache_status_url": "http://%%host%%/server-status?auto"
}
]
}
}
spec:
containers:
- name: apache

Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Kubernetes Log Collection.

Then, set Log Integrations as pod annotations. This can also be configured with a file, a configmap, or a key-value store.

Annotations v1/v2

apiVersion: v1
kind: Pod
metadata:
name: apache
annotations:
ad.datadoghq.com/apache.logs: '[{"source":"apache","service":""}]'
spec:
containers:
- name: apache

ECS

To configure this check for an Agent running on ECS:
Metric collection

Set Autodiscovery Integrations Templates as Docker labels on your application container:

{
"containerDefinitions": [{
"name": "apache",
"image": "apache:latest",
"dockerLabels": {
"com.datadoghq.ad.check_names": "[\"apache\"]",
"com.datadoghq.ad.init_configs": "[{}]",
"com.datadoghq.ad.instances": "[{\"apache_status_url\": \"http://%%host%%/server-status?auto\"}]"
}
}]
}

Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see ECS Log Collection.

Then, set Log Integrations as Docker labels:

{
"containerDefinitions": [{
"name": "apache",
"image": "apache:latest",
"dockerLabels": {
"com.datadoghq.ad.logs": "[{\"source\":\"apache\",\"service\":\"\"}]"
}
}]
}

Validation

Run the Agent's status subcommand and look for apache under the Checks section.

Write it down a step to collect tomcat metrices to datadog

The Tomcat check is included in the Datadog Agent package, so you don't need to install anything else on your Tomcat servers.

This check is JMX-based, so you need to enable JMX Remote on your Tomcat servers. Follow the instructions in Monitoring and Managing Tomcat.
Configuration
Host

To configure this check for an Agent running on a host:

Edit the tomcat.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration directory to collect Tomcat metrics and logs. See the sample tomcat.d/conf.yaml for all available configuration options.

Restart the Agent.

See the JMX Check documentation for a list of configuration options usable by all JMX-based checks.
List of metrics

The conf parameter is a list of metrics to be collected by the integration. Only two keys are allowed:

include (mandatory): A dictionary of filters. Any attribute that matches these filters is collected unless it also matches the exclude filters (see below).
exclude (optional): A dictionary of filters. Attributes that match these filters are not collected.

For a given bean, metrics get tagged in the following manner:

mydomain:attr0=val0,attr1=val1

In this example, your metric is mydomain (or some variation depending on the attribute inside the bean) and has the tags attr0:val0, attr1:val1, and domain:mydomain.

If you specify an alias in an include key that is formatted as camel case, it is converted to snake case. For example, MyMetricName is shown in Datadog as my_metric_name.
The attribute filter

The attribute filter can accept two types of values:

A dictionary whose keys are attributes names (see below). For this case, you can specify an alias for the metric that becomes the metric name in Datadog. You can also specify the metric type as a gauge or counter. If you choose counter, a rate per second is computed for the metric.

conf:
  - include:
    attribute:
      maxThreads:
        alias: tomcat.threads.max
        metric_type: gauge
      currentThreadCount:
        alias: tomcat.threads.count
        metric_type: gauge
      bytesReceived:
        alias: tomcat.bytes_rcvd
        metric_type: counter

A list of attributes names (see below). For this case, the metric type is a gauge, and the metric name is jmx.\[DOMAIN_NAME].\[ATTRIBUTE_NAME].

conf:
  - include:
    domain: org.apache.cassandra.db
    attribute:
      - BloomFilterDiskSpaceUsed
      - BloomFilterFalsePositives
      - BloomFilterFalseRatio
      - Capacity
      - CompressionRatio
      - CompletedTasks
      - ExceptionCount
      - Hits
      - RecentHitRate

Older versions

List of filters is only supported in Datadog Agent > 5.3.0. If you are using an older version, use singletons and multiple include statements instead.

Datadog Agent > 5.3.0

conf:
- include:
domain: domain_name
bean:
- first_bean_name
- second_bean_name

Older Datadog Agent versions

conf:
- include:
domain: domain_name
bean: first_bean_name
- include:
domain: domain_name
bean: second_bean_name

Log collection

To submit logs to Datadog, Tomcat uses the log4j logger. For versions of Tomcat before 8.0, log4j is configured by default. For Tomcat 8.0+, you must configure Tomcat to use log4j, see Using Log4j. In the first step of those instructions, edit the log4j.properties file in the $CATALINA_BASE/lib directory as follows:

  log4j.rootLogger = INFO, CATALINA

  # Define all the appenders
  log4j.appender.CATALINA = org.apache.log4j.DailyRollingFileAppender
  log4j.appender.CATALINA.File = /var/log/tomcat/catalina.log
  log4j.appender.CATALINA.Append = true

  # Roll-over the log once per day
  log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout
  log4j.appender.CATALINA.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

  log4j.appender.LOCALHOST = org.apache.log4j.DailyRollingFileAppender
  log4j.appender.LOCALHOST.File = /var/log/tomcat/localhost.log
  log4j.appender.LOCALHOST.Append = true
  log4j.appender.LOCALHOST.layout = org.apache.log4j.PatternLayout
  log4j.appender.LOCALHOST.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

  log4j.appender.MANAGER = org.apache.log4j.DailyRollingFileAppender
  log4j.appender.MANAGER.File = /var/log/tomcat/manager.log
  log4j.appender.MANAGER.Append = true
  log4j.appender.MANAGER.layout = org.apache.log4j.PatternLayout
  log4j.appender.MANAGER.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

  log4j.appender.HOST-MANAGER = org.apache.log4j.DailyRollingFileAppender
  log4j.appender.HOST-MANAGER.File = /var/log/tomcat/host-manager.log
  log4j.appender.HOST-MANAGER.Append = true
  log4j.appender.HOST-MANAGER.layout = org.apache.log4j.PatternLayout
  log4j.appender.HOST-MANAGER.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

  log4j.appender.CONSOLE = org.apache.log4j.ConsoleAppender
  log4j.appender.CONSOLE.layout = org.apache.log4j.PatternLayout
  log4j.appender.CONSOLE.layout.ConversionPattern = %d{yyyy-MM-dd HH:mm:ss} %-5p [%t] %c{1}:%L - %m%n

  # Configure which loggers log to which appenders
  log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost] = INFO, LOCALHOST
  log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] =\
    INFO, MANAGER
  log4j.logger.org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host-manager] =\
    INFO, HOST-MANAGER

Then follow the remaining steps in the Tomcat docs for configuring log4j.

By default, Datadog's integration pipeline support the following conversion patterns:

  %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
  %d [%t] %-5p %c - %m%n

Clone and edit the integration pipeline if you have a different format. See Logging in Tomcat for details on Tomcat logging capabilities.

Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

logs_enabled: true

Add this configuration block to your tomcat.d/conf.yaml file to start collecting your Tomcat Logs:

logs:
  - type: file
    path: /var/log/tomcat/*.log
    source: tomcat
    service: "<SERVICE>"
    #To handle multi line that starts with yyyy-mm-dd use the following pattern
    #log_processing_rules:
    #  - type: multi_line
    #    name: log_start_with_date
    #    pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])

Change the path and service parameter values and configure them for your environment. See the sample tomcat.yaml for all available configuration options.

Restart the Agent.

Containerized

For containerized environments, see the Autodiscovery with JMX guide.
Validation

Run the Agent's status subcommand and look for tomcat under the Checks section.

Write it down a step to collect docker metrices to datadog

Make sure that Docker is installed and running on your server.

Add the user running the Agent to Docker's group.

usermod -a -G docker dd-agent

Configure the Agent to connect to Docker.
Edit conf.d/docker.d/docker_daemon.yaml :

init_config:

instances:
    - url: "unix://var/run/docker.sock"

Restart the Agent
Execute the Agent status command and verify that the integration check has passed. Look for docker under the Checks section.

Write it down a step to collect mysql metrices to datadog

To install Database Monitoring for MySQL, select your hosting solution in the Database Monitoring documentation for instructions.

Proceed with the following steps in this guide only if you are installing the standard integration alone.

On each MySQL server, create a database user for the Datadog Agent.

The following instructions grant the Agent permission to login from any host using datadog@'%'. You can restrict the datadog user to be allowed to login only from localhost by using datadog@'localhost'. See MySQL Adding Accounts, Assigning Privileges, and Dropping Accounts for more info.

Create the datadog user with the following command:

mysql> CREATE USER 'datadog'@'%' IDENTIFIED BY '';
Query OK, 0 rows affected (0.00 sec)

Verify the user was created successfully using the following commands - replace with the password you created above:

mysql -u datadog --password= -e "show status" | \
grep Uptime && echo -e "\033[0;32mMySQL user - OK\033[0m" || \
echo -e "\033[0;31mCannot connect to MySQL\033[0m"

The Agent needs a few privileges to collect metrics. Grant the datadog user only the following limited privileges.

For MySQL versions 5.6 and 5.7, grant replication client and set max_user_connections with the following command:

mysql> GRANT REPLICATION CLIENT ON . TO 'datadog'@'%' WITH MAX_USER_CONNECTIONS 5;
Query OK, 0 rows affected, 1 warning (0.00 sec)

For MySQL 8.0 or greater, grant replication client and set max_user_connections with the following commands:

mysql> GRANT REPLICATION CLIENT ON . TO 'datadog'@'%'
Query OK, 0 rows affected (0.00 sec)
mysql> ALTER USER 'datadog'@'%' WITH MAX_USER_CONNECTIONS 5;
Query OK, 0 rows affected (0.00 sec)

Grant the datadog user the process privilege:

mysql> GRANT PROCESS ON . TO 'datadog'@'%';
Query OK, 0 rows affected (0.00 sec)

Verify the replication client. Replace with the password you created above:

mysql -u datadog --password= -e "show slave status" && \
echo -e "\033[0;32mMySQL grant - OK\033[0m" || \
echo -e "\033[0;31mMissing REPLICATION CLIENT grant\033[0m"

If enabled, metrics can be collected from the performance_schema database by granting an additional privilege:

mysql> show databases like 'performance_schema';
+-------------------------------+
| Database (performance_schema) |
+-------------------------------+
| performance_schema |
+-------------------------------+
1 row in set (0.00 sec)

mysql> GRANT SELECT ON performance_schema.* TO 'datadog'@'%';
Query OK, 0 rows affected (0.00 sec)

Configuration

Follow the instructions below to configure this check for an Agent running on a host. For containerized environments, see the Docker, Kubernetes, or ECS sections.
Host

To configure this check for an Agent running on a host:

Edit the mysql.d/conf.yaml file, in the conf.d/ folder at the root of your Agent's configuration directory to start collecting your MySQL metrics and logs. See the sample mysql.d/conf.yaml for all available configuration options.
Metric collection

Add this configuration block to your mysql.d/conf.yaml to collect your MySQL metrics:

init_config:

instances:
  - host: 127.0.0.1
    username: datadog
    password: "<YOUR_CHOSEN_PASSWORD>" # from the CREATE USER step earlier
    port: "<YOUR_MYSQL_PORT>" # e.g. 3306
    options:
      replication: false
      galera_cluster: true
      extra_status_metrics: true
      extra_innodb_metrics: true
      extra_performance_metrics: true
      schema_size_metrics: false
      disable_innodb_metrics: false

Note: Wrap your password in single quotes in case a special character is present.

To collect extra_performance_metrics, your MySQL server must have performance_schema enabled - otherwise set extra_performance_metrics to false. For more information on performance_schema, see MySQL Performance Schema Quick Start.

Note: The datadog user should be set up in the MySQL integration configuration as host: 127.0.0.1 instead of localhost. Alternatively, you may also use sock.

See the sample mysql.yaml for all available configuration options, including those for custom metrics.

Restart the Agent to start sending MySQL metrics to Datadog.
Log collection

Available for Agent versions >6.0

By default MySQL logs everything in /var/log/syslog which requires root access to read. To make the logs more accessible, follow these steps:

    Edit /etc/mysql/conf.d/mysqld_safe_syslog.cnf and remove or comment the lines.

    Edit /etc/mysql/my.cnf and add following lines to enable general, error, and slow query logs:

      [mysqld_safe]
      log_error = /var/log/mysql/mysql_error.log

      [mysqld]
      general_log = on
      general_log_file = /var/log/mysql/mysql.log
      log_error = /var/log/mysql/mysql_error.log
      slow_query_log = on
      slow_query_log_file = /var/log/mysql/mysql_slow.log
      long_query_time = 2

    Save the file and restart MySQL using following commands: service mysql restart

    Make sure the Agent has read access on the /var/log/mysql directory and all of the files within. Double-check your logrotate configuration to make sure those files are taken into account and that the permissions are correctly set there as well.

    In /etc/logrotate.d/mysql-server there should be something similar to:

      /var/log/mysql.log /var/log/mysql/mysql.log /var/log/mysql/mysql_slow.log {
              daily
              rotate 7
              missingok
              create 644 mysql adm
              Compress
      }

Collecting logs is disabled by default in the Datadog Agent, enable it in your datadog.yaml file:

logs_enabled: true

Add this configuration block to your mysql.d/conf.yaml file to start collecting your MySQL logs:

logs:
  - type: file
    path: "<ERROR_LOG_FILE_PATH>"
    source: mysql
    service: "<SERVICE_NAME>"

  - type: file
    path: "<SLOW_QUERY_LOG_FILE_PATH>"
    source: mysql
    service: "<SERVICE_NAME>"
    log_processing_rules:
      - type: multi_line
        name: new_slow_query_log_entry
        pattern: "# Time:"
        # If mysqld was started with `--log-short-format`, use:
        # pattern: "# Query_time:"
        # If using mysql version <5.7, use the following rules instead:
        # - type: multi_line
        #   name: new_slow_query_log_entry
        #   pattern: "# Time|# User@Host"
        # - type: exclude_at_match
        #   name: exclude_timestamp_only_line
        #   pattern: "# Time:"

  - type: file
    path: "<GENERAL_LOG_FILE_PATH>"
    source: mysql
    service: "<SERVICE_NAME>"
    # For multiline logs, if they start by the date with the format yyyy-mm-dd uncomment the following processing rule
    # log_processing_rules:
    #   - type: multi_line
    #     name: new_log_start_with_date
    #     pattern: \d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01])
    # If the logs start with a date with the format yymmdd but include a timestamp with each new second, rather than with each log, uncomment the following processing rule
    # log_processing_rules:
    #   - type: multi_line
    #     name: new_logs_do_not_always_start_with_timestamp
    #     pattern: \t\t\s*\d+\s+|\d{6}\s+\d{,2}:\d{2}:\d{2}\t\s*\d+\s+

See the sample mysql.yaml for all available configuration options, including those for custom metrics.

Restart the Agent.

Docker

To configure this check for an Agent running on a container:
Metric collection

Set Autodiscovery Integration Templates as Docker labels on your application container:

LABEL "com.datadoghq.ad.check_names"='["mysql"]'
LABEL "com.datadoghq.ad.init_configs"='[{}]'
LABEL "com.datadoghq.ad.instances"='[{"server": "%%host%%", "username": "datadog","password": ""}]'

See Autodiscovery template variables for details on using as an environment variable instead of a label.
Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Docker Log Collection.

Then, set Log Integrations as Docker labels:

LABEL "com.datadoghq.ad.logs"='[{"source":"mysql","service":"mysql"}]'

Kubernetes

To configure this check for an Agent running on Kubernetes:
Metric collection

Set Autodiscovery Integrations Templates as pod annotations on your application container. Alternatively, you can configure templates with a file, configmap, or key-value store.

Annotations v1 (for Datadog Agent < v7.36)

apiVersion: v1
kind: Pod
metadata:
name: mysql
annotations:
ad.datadoghq.com/mysql.check_names: '["mysql"]'
ad.datadoghq.com/mysql.init_configs: '[{}]'
ad.datadoghq.com/mysql.instances: |
[
{
"server": "%%host%%",
"username": "datadog",
"password": ""
}
]
labels:
name: mysql
spec:
containers:
- name: mysql

Annotations v2 (for Datadog Agent v7.36+)

apiVersion: v1
kind: Pod
metadata:
name: mysql
annotations:
ad.datadoghq.com/mysql.checks: |
{
"mysql": {
"instances": [
{
"server": "%%host%%",
"username": "datadog",
"password": ""
}
]
}
}
labels:
name: mysql
spec:
containers:
- name: mysql

See Autodiscovery template variables for details on using as an environment variable instead of a label.
Log collection

Collecting logs is disabled by default in the Datadog Agent. To enable it, see Kubernetes Log Collection.

Then, set Log Integrations as pod annotations. Alternatively, you can configure this with a file, configmap, or key-value store.

Annotations v1/v2

apiVersion: v1
kind: Pod
metadata:
name: mysql
annotations:
ad.datadoghq.com/mysql.logs: '[{"source": "mysql", "service": "mysql"}]'
labels:
name: mysql

ECS

To configure this check for an Agent running on ECS:
Metric collection

Set Autodiscovery Integrations Templates as Docker labels on your application container:

{
"containerDefinitions": [{
"name": "mysql",
"image": "mysql:latest",
"dockerLabels": {
"com.datadoghq.ad.check_names": "[\"mysql\"]",
"com.datadoghq.ad.init_configs": "[{}]",
"com.datadoghq.ad.instances": "[{\"server\": \"%%host%%\", \"username\": \"datadog\",\"password\": \"\"}]"
}
}]
}

See Autodiscovery template variables for details on using as an environment variable instead of a label.
Log collection

Available for Agent versions >6.0

Collecting logs is disabled by default in the Datadog Agent. To enable it, see ECS Log Collection.

Then, set Log Integrations as Docker labels:

{
"containerDefinitions": [{
"name": "mysql",
"image": "mysql:latest",
"dockerLabels": {
"com.datadoghq.ad.logs": "[{\"source\":\"mysql\",\"service\":\"mysql\"}]"
}
}]
}

Validation

Run the Agent's status subcommand and look for mysql under the Checks section.