I tried Slack integration with Datadog for alert notifications and testing the /datadog command

Hello. I'm Shiina from the Operations Department.

Introduction

Are you struggling with missed Datadog alert emails, wondering if an alert has been resolved, or if someone is already working on it?
By integrating Datadog with Slack, you can receive real-time alerts directly in the channels you use daily.
Utilizing Slack makes sharing incident response status and team communication significantly smoother.
This time, I tried setting up the Datadog-Slack integration and testing the useful /datadog command.

Prerequisites

• Slack workspace and notification channel already created
• Monitor already set up

Slack Integration Setup

Here's how to set up the integration between Slack and Datadog.

1. Integration Setup

Install the Slack application.

1. Select "Integrations" from the Datadog console.

2. Enter "Slack" in the search field and search.

3. Select "Install" from the Slack panel in the list.
   

4. From the Configure tab, select "Add a workspace".
   

5. You'll be automatically redirected to the Slack permission page. Confirm the workspace name you want to connect to and select "Authorize".
   

6. When the Slack workspace integration is successful, the workspace name will appear in the Configure list.

7. You can confirm that the Datadog application appears in the Slack workspace Apps.
   

2. Datadog Connection Setup

Once connected, you can use the /datadog command from the channel to access functions such as creating Datadog incidents, sharing dashboards, and executing workflows.

1. Select the connected workspace name from the Configure tab.

2. Select "Connect".
   

3. When the Slack account connection confirmation appears, select "Sign in with Slack".
   

4. You'll be automatically redirected to the Slack approval page, select "Accept and continue".
   

5. If the target workspace name is displayed in Connected Workspaces, the connection is successful.
   

3. Channel Notification Setup

Configure the Slack channel to receive Datadog notifications.
The setup method differs between public and private channels.

Public Channel
Configure from the Datadog console.

1. Select the connected workspace name from the Configure tab, then select "Add Channel".
   

2. When a new input field appears, select the desired channel name from the CHANNEL dropdown menu.

3. Select the checkmark to save the settings.
   

Private Channel
As you cannot configure from the Datadog console, use the Slack command to set up notification reception.

1. Run the /invite @Datadog command in the Slack private channel.
   

2. Confirm that it has been added.
   

Notification Test
Test if Datadog alert notifications are properly received.

1. Select the connected workspace name from the Configure tab.

2. For the target channel, select "Test" which appears when moving the cursor to the right side of MONITOR ACTIONS.
   

3. Confirm that a Test Notification message appears in the Slack channel.
   

4. Monitor Notification Setup

Specify the Slack integration as the monitor's notification recipient to send notifications to the channel.

1. Select "Monitoring List" from Monitoring in the Datadog console.
   

2. Move your cursor to the monitor name you want to send notifications for, and select "Edit" that appears.
   

3. Enter @slack in the Configure notifications & automations field.

4. Select the appropriate one from the list that appears in the format @slack-[workspace name]-[channel name].

5. Select "Save and Publish" to save.
   

Alert Notifications

Let's artificially trigger a Datadog monitor alert and see the notification in the Slack channel.
For this, we used a message template with variables in the monitor's Configure notifications & automations.
Using variables allows you to add graphs, detection messages, links, etc., to the notification, enabling smoother incident response.

Metrics Monitor

• Message template example

@slack-[workspace name]-[channel name]
High CPU Usage Detected

Host: {{host.name}} 
Current CPU Usage: {{value}}% 
Threshold: {{threshold}}%

{{#snapshot}} avg:aws.ec2.cpuutilization{*} by {host} {{/snapshot}}

When an alert is detected, a message starting with Triggered: is notified to the Slack channel.
You can see information on the host name, current usage rate, threshold value from variables, and a graph displayed by the snapshot.

When the alert is resolved, a message starting with Recovered: is notified to the Slack channel.

Log Monitor

• Message template example

@slack-[workspace name]-[channel name]

Error Log Detected

Message: {{log.message}}
Log URL: {{log.link}}

When an alert is detected, a message starting with Triggered: is notified to the Slack channel.
You can see the detection message from variables and a link to the Datadog console where you can check the log message.
For log monitors, even without specifying variables, a "View in Log Explorer" link is automatically added.

When the alert is resolved, a message starting with Recovered: is notified to the Slack channel.

Slack Commands

By installing the application, you can operate Datadog from the Slack channel using commands.

Command List

The command list can be checked with /datadog help.
As of writing this article, the following commands are available:

• Incident Response

• Workflow

• Dashboard

• Monitor

• Account Management

/datadog monitors Command

You can display a list of monitors that continue to be in alert state and have been notified to the Slack channel.
Those that have already been resolved are not displayed.
This is useful for preventing response oversights or missed alerts when multiple alerts are detected.
When executing the command, a popup window appears displaying the names of monitors in an alert state.

If there are no targets, it displays There are no triggered monitors for this channel.

/datadog workflow Command

You can call workflow execution from the Slack channel.
Workflows can be used for recovery processes and initial response use cases.
For example, when an alert occurs, you can execute a recovery script using AWS Systems Manager Run Command on an EC2 instance.

Note that when executing from Slack, the workflow must be published in advance.

/datadog dashboard Command

You can share widgets from dashboards in image format to the Slack channel.
This can be used when you want to quickly share clues or traces of the cause of an alert with your team.

Summary

We've tried setting up Slack integration, configuring monitor notifications, and operating with the /datadog command.
By integrating Slack and Datadog, you can not only receive real-time alert notifications but also complete incident management and dashboard sharing on the channel.
If you're considering optimizing your alert operations, please use this as a reference.

I hope this article was helpful.

References

https://docs.datadoghq.com/ja/integrations/slack/?tab=datadogforslack
https://docs.datadoghq.com/ja/monitors/notify/variables/?tab=is_alert