How to ingest LimaCharlie output into Datadog

Integrating LimaCharlie with Datadog increases visibility for LimaCharlie users. In this article, we will look at two ways you can configure the integration to help security teams streamline workflows. 

What is Datadog?

Datadog is a cloud-based monitoring and analytics platform that provides full-stack visibility and infrastructure management for organizations of all sizes. It integrates with various technologies, including cloud platforms, servers, databases, applications, and more, to collect and analyze data from various sources and provide real-time insights and metrics. With Datadog, organizations can monitor their entire stack and infrastructure, troubleshoot issues quickly, and optimize performance and capacity.

The benefits of using Datadog

• Full-Stack Visibility: Datadog provides a unified view of all the components of an organization's technology stack, making it easier to identify and resolve issues.

• Real-Time Insights: Datadog provides real-time metrics and analytics, so users can identify and respond to issues as they arise.

• Improved Collaboration: Datadog's platform allows teams to work together more effectively by providing a single source of truth and enabling cross-functional collaboration.

• Enhanced Troubleshooting: With Datadog's monitoring and logging capabilities, users can quickly diagnose and resolve issues, reducing downtime and improving system performance.

• Optimized Performance: Datadog's performance metrics and analytics help users optimize their technology stack for maximum efficiency and capacity.

• Compliance: Datadog helps organizations meet regulatory requirements by providing comprehensive logs and alerts for auditing purposes.

• Integration: Datadog integrates with a wide range of technologies, enabling users to seamlessly monitor their entire stack from a single platform.

Datadog + LimaCharlie

With all relevant data in one place, integrating LimaCharlie and Datadog can help streamline the workflow for security teams giving them the ability to leverage functionalities such as pattern recognition, anomaly detection, etc.

For example, anomaly detection, as an algorithmic feature, can identify when a metric is behaving differently than it has in the past, taking into account trends, seasonal day-of-week, and time-of-day patterns (stateful realm).

Leveraging LimaCharlie-GCP output for Datadog integration

Datadog has comprehensive documentation on how to perform log collection - integrating Google Cloud Platform (GCP) which you can follow along here: Datadog documentation. 

As a direct consequence, it is natively possible for Datadog to ingest LimaCharlie telemetry when configured with the proper Google Cloud output.

GCP gives users a lot of interesting features, but in the specific case of LimaCharlie telemetry, it is not solving an important problem: cost.

De-facto, three platforms are part of what we call the "long path" to integration:

• LimaCharlie

• Google Cloud

• Datadog

As an alternative, we are documenting a simpler way to send data in a “cloud-to-cloud” fashion, from LimaCharlie to Datadog, without the need of an additional databank—considering that both LimaCharlie and Datadog include data retention (LimaCharlie includes one year of data retention at no additional cost).

Integrating LimaCharlie and Datadog with GCP

This procedure is pretty standard for those who are familiar with the Google Cloud Platform. In fact, GCP logs are collected via Stackdriver, sent to a Cloud Pub/Sub, and then to Datadog with a HTTP Push forwarder.

Here's a generic way to send LimaCharlie Google Cloud outputs to Datadog:

• Configure a proper output in the LimaCharlie app:

• Create a Subscription on Google Cloud:

• Create a Sink on Google Cloud:

• Check for telemetry flowing correctly using Google Cloud Logs Explorer:

• Check for telemetry flowing correctly into Datadog platform:

• The LimaCharlie → GCP → Datadog integration is complete:

Integrating LimaCharlie and DataDog directly

• Retrieve (copy) the URL containing the intake datacenter and the assigned API key from the Datadog cloud configuration tab:

• Use LimaCharlie webhook_bulk output for events and webhook for all the other cases:

• Configure LimaCharlie output stream using (paste) the URL previously retrieved into the destination host field:

• Save the new output on the LimaCharlie platform and done!

Telemetry will be flowing from LimaCharlie to Datadog in real time in a flawless way.

In the workbench we used for this test, we wanted to compare the real-time performance using both the “long” and the “short” path, and both solutions worked in a solid and smooth way with nearly zero delay between events being sent out and then represented into the several Datadog dashboards.

Learning more about LimaCharlie

LimaCharlie has an active community Slack channel and holds weekly office hours every Friday at 9:00 AM PT. These are great places to drop by and ask a question, get some help, or even request a new feature. 

To see the Datadog integration discussed in this post in action, try LimaCharlie for free or book a demo today.