Data Lake - Custom Source monitor

The Custom Source option in Data Lake allows you to push data from your applications using APIs or SDKs. This enables seamless monitoring and analysis, giving you complete control over your custom metrics.

Get started

Follow the below steps to push data via Custom Source

1. Click Integrate Now next to Custom Source to begin.
2. In the Add Data Lake Monitor page, provide the following details:
   • Data Lake Name: Provide a unique name for your monitor.
   • Associate with Monitor Group(s): Choose a monitor group from the drop-down list to logically group your monitors. Refer to Monitor Groups for more details on creating groups.
   • Include Tags: Select how you want to filter the data pushed via SDK/API.
     • All Tags: Push all available data.
     • Starts With: Push data that starts with a specific term.
     • Contains: Push data containing a specific keyword.
     • Ends With: Push data ending with a specific term.
3. Click Save.
   • You will be redirected to the dashboard, where you can see the newly created monitor.
   • You are all set. You must now push data to the newly added monitor.

Import data to the created monitor

You can push data via SDK or API.

Push data via SDK

This option is used to send raw data (unaggregated data), and the aggregation will be done at our end.

Step 1: Add/Set up the SDK to your application

1. Go to the Maven Central Repository.
2. Add the dependency into your application based on your build framework. 
   
   Note

   You can also download and add the SDK jar to your application.

Step 2: Create the object of SDK MetricProvider

Create a MetricProvider object with the Site24x7 exporter using the MetricProviderBuilder class. This object is used to push metric data.

// Create Metric provider using the following code snippet
MetricProvider provider = new MetricProvider("");
DataLakeProvider dataLake = provider.createDataLake(""); // The app key associated with the Data-lake monitor.
DataLakeResource resource = dataLake.createResource(new HashMap(){{
put("host", "192.168.10.254");
}});// Set of tags to associate with a list of metrics created using the appkey resource

// Push data using different methods like Count, Rate, Gauge, Histogram.
resource.gauge(
"ResponseTime", // The metric name (field name)
180, // The numerical value of the metric
"ms",// Unit of the metric (Default unit will be Count)
new HashMap(){{
	put("host", "192.168.10.254");
}}); // Set of tags to be associated with the metric

Step 3: Provide raw data to the SDK

Specify the accurate method to properly aggregate the values of the defined field.

Count

metricProvider.count(
    "<app key>", // The app key associated with the Data-lake monitor
    "db_write", // The metric name (field name)
    2.0, // Numerical value of the metric
    Collections.singletonMap("host", "192.168.10.254") // Set of tags to associate with the metric
);

Rate

metricProvider.rate(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

Gauge

metricProvider.gauge(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

Histogram

metricProvider.histogram(
 "<app key>", 
 "disk_free",
  16868264.0,
  Collections.singletonMap("host", "192.168.10.254")
);

Where,

• <app_key> can be obtained from the Show Post URL option
• disk_free is the numeric field
• Collections.singletonMap ("host", "192.168.10.254") is the String field, which is optional
• The default values for units are as follows:
  • ms for Time
  • bytes for Size
  • one for Count

Push data via API

This option is used to push the aggregated data.

1. Go to the data lake monitor listed and click on it.
2. Hover over the hamburger icon () next to the monitor's name.
3. Select Show Post URL from the drop-down menu.
   
   
4. Follow the on-screen instructions.
   
   Note

   Sample Payload:
   

   [
      {
         "metrics":[
            {
               "unit":"ms",
               "datavalues":[
                  {
                     "p0":0,
                     "p99":16,
                     "p100":17,
                     "time_stamp":"1739341985265",
                     "count":10,
                     "p50":12,
                     "value":100,
                     "p95":15,
                     "tags":{
                        "metric_tag_2":"metricTag2Data",
                        "metric_tag_1":"metricTag1Data"
                     }
                  }
               ],
               "name":"MetricName",
               "type":"rate",
               "description ":"Sample Description"
            }
         ],
         "tags":{
            "group_tag_1":"groupTag1data",
            "group_tag_2":"groupTag2data"
         }
      }
   ]

   Where,

   • value: Processed value of the metric.
   • count: Number of metrics recorded within the interval used for value calculation. If not provided, the default value is 1.

Supported units

Related articles

Add a Data Lake monitor

Add a REST API Metric monitor

Performance metrics of Data Lake

Data Lake dashboard

Add a Custom Attribute monitor