Open-sourcing our StatsD plugin for RedisTimeSeries

Hashedin

Samrudh Shetty, Supriya GM and Nishant George

08 May 2020

“If you don’t collect any metrics, you’re flying blind. If you collect and focus on too many, they may be obstructing your field of view.”

Scott M. Graffius

 

Collecting metrics has become an important goal for any agile focused company, it allows us to detect problems and opportunities for improvement as they arise. At HashedIn we run applications both big and small. Our experience with Redis has been phenomenal, and we use it across multiple projects. 

Before venturing into the plugin let us look into StatsD for the uninitiated. StatsD is an Open source daemon for collecting and aggregating data that can be sent to the backend. Traditionally, this backend is a Graphite server that uses a file-based fixed-size database called Whisper. Although there are many advantages using the default set of tools with statsD, we thought moving it to Redis would give us many advantages inherent to it. 

RedisTimeSeries is the module for Redis which turns it into a time-series database, with StatsD as our data collection tool that sends time series data, RTS Redis forms the perfect database for storing real-time metrics.

 

 

Why you should keep your metrics in RedisTimeSeries?

  • Redis usage is exploding everywhere with the introduction of modules such as RedisSearch and RedisML, you can do many things right in Redis.
  • The in-Memory database is very performant and fast.
  • Built-in features such as retention, downsampling, and aggregations.
  • Integrated with visualization tools such as RedisInsight.

Since the advantages of using Redis are huge, we jotted down our own plugin for StatsD, which can now directly add data into RedisTimeSeries. The inserts are pipelined for improved performance. Overall, we think Redis is the tool of the future for collecting time series data and analyzing it in real-time using other services.

 

 

A small tutorial showing how easy it is to get set up with the StatsD-Redis time series.

 

1. Install StatsD-redistimeseries backend in your StatsD’s server directory.

npm i statsd-redistimeseries

2. Add configuration keys to StatsD config, for example in config.js


{
        redisHost : “redis-host”,
        redisPort : 6379,
        authpassword:'password',
        tlsConnection:false,
        retention:500000,
        backends:[ "statsd-redistimeseries" ], 
}

You can specify multiple backends of your choice, TLS support can be enabled by setting tlsConnection to true.   Do note Redis 5 does not support native TLS so you would need to use a TLS proxy such as stunnel for this. For details of configurable options refer to the README in the Github repository.

 3. Run node statsd config.js to start the StatsD server with your custom configuration. Now StatsD server is set to receive metrics from a StatsD client which it aggregates and periodically flushes into the backend, which is RedisTimeSeries in our case. The backend performs the necessary tasks and contains the code to communicate with the Redis instance. By default, StatsD flushes to the backend every 10 seconds, and the inserts are pipelined for increased performance.

 

It is as easy as that!. To learn more about StatsD or Redis visit the following links.

 

  1. Redis Documentation – https://redis.io/documentation
  2. RedisTimeSeries – redistimeseries.io
  3. StatsD – https://github.com/statsd/statsd
  4. Statsd-redistimeseries – https://github.com/hashedin/statsd-redistimeseries-backend
  5. RedisInsight – RedisInsight: Monitoring Redis
  6. RedisTimeSeries – https://redislabs.com/redis-enterprise/redis-time-series/

Have a question?

Need Technology advice?

Connect

+1 669 253 9011

contact@hashedin.com

linkedIn youtube