Graphite

Question 1

whisper

Answer 1

DB library storing time-series numeric data w/ create, update, fetch

Question 2

carbon-cache

Answer 2

backend to store data points for metrics provided by clients

Question 3

front-end webapp

Answer 3

renders graphs and provides bsic UI, custom graphs via URL params

Question 4

metrics

Answer 4

unique name, with dot-separated hierarchy, e.g. servers.www01.cpuUsage

Question 5

data point

Answer 5

(timestamp, value) pair corresponding to the measured value of a metric

Question 6

Problem: Users’ dashboards w/ identical requests are pinning the CPU

Answer 6

• Render once & cache in memcached with even a few seconds expiry reduces burden significantly
• Cache underlying data so there’s less processing when a user tweaks and redraws graph

Question 7

Problem: Volume of file writes exceeds ability of storage

Answer 7

• Seek time is the major bottleneck, even for whisper files 12 bytes in size, so write files contiguously to disk in a single write
• buffering queues added to carbon based on metric name
• adds resiliency to handle temp I/O slowdowns
• queues simply grow if system must perform other I/O work (as long as there is sufficient memory to hold the queues)

Question 8

Problem:

Volume of queued data waiting for disk is noticeable to users who don’t see real-time data

Answer 8

• Added a socket listener to carbon to retrieve buffered data
• Webapp combines buffer + disk data and renders real-time graphs

Question 9

Problem:

When kernel runs out of memory, writes become synchronous, which slows writes, grows queues, and everything screams

Answer 9

• Configurable limits on carbon: how big of queues, whisper rate limits

Question 10

clustering

Answer 10

• find function was extended to be able to make remote HTTP calls to other graphite servers
• fetch for remote data also made available
• remote calls are the same as a client would call, except includes a param specifying that the operation should only be local & not redistributed throughout the cluster

Question 11

efficiency

Answer 11

memcached the results of find requests

Question 12

distribution

Answer 12

carbon-relay applies rules to determine which server to send metrics to