IBM was positioned furthest in Vision and highest in Execution for IBM Cloudability on the Gartner® Magic Quadrant™ for Cloud Financial Management Tools.
Read the Report
Insights from over 1,000 senior technology decision makers provide a strategic framework for maximizing the ROI of tech investments.
Get the Report
Knowing the resource usage and performance characteristics of your applications is key to their optimization and lifecycle management. Since containerized applications are becoming the norm, there is a need for a reliable and efficient way to gather usage and performance metrics from them, despite their tendency to be heavily distributed.
One way to do this effectively would be to use an agent that collects and exports data directly from the containers themselves. cAdvisor is a tool that does exactly this. During this article, we will demonstrate how to install and best utilize cAdvisor, allowing you to keep track of container resource utilization and performance.
What is cAdvisor?
cAdvisor, short for Container Advisor, is an open-source tool developed by Google to monitor containers. It can collect, aggregate, process, and export container-based metrics such as CPU and memory usage, filesystem and network statistics. You can easily use this tool in any containerized environment, ranging from a single Docker installation to a complete Kubernetes cluster. Kubecost uses cAdvisor as part of its
core architecture
.
Features of cAdvisor
cAdvisor has become one of the leading solutions for monitoring containers because of its user-friendliness, flexibility, and ability to meet almost any monitoring requirement. cAdvisor has the following features:
-
Native support for Docker containers and out-of-the-box support for other container types
-
Automatically discovers containers within its node and starts collecting data
-
Multiple implementation methods, including running as a standalone application at the OS level, a Docker container, or a DaemonSet in Kubernetes
-
Can export data to storage plugins such as Prometheus, Elasticsearch, and InfluxDB for further analysis and processing
-
Has a built-in Web UI that exposes real-time metrics from collected data
-
Ability to provide overall node resource usage by analyzing the root container
-
Powerful
REST API
to query container metrics directly from cAdvisor
Limitations of cAdvisor
While cAdvisor is a powerful and versatile tool, there are some limitations that you should be aware of:
-
cAdvisor only collects basic resource utilization information and may not be sufficient if in-depth metrics are needed
-
Different OSs will require specific
configurations
to gather metrics, such as running in privileged mode for RHEL and CentOS, or enabling memory cgroups in Debian
-
Collecting metrics for custom hardware like GPUs requires additional configuration, which will differ depending on the underlying infrastructure
-
cAdvisor does not provide a method to modify runtime options after initial configuration. Users will need to redeploy the cAdvisor container with the new runtime options if this is required
-
cAdvisor requires external tooling to to store collected data long-term and to run any further analytics
How to implement cAdvisor
Now we have a better understanding of cAdvisor, let’s look at some different implementation methods.
Implement cAdvisor as a container
This is the simplest method to get cAdvisor up and running. Here, users deploy the cAdvisor image and run it as an independent container in a Docker installation.
The cAdvisor container image is available via Google
here
, and anyone can pull the image to create a running cAdvisor container. Our example below uses Docker 20.10.14 in an Ubuntu 20.04.1 environment.
Run the following command to create a cAdvisor container in Docker:
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
gcr.io/cadvisor/cadvisor:v0.39.3
<![CDATA[
wget https://github.com/google/cadvisor/releases/download/v0.39.3/cadvisor
chmod +x cadvisor
sudo ./cadvisor
<p>As before, users can access the cAdvisor interface via port 8080.</p><p><b>Note:</b> in this case a cAdvisor container will not exist as it is running as a standalone application outside of Docker.</p><p>When running as a standalone application, cAdvisor will typically need root privileges. This is because accessing certain system data requires elevated permissions. Usefully, cAdvisor will gracefully degrade its features to match the available data sources that it’s allowed to access.</p><h3>Implement cAdvisor in Kubernetes</h3><p>cAdvisor can also be deployed in Kubernetes. A typical Kubernetes cluster consists of many nodes and cAdvisor should be installed on all of them to be able to export metrics from your containers successfully.</p><p>Manually installing cAdvisor into each cluster node is typically not feasible due to scale. Instead, make use of a Kubernetes DaemonSet, which will automatically deploy an instance of a given container to every node for you.</p><h3>Installing cAdvisor as a DaemonSet</h3><p>cAdvisor <a href="https://github.com/google/cadvisor/tree/master/deploy/kubernetes" target="_blank" rel="noopener nofollow">provides support for Kustomize</a>. Kustomize allows users to easily customize and install cAdvisor as a DeamonSet in any Kubernetes cluster. Our example demonstrates this using a <a href="https://minikube.sigs.k8s.io/docs/start/" target="_blank" rel="noopener nofollow">minkube 1.25.2</a> K8s cluster and Kustomize 4.5.4.</p> </div>
<p>Download the Kustomize files from the git repository.</p> </div>
<p>That’s it, you have deployed cAdvisor as a DeamonSet using Kustomize!</p><h3>Custom cAdvisor configuration</h3><p>You don’t have to use Kustomize to deploy cAdvisor on Kubernetes. If you prefer, you can create your own DaemonSet YAML manifest instead. The following example <i>cadvisor.yml</i> will deploy cAdvisor directly to the cluster without requiring Kustomize.</p> </div>
<section><div><p><b>Note:</b> Depending on the requirement, provide the necessary runtime arguments to modify the behavior of cAdvisor.</p></div></section><section><h2>Advisor runtime options</h2><div><p>Now we know how to configure cAdvisor for different environments, let’s see how to modify its behavior. The <a href="https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#cadvisor-runtime-options" target="_blank" rel="noopener nofollow">runtime options</a> provided by cAdvisor enable users to change the defaults to meet different monitoring requirements. Here are some useful ones organized in a handy table.</p><p><i>Table 1 – cAdvisor Runtime Options</i></p></div></section> </div>
<td>--store_container_labels=false</td>
<td>stops converting container labels and environment variables</td>
<td>--docker_only=false</td>
<td>stops reporting raw cgroup metrics except for the root cgroup</td>
<td>--enable_load_reader=false</td>
<td>Enable CPU load reader</td>
<td>--profiling=true</td>
<td>enable profiling via the Web interface</td>
<td>--log_cadvisor_usage=true</td>
<td>enable cAdvisor container usage logs</td>
<td>--allow_dynamic_housekeeping=true</td>
<td>enable dynamic housekeeping interval</td>
<td>--global_housekeeping_interval</td>
<td rowspan="2">set the housekeeping interval at a global or per container level. This ensures that periodic housekeeping tasks are carried out at these specified intervals.</td>
<td>--housekeeping_interval</td>
<td>--http_auth_file=</td>
<td>authentication file for the Web UI</td>
<td>--storage_driver</td>
<td>storage driver to be used for persistent data storage</td>
<td>--disable_metrics / --enable_metrics</td>
<td>enable and disable specific metrics</td>
<h2>Export data using cAdvisor</h2><div><p>As mentioned earlier, cAdvisor is not a tool that can store data long-term or perform analytics. However, it can export collected data into different supported storage <a href="https://github.com/google/cadvisor/blob/master/docs/storage/README.md" target="_blank" rel="noopener nofollow">backends</a>, including ElasticSearch, Prometheus, BigQuery, and InfluxDB.</p><p>In this example, we will configure cAdvisor to export data to an Elasticsearch cluster. Use the following command, with the runtime options pointing to Elasticsearch, to create the cAdvisor container.</p></div> </div>
--volume=/var/run:/var/run:rw \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:ro \
--publish=8080:8080 \
--detach=true \
--name=cadvisor \
gcr.io/cadvisor/cadvisor:v0.39.3 -storage_driver=elasticsearch -storage_driver_es_host="http://10.10.10.100:9200" -storage_driver_es_type="stats"
<section><div><p>In the runtime commands, the storage_driver points to the storage plugin, the storage_driver_es_host points to the Elasticsearch cluster and the storage_driver_es_type option defines the Elasticsearch type name. See <a href="https://github.com/google/cadvisor/blob/master/docs/storage/elasticsearch.md" target="_blank" rel="noopener nofollow">here</a> for details. Users can exchange the runtime options for storage-plugin specific ones to configure different backends.</p></div></section><section><h2>Best practices when using cAdvisor</h2><div><p>cAdvisor gives you the ability to capture data from containers. However, if not properly configured and optimized, cAdvisor could potentially decrease the performance of your applications by increasing resource consumption. Here are some best practices to minimize performance problems:</p><ul><li><ul>Running the default cAdvisor configuration can cause high CPU usage. To avoid this, you can configure runtime options that specifically define the data that cAdvisor should capture, and how oftenv<li>housekeeping_interval: setting this to a value of 30 seconds ensures that periodic actions to gather container stats run at a set interval</li><li>docker_only: setting this to true means it reports raw cgroup metrics</li><li>disable_metrics: disable all unused metrics individually. This minimizes total load as it reduces the metrics that cAdvisor queries</li></ul></li><li>configure a storage plugin when implementing cAdvisor. This allows you to store the collected data from the start. It’s always recommended to use a <a href="https://github.com/google/cadvisor/blob/master/docs/storage/README.md" target="_blank" rel="noopener nofollow">storage plugin</a> when implementing cAdvisor unless in a development environment</li><li>cAdvisor requires adequate resources to function and you should ensure that there is enough headroom in the containerized environment to account for this. In a Kubernetes cluster, with cAdvisor deployed as a DemonSet, the resource requirements will depend on the number of Pods within each node and what data is being queried. The more pods that data are collected from, the more resources are required to process and export that data.</li><li>With the <a href="https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/" target="_blank" rel="noopener nofollow">introduction of the Metrics API and Metrics Server</a> in Kubernetes, most of the data collected from cAdvisor can be queried natively through Kubernetes itself. You should therefore not overlap data collected through cAdvisor and any other Kubernetes tool.</li></ul></div></section><section><h2 class="&quot;pillar-body__section-title">Conclusion</h2><div><p>cAdvisor a simple yet powerful tool, capable of extracting resource usage and performance characteristics from containers. Its support for many platforms, ranging from local Docker installations to fully-fledged orchestration platforms like Kubernetes, allows you to adapt cAdvisor to most monitoring needs. When combined with plugins that export data to tools like Prometheus and Elasticsearch, cAdvisor becomes one of the most versatile metric collection solutions available.</p></div></section> </div>
]]>
