epilot's Journey: from Redshift and InfluxDB to ClickHouse Cloud

epilot is a SaaS platform specializing in the energy sector, offering comprehensive solutions to help utilities and grid operators set up and manage their businesses online. It is the 360° platform for the energy transition. epilot allows its users to digitalize their sales, service & grid processes from initial customer inquiries to ongoing support for more efficient processes, flexible market launches and happier customers.

As epilot scaled, it faced challenges with its existing data infrastructure. The company initially used AWS Redshift and S3 for data storage and processing. However, these tools presented limitations in performance and real-time analytics, crucial for epilot’s needs in providing a seamless and responsive platform for its clients.

One of the significant pain points was the lack of a centralized data platform, which made it difficult for different teams within epilot to access and analyze data cohesively. Easy data access to allow all teams to build products on top of the data was a key goal. Additionally, the costs associated with Redshift were becoming a concern, especially as the company anticipated increasing data volumes with the expansion of new features and use cases.

To address these challenges, epilot explored alternative database solutions, eventually choosing ClickHouse as their central data storage layer. Suresh Sivasankaran, Principal Engineer at epilot, was familiar with ClickHouse from previous projects and recognized its potential to meet their needs. ClickHouse Cloud on AWS was particularly appealing due to epilot’s preference for a serverless infrastructure aligning with their existing tech stack.

The switch to ClickHouse offered several advantages:

Performance and Scalability : ClickHouse’s ability to handle large volumes of data with low latency was a game-changer. It allowed epilot to provide real-time analytics and insights, essential for their clients in the energy sector who need up-to-date information to make informed decisions.

Cost Efficiency : Compared to other solutions like Redshift and Rockset, ClickHouse offered a more cost-effective option. This was particularly important for epilot, which aimed to balance performance needs with budget considerations.

Flexibility and Integration : ClickHouse’s support for complex queries and integration capabilities with BI tools enabled epilot to create a more unified data environment. This integration was crucial for generating dashboards and reports that could be easily accessed and used by various teams within the company.

The implementation of ClickHouse at epilot began with a proof of concept, which proved successful, leading to a broader rollout. Initially, the focus was on internal dashboards and metrics, but the use of ClickHouse quickly expanded to other areas, including a data lake feature that allows customers to connect their BI tools and build custom reports.

One of the immediate benefits was the improvement in query performance and the ability to handle real-time analytics. This improvement was particularly evident in new features like an end-customer facing portal, which generated substantial data points. With ClickHouse, epilot could manage and analyze these data sets efficiently, enabling better service delivery and customer insights.

Suresh said:

"With the integration of ClickHouse, our dashboards now provide real-time metrics instead of relying on cached data and on-demand updates. Additionally, the data load speed for our data lake offering, used by our customers, has significantly improved, reducing the time to transfer data from ClickHouse to BI tools from over 30 minutes to under 3 minutes."

Moreover, ClickHouse’s capabilities allowed epilot to consolidate data from different sources, providing a central data foundation for the entire platform. This consolidation made it easier for teams to collaborate and innovate, leveraging data to enhance product offerings and optimize internal processes and effectively build products faster on top of the data foundation.

Looking ahead, epilot plans to expand its use of ClickHouse further. As they onboard more customers and develop new features, the company expects data volumes to grow significantly. ClickHouse’s scalability and performance will be critical in supporting this growth, enabling epilot to continue delivering high-quality, data-driven services.

One exciting area of development is using ClickHouse in conjunction with AWS Bedrock for AI and machine learning applications. This integration will allow epilot to refine its models based on data stored in ClickHouse, enhancing features like automated summaries and customer engagement tools.