2017-03-30

Benchmarking Kafka Performance Part 1: Write Throughput

We have offered a fully managed Kafka service for some time now, and we are quite often asked about just how many messages can you pipe through a given service plan tier on a selected cloud. So here's a benchmark we conducted to give you a rough idea on just how well Apache Kafka performs in the public cloud.

This is the first post in a series that explores Kafka performance on multiple public cloud providers.

What is Kafka?


Apache Kafka is a high-performance open-source stream processing platform for collecting and processing large numbers of messages in real-time. It enables you to accept streaming data such as website click streams, events, transactions or other telemetry in real-time and at scale, and serve it downstream to stream processing applications.

Kafka is built distributed for both scalability as well as fault tolerance. Adding more horizontal nodes to tackle growing loads is fairly straightforward and automatic replication of the data over more than one node maintains availability when nodes fail.

The basic concepts in Kafka are producers and consumers.

A producer is an application that generates data but only to provide it to some other application.

An example of a producer application could be a web server that produces "page hits" that tell when a web page was accessed, from which IP address, what the page was and how long it took to render the page by the web server.

On the consumer side there could be multiple systems interested in the same page hit data stream:
  • A time series database that is used to plot the total number of page hits over time
  • A reporting application collecting summaries of the pages accessed and sending them to a data warehouse database system
  • A DDoS detection system trying to find abnormal access patterns
  • A rate limiting monitor counting the number of hits from a specific source address
  • And so on...
Kafka suits these kinds of applications very well: it provides a method of getting the data out of the hands of the producing application quickly and safely. Once the producer has written the message to Kafka, it can be sure that its part of the job is done. The producer application does not need to know how the data is used and by which applications, it just stores it in Kafka and moves on.

On the consumer side a powerful feature of Kafka is that it allows multiple consumers to read the same messages. In our web page hit example above, each of the consumer applications get their own read cursor to the data and they can process the messages at their own pace, all without causing any performance issues or delays for the producer application.

Here's what it roughly looks like:



The Zookeeper cluster is a critical piece in keeping Kafka healthy and up and running. It maintains Kafka's metadata and most importantly, a consensus between the Kafka nodes of who is doing what.

Aiven Kafka as a Service


Aiven Kafka is a a fully managed service based on the Apache Kafka technology. Our aim is to make it as easy as possible to use Kafka clusters with the least amount of operational effort possible. We handle the Kafka and Zookeeper setup and operations for you, so you can focus on value-adding application logic instead of infrastructure maintenance. Aiven Kafka services can be launched in minutes, and we'll ensure they remain operational, well performing, up-to-date and secure at all times. Nodes are automatically distributed evenly across the available availability zones in order to minimize the impact of losing any of the zones.

Aiven Kafka is available in Amazon Web Services, Microsoft Azure, Google Cloud Platform, UpCloud and DigitalOcean with a total coverage of 53 cloud regions. In this performance comparison we ran the benchmark on all of these except DigitalOcean, where our Kafka offering is limited by the available plans.

Each Kafka service used in these tests is a regular Aiven-provided service with no alterations to its default settings.

Benchmark Setup


In this first Kafka benchmark post, we set out to estimate maximum write throughput rates for various Aiven Kafka plan tiers in different clouds. We wanted to use a typical customer message sizes and standard tools for producing load. We also wanted to generate the load from separate systems over the network to make sure the load could mimic the actual customer workloads as closely as possible.

High-level view of the test setup, a single Aiven Kafka service with five nodes, distributed evenly over the availability zones:





We picked message size of 512 bytes for our tests. Based on our experience, one of the most typical payloads is a JSON encoded message ranging somewhere between 100 bytes to 10 kilobytes in size.

In these tests, we use a single topic with the partition count matching the node count of each Aiven plan tier. For more complex topic/partition setups Aiven actively balances the placement of the partitions, trying to achieve a "perfect" distribution of partitions. In the case of this test there is just a single partition for each node, so this is rather simple. We set the replication factor to one (1) in the case of this test, meaning each of the messages only resides on a single Kafka node.

Apache Kafka version used was 0.10.1.1.

For load generation, we chose to use librdkafka and rdkafka_performance from the provided examples. We are using default settings for the most part, but bumped up single request timeout to 60 seconds as we expect the Kafka brokers to be under extreme load and request processing to take longer than under a normal healthy load level. Also, since Aiven Kafka services are offered only over encrypted TLS connections, we included the configuration for these, namely the required certificates and keys.

librdkafka defaults to a maximum batch size of 10000 messages or to a maximun request size of one million bytes per request, whichever is met first. In these tests, we did not employ compression.

producer.props configuration:

  metadata.broker.list=target-kafka.benchmark.aivencloud.com:10947
  security.protocol=ssl
  ssl.key.location=client.key
  ssl.certificate.location=client.crt
  ssl.ca.location=ca.crt
  request.timeout.ms=60000
 
We ran several instances of rdkafka_performance on multiple VMs on a different cloud provider from the one being tested. So all of the test load was coming from the internet thru the nodes' public network interfaces.

We kept increasing the number of instances until we could find the saturation point and the maximum message rates for each plan.

Each rdkafka_performance instance was started on the command line with:

  rdkafka_performance -P -s 512 -t target-topic -X file=producer.props
 

Benchmark Results


First set of tests was run on an Aiven Kafka Business-4 plan, which is a three node cluster and a common starting point for many of our customers. Each node in this plan has 4 gigabytes of RAM, a single CPU core and 200 gigabytes of disk on each node, providing a total 600 gigabytes of raw Kafka storage capacity in the cluster.

Write performance (3 nodes @ 4 GB RAM, 1 CPU, 200 GB disk each):


On UpCloud, we hit 200,000 messages per second. Azure and Google plans saturated at 120,000 and 130,000 messages per second and the Amazon deployment reached 50,000 messages per second.
The performance is pretty respectable. The performance on Amazon is a bit behind the others because of the node types available and we will be looking at ways to optimize that in the future. As you will see in the next graph for the test with the bigger plan, the AWS performance is already more in line with the other providers. As you will see in the next graph for the test with the bigger plan, the AWS performance is already more in line with the other providers.

Next, we tested three node clusters but with larger underlying instances using the Business-8 plan. This plan has nodes with 8 gigabytes of RAM, two CPU cores and 400 gigabytes of disk per node, i.e. all the primary resources are doubled when compared to the Business-4 plan. This test indicates how well Kafka scales vertically with increased resources.

Write performance (3 nodes @ 8 GB RAM, 2 CPU, 400 GB disk each):



We see a nice increase in performance, with 320,000 messages per second on UpCloud, 205,000 on Azure, 170,000 on Google and 160,000 messages per second on AWS.

In the last test, we wanted to verify how well Kafka scales horizontally. With this test, we went from the Business plan tier to the Premium tier, which bumps the node count from three to five, while keeping the node specs otherwise identical. Also the test setup was updated to utilize a partition count of five (vs. three) for this test.

Write performance (5 nodes @ 8 GB RAM, 2 CPU, 400 GB disk each):



The results here are solid for Kafka: a two-thirds increase in the number of nodes resulted in a straight 2/3 increase in write performance. Awesome!

Aiven Kafka Premium-8 on UpCloud handled 535,000 messages per second, Azure 400,000, Google 330,000 and Amazon 280,000 messages / second.

Benchmark Conclusions


Apache Kafka performs just as well as we expected and scales nicely with added resources and increased cluster size. We welcome you to benchmark your own workloads with Aiven and to share your results.

We utilize Kafka as a message broker within Aiven as well as use it as a medium for piping all of our telemetry metrics and logs. We are happy with with our technical choice, and can recommend Apache Kafka for handling all kinds of streaming data.

Find out more about Aiven Kafka at https://aiven.io/kafka.

2017-03-16

Bigger Aiven Kafka plans and introducing Amazon VPC peering


New larger Aiven Kafka plans available


We've included larger 32GB and 64GB plan tiers to our Kafka offerings with increased core counts and larger memory. These offerings are available in Business and Premium Kafka plans in all Google Cloud Platform, Amazon Web Services, Microsoft Azure and UpCloud regions.


Plan Cluster
Nodes
CPU Memory Total Storage Data Retention
Startup-2 3 1 2 GB 90 GB 1 week
Business-4 3 1 4 GB 600 GB 4 weeks
Business-8 3 2 8 GB 1200 GB 6 weeks
Business-16 3 4 16 GB 2400 GB 8 weeks
Business-32 3 8 32 GB 4200 GB 12 weeks
Business-64 3 16 64 GB 6000 GB 18 weeks
Premium-4 5 1 4 GB 1000 GB 6 weeks
Premium-8 5 2 8 GB 2000 GB 10 weeks
Premium-16 5 4 16 GB 4000 GB 14 weeks
Premium-32 5 8 32 GB 8000 GB 20 weeks
Premium-64 5 16 64 GB 10000 GB 30 weeks

If you need further horizontal scale, please contact us for custom Kafka plans.

Schema Registry now available in Kafka plans


We've added support for Confluent's Kafka Schema Registry which allows you to store your Kafka message descriptions in a centralized registry. The feature is available in our Business and Premium level Aiven Kafka plans. It also hooks up automatically with our existing Kafka REST service.

In order to use these features you can easily enable them from the web console.





Note that if you're using these with the Java client you will want to patch your Schema Registry client with this small patch that has been suggested for inclusion into the main Schema Registry project.

AWS VPC peering available


We've launched support for Amazon Web Services Virtual Private Cloud (VPC) peering. With VPC peering, you can link Aiven services directly into your own VPC networks. You can access the services with private IP addresses as if they were in the same network.

VPC peering support is now available for all service types and Startup, Business and Premium plans in all Amazon Web Services regions.

Please contact us if you want to try out our VPC support for yourself.

Give Aiven services a whirl


Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!

Cheers,
Team Aiven

2017-03-07

Aiven Elasticsearch 5 available with easy upgrade from 2.x

We're making Elasticsearch 5 available to all our customers today, including a simple single-click upgrade path for our existing Elasticsearch 2 users.


Improved performance and usability


Elasticsearch 5 (version 5.2.2 at the moment) and its accompanying new Kibana version (also version 5.2.2) bring many useful new features from a developer console in Kibana that you can test your queries with against the REST API to in to much improved index performance in Elasticsearch.

However, perhaps the most exciting features are the performance improvements, which range from around 25-80% faster performance, depending on the usage scenario.

You can find more about the new improvements from the Elasticsearch blog, which has a separate story about each 5.0, 5.1 and 5.2. The full release notes are also available here: 5.0, 5.1, 5.2

Easy upgrade from Elasticsearch 2.x


Upgrading your existing Aiven Elasticsearch 2 service is easy and will only take a moment.

First, open your service's overview page in our web console:




We can see that the currently running version is 2.4.2 and there is an upgrade button beside it. Let's press the button!





Clicking the Upgrade button in the confirmation dialog will immediately start the upgrade. Unlike most Aiven software upgrades this upgrade is performed in-place in the running service nodes, i.e. this is not the usual rolling forward upgrade that we provide. This way we were able to squeeze down the upgrade downtime to the absolute minimum required.

NOTE: After clicking the Upgrade button it is no longer possible to downgrade back to version 2.x with the chosen service. Please test your application carefully with Elasticsearch 5 before committing to the upgrade.

Confirming the upgrade pops up a banner that stays on until the upgrade is complete. Typically the upgrade takes from seconds to a couple of minutes, depending on the number of indexes in the Elasticsearch database.

Once the upgrade is complete, the yellow banner disappears and the new version number is updated in the service information:




And we are all done with just a couple of clicks!

In order to upgrade you may either create a new service or you can upgrade your current Elasticsearch cluster to the latest version.

Try the new Aiven Elasticsearch 5 for free


Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!

Cheers,
Team Aiven

2017-02-28

Handling the AWS US-EAST-1 outage

The ongoing outage in AWS US-EAST-1 (N. Virginia) affected a number of Aiven users, but the combination of our automation and manual actions taken by the operations team has resolved the issues for all Aiven users.

Our 24/7 monitoring alerted us to the outage, initially thought to be limited to S3 but later revealed to affect more AWS resources, including EBS volumes and launching new EC2 instances.  A number of Aiven services running on affected EBS volumes in affected availability zones were migrated online to new instances in AWS US-EAST-2 (Ohio) ensuring service availability while the N. Virginia region is experiencing issues.

There was also a number of Aiven services running on EC2 instances that continued to operate normally, but where access to S3 for backups started timing out and failing.   The affected services have been updated to store backups to the Ohio region until the N. Virginia region recovers.

This ensures that all Aiven services have proper backups during the outage.  It won't be possible to spin up new instances in N. Virginia or perform a PostgreSQL point-in-time-recovery to a backup location stored only in N. Virginia while the outage is ongoing, but once AWS has resolved the issue Aiven services will be automatically restored.

Update: Please see our help site for more details about the outage.

Cheers,
Team Aiven

2017-01-31

Lower PostgreSQL pricing in AWS

We're happy to announce pricing changes for Aiven PostgreSQL plans in Amazon Web Services

The prices of Aiven PostgreSQL in AWS have been updated to take the latest AWS price cuts in account, lowering the price of various services by 10 to 30%. The highly-available business plans have seen the largest cuts in pricing.  The new prices are effective on February 1st for all current and new customers.

Please check out our new pricing for more detail and sign up for a free trial!


Cheers,
Team Aiven

2016-12-28

Aiven PostgreSQL read-only replicas

We are happy to announce that we have enabled read-only replica access to all of our PostgreSQL plans that have one or more standby nodes. Utilizing the standby server nodes for read-only queries allows you to scale your database reads by moving some of the load away from the master server node to the otherwise mostly idle replica nodes.

What are master and standby nodes?

PostgreSQL master node is the primary server node that processes SQL queries, makes the necessary changes to the database files on the disk and returns back the results to the client application.

PostgreSQL standby nodes replicate (which is why they are also called "replicas") the changes from the master node and try to maintain an up-to-date copy of the same database files that exists on the master.

Standby nodes are useful for multiple reasons:
  • Having another physical copy of the data in case of hardware/software/network failures
  • Having a standby node typically reduces the data loss window in disaster scenarios
  • Restoring the database back to operation is quick by a controlled failover in case of failures, as the standby is already installed, running and in-sync with the data
  • Standby nodes can be used for read-only queries to reduce the load on the master server

What is the difference between having 0, 1 or even 2 standby nodes?

Aiven offers PostgreSQL plans with different number of standby server nodes in each:
  • Hobbyist and Startup plans have just a single master node and no standby nodes
  • Business plans have one master node and one standby node
  • Premium plans have one master node and two standby nodes
The difference between the plans is primary the behavior during failure scenarios. The are many bad things that can happen to cloud servers (or any server in general): hardware failures, disk system crashes, network failures, power failures, software errors, running our of memory, operator mistakes, fires, floods and so on.

Single node plans are most prone to data loss during failures. For example, if the server power suddenly goes out, some of the latest database changes may not have made it out from the server into backups. The size of the data loss window is dependent on the backup method used.

Single node plans are also the slowest to recover back to operation from failures. When the server virtual machine fails, it takes time to launch a new virtual machine and to restore it from the backups. The restore time can be anything from a couple of minutes to several hours, the primary factor in it being the size of the database that needs to be restored.

Adding a "hot" standby node helps with both of the above issues: the data loss window can be much smaller as the master is streaming out the data changes in real-time to the standby as they happen. The "lag" between the master and standby is typically very low, from tens of bytes to hundreds of bytes of data.

Also recovery from failure is much faster as the standby node is already up and running and just waiting to get the signal to get promoted as the master, so that it can replace the old failed master.

What about having two standby nodes? What is the point in that?

The added value of having a second standby node is that even during recovery from (single-node) failures, there are always two copies of the data on two different nodes. If another failure strikes after a failover when there is just a single master node running, we again risk losing some of the latest changes written to the database. It takes time to rebuild a new standby node and getting it in sync node after a failover when there is a lot of data in the database, and it often makes sense to protect the data over this time period by having another replica. This is especially important when the database size is large and recreating a replacement node for the faulted one can take hours.

Using standby nodes for read-only queries


Standby nodes are also useful for distributing the load away from the master server. In Aiven the replica nodes can be accessed by using the separate "Read-only replica URL" visible in the Aiven web console:



Using the replica URL in a database client application will connect to one of the available replica server nodes. Previously replica node access was only available in our Premium plans (master + two standbys) and now we have enabled it in our Business plans (master + one standby) as well.

So if you have had high CPU usage on the master node of your Startup plan, it may be worthwhile looking into the possibility of increasing your read throughput by using the replica servers for reads. Of course in addition by using a Business plan you'll also make the service have better high availability characteristics by having a standby to fail over to.

A good thing to note is that since the PostgreSQL replication used in Aiven PostgreSQL is asynchronous there is a small replication lag involved. What this means in practice is that if you do an INSERT on the master it takes a while (usually much less than a second) for the change to be propagated to the standby and to visible there.

Replica Usage

To start using your read-replica find its database URL and after that you can connect to it by copying the Read-only replica URL:

$ psql postgres://avnadmin:foo@replica.demopg.demoprj.aivencloud.com:10546/defaultdb?sslmode=require
psql (9.6.1, server 9.6.1)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)Type "help" for help.

defaultdb=>

After which you can run any read-only query without slowing down the master.

Also while connected, PostgreSQL can tell you whether you're connected to either a master or standby node. To check that out you can run:

defaultdb=> SELECT * FROM pg_is_in_recovery();
 pg_is_in_recovery
-------------------
 t
(1 row)

If it returns TRUE you're connected to the replica, if it returns FALSE you're connected to the master server.

Try PostgreSQL 9.6 for free in Aiven

Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!

Cheers,
Team Aiven

2016-12-21

Aiven PostgreSQL connection pooling

We're happy to announce that Aiven PostgreSQL now has support for connection pools. Connection pooling allow you to maintain very large numbers of connections to a database while keeping the server resource usage low.

Aiven PostgreSQL connection pooling utilizes PGBouncer for managing the database connection and each pool can handle up to 5000 database client connections. Unlike when connecting directly to the PostgreSQL server, each client connection does not require a separate backend process on the server. PGBouncer automatically interleaves the client queries and only uses a limited number of actual backend connections, leading to lower resource usage on the server and better total performance.

Why connection pooling?


Eventually a high number of backend connections becomes a problem with PostgreSQL as the resource cost per connection is quite high due to the way PostgreSQL manages client connections. PostgreSQL creates a separate backend process for each connection and the unnecessary memory usage caused by the processes will start hurting the total throughput of the system at some point. Also, if each connection is very active, the performance can be affected by the high number of parallel executing tasks.

It makes sense to have enough connections so that each CPU core on the server has something to do (each connection can only utilize a single CPU core [1]), but a hundred connections per CPU core may be too much. All this is workload specific, but often a good number of connections to have is in the ballpark of 3-5 times the CPU core count.

[1] PostgreSQL 9.6 introduced limited parallelization support for running queries in parallel on multiple CPU cores.

Without a connection pooler the database connections are handled directly by PostgreSQL backend processes, one process per connection:





Adding a PGBouncer pooler that utilizes fewer backend connections frees up server resources for more important uses, such as disk caching:




Many frameworks and libraries (ORMs, Django, Rails, etc.) support client-side pooling, which solves much the same problem. However, when there are many distributed applications or devices accessing the same database, a client-side solution is not enough.

Connection pooling modes


Aiven PostgreSQL supports three different operational pool modes: "session", "transaction" and "statement".

  • The "session" pooling mode means that once a client connection is granted access to a PostgreSQL server-side connection, it can hold it until the client disconnects from the pooler. After this the server connection will be returned back into the connection pooler's free connection list to wait for its next client connection. Client connections will be accepted (at TCP level), but their queries will only proceed once another client disconnects and frees up its backend connection back into the pool. This mode can be helpful in some cases for providing a wait queue for incoming connections while keeping the server memory usage low, but has limited usefulness under most common scenarios due to the slow recycling of the backend connections.
  • The "transaction" pooling mode on the other hand allows each client connection to take their turn in using a backend connection for the duration of a single transaction. After the transaction is committed, the backend connection is returned back into the pool and the next waiting client connection gets to reuse the same connection immediately. In practise this provides quick response times for queries as long as the typical transaction execution times are not excessively long. This is the most commonly used PGBouncer mode and also the Aiven PostgreSQL default pooling mode.
  • The third operational pooling mode is "statement" and it is similar to the "transaction" pool mode, except that instead of allowing a full transaction to be run, it cycles the server side connections after each and every database statement (SELECT, INSERT, UPDATE, DELETE statements, etc.) Transactions containing multiple SQL statements are not allowed in this mode. This mode is sometimes used for example when running specialized sharding front-end proxies.

How to get started with Aiven PostgreSQL connection pooling


First you need an Aiven PostgreSQL service, for the purposes of this tutorial we assume you already have created one. A quick Getting Started guide is available that walks you through the service creation part.

This the overview page for our PostgreSQL service in the Aiven web console. You can connect directly to the PostgreSQL server using the settings described next to "Connection parameters" and "Service URL", but note that these connections will not utilize PGBouncer pooling.



Clicking the "Pools" tab opens a list of PGBouncer connection pools defined for the service. Since this service was launched, there are no pools defined yet:

 
 


To add a new pool click on the "Add pool" button:


The pool settings are:
  • Pool name: Allows you to name your connection pool. This will also become the "database" or "dbname" connection parameter for your pooled client connections.
  • Database: Allows you to choose which database to connect to. Each pool can only connect to a single database.
  • Username: Selects which database username to use when connecting to the backend database.
  • Pool mode: Refers to the pooling mode descried in more detail earlier in this article.
  • Pool size: How many PostgreSQL server connections can this pool use at a time.

For the purposes of this tutorial we'll name the pool as "mypool" and set the pool size as 1 and the pool mode as "statement". Confirming the settings by clicking "Add pool" will immediately create the pool and the pool list is updated:


Clicking the "Info" button next to the pool information shows you the database connection settings for this pool. Note that PGBouncer pools are available under a different port number from the regular unpooled PostgreSQL server port. Both pooled and unpooled connections can be used at the same time.



Verifying the connection pool


We can use the psql command-line client to verify that the pooling works as supposed:

From terminal #1:

$ psql <pool-uri>

From terminal #2:

$ psql <pool-uri>
Now we have two open client connections to the PGBouncer pooler. Let's verify that each connection is able access the database:

Terminal #1:

mypool=> SELECT 1;
 ?column?
──────────
        1
(1 row)


Terminal #2:

mypool=> SELECT 1;
 ?column?
──────────
        1
(1 row)


Both connections respond as they should. Now let's check how many connections there are to the PostgreSQL backend  database:

Terminal #1:

mypool=> SELECT COUNT(*) FROM pg_stat_activity WHERE usename = 'avnadmin';
 count
-------
     1
(1 row)

And as we can see from the pg_stat_activity output the two psql sessions use the same PostgreSQL server database connection.

Summary


The more client connections you have to your database, the more useful connection pooling becomes. Aiven PostgreSQL makes using connection pooling an easy task and migrating from non-pooled connections to pooled connections is just a matter of gradually changing your client-side connection database name and port number!

Try PostgreSQL 9.6 for free in Aiven


Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!

Cheers,
Team Aiven