2016-07-22

Backing up tablespaces and streaming WAL with PGHoard

We've just released a new version of PGHoard, the PostgreSQL cloud backup tool we initially developed for Aiven and later open sourced.

Version 1.4.0 comes with the following new features:
  • Support for PostgreSQL 9.6 beta3
  • Support for backing up multiple tablespaces
  • Support for StatsD and DataDog metrics collection
  • Basebackup restoration now shows download progress
  • Experimental new WAL streaming mode walreceiver, which reads the write-ahead log data directly from the PostgreSQL server using the streaming replication protocol
  • New status API in the internal REST HTTP server
Please see our previous blog post about PGHoard for more information about the tool and a guide for deploying it.

Backing up multiple tablespaces

This is the first version of PGHoard capable of backing up multiple tablespaces. Multiple tablespaces require using the new local-tar backup option for reading files directly from the disk instead of streaming them using pg_basebackup as pg_basebackup doesn't currently allow streaming multiple tablespaces without writing them to the local filesystem.

The current version of PGHoard can utilize the local-tar backup mode only on a PG master server, PostgreSQL versions prior to 9.6 don't allow users to run the necessary control commands on a standby server without using the pgespresso extension. pgespresso also required fixes which we contributed to support multiple tablespaces - once a fixed version has been released we'll add support for it to PGHoard.

The next version of PGHoard, due out by the time of PostgreSQL 9.6 final release, will support local-tar backups from standby servers, natively when running 9.6 and using the pgespresso extension when running older versions with the latest version of the extension.

A future version of PGHoard will support backing up and restoring PostgreSQL basebackups in parallel mode when using the local-tar mode.  This will greatly reduce the time required for setting up a new standby server or restoring a system from backups.

Streaming replication support

This version adds experimental support for reading PostgreSQL's write-ahead log directly from the server using the streaming replication protocol which is also used by PostgreSQL's native replication and related tools such as pg_basebackup and pg_receivexlog. The functionality currently depends on an unmerged psycopg2 pull request which we hope to see land in a psycopg2 release soon.

While the walreceiver mode is still experimental it has a number of benefits over other methods of backing up the WAL and allows implementing new features in the future: temporary, uncompressed, files as written by pg_receivexlog are no longer needed saving disk space and I/O and incomplete WAL segments can be archived at specified intervals or, for example, whenever a new COMMIT appears in the WAL stream.

New contributors

The following people contributed their first patches to PGHoard in this release:
  • Brad Durrow
  • Tarvi Pillessaar

PGHoard in Aiven.io

We're happy to talk more about PGHoard and help you set up your backups with it.  You can also sign up for a free trial of our Aiven.io PostgreSQL service where PGHoard will take care of your backups.


Cheers,
Team Aiven

2016-07-19

New, bigger InfluxDB plans now available

We're happy to announce the immediate availability of new, bigger InfluxDB plans in Aiven. The new plans allow you to store up to 750 gigabytes of time-series data in a fully-managed InfluxDB database.

InfluxDB can be used to store time-series data from various data sources using data collection tools like Telegraf. The collected data is typically operating system and application metric data like CPU utilization and disk space usage, but we've also for example helped set up InfluxDB to host time-series data for an industrial manufacturing line where our Grafana service is used for data visualization.

Our InfluxDB Startup-4 plan, available in all AWS, Google Cloud, UpCloud and DigitalOcean regions, was expanded to 16 gigabytes of storage space and we've announced all new Startup-8, 16, 32 and 64 plans available in all AWS, Google Cloud and UpCloud regions with CPU counts ranging from 1 to 16, RAM from 4 to 64 gigabytes and storage space between 50 and 750 gigabytes.

Trying Aiven is free, no credit card required

Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans.

Go to https://aiven.io/ to get started!


Cheers,

    Team Aiven

2016-07-13

Aiven Kafka now publicly available!

In a world filled with microservices we're delighted to announce yet another expansion of the Aiven service portfolio in the form of Aiven Kafka. Aiven Kafka adds streaming data capabilities in the form of a distributed commit log. For the last three months we've been offering Apache Kafka in private beta and now we're making it publicly available!


Aiven Kafka is a service that can be used to ingest and read back large quantities of log event data. This allows you to write your whole event stream durably in a fire hose like fashion and then process it at your leisure. Kafka is being used in some of the largest companies on the planet for many mission-critical workloads. Besides using it for streaming data you can also use it as a message broker connecting your myriad services with each other.

Historically Kafka itself and especially its reliance on Apache ZooKeeper has made its setup require considerable time and effort and requiring skilled staff to maintain and operate it. Aiven Kafka is now making it trivially easy to have your own managed Kafka cluster.

The easy streaming log service for your microservices

Our Web Console allows you to launch Aiven Kafka in any of our supported clouds and regions with a couple of clicks. All Aiven services are available in all Amazon Web Services, Google Cloud, DigitalOcean and UpCloud regions allowing you to launch services near you in minutes.



Aiven Kafka is a first-class service in Aiven, meaning we'll take care of fault-tolerance, monitoring and maintenance operations on your behalf. In case you need to get more performance out of your Kafka cluster, you can simply expand your cluster by selecting a bigger plan and all your data will be automatically migrated to beefier nodes without any downtime.

Our startup Kafka plan

If you want to try out Kafka on a modestly powered three node cluster and don't need Kafka REST our Startup-2 plan will get you started. After getting started you can easily later upgrade to a larger plan if needed.
  • Startup-2: 1 CPU, 2 GB RAM, 30 GB SSD at $200 / month ($0.274 / hour)

Our three node business Kafka plans

Our Business plans are three node clusters which are deployed alongside Kafka REST to allow the use of HTTP REST calls for interacting with Kafka.
  • Business-4: 1 CPU, 4 GB RAM, 200 GB SSD at $500 / month ($0.685 / hour)
  • Business-8: 2 CPU, 8 GB RAM, 400 GB SSD at $1000 / month ($1.370 / hour)
  • Business-16: 4 CPU, 16 GB RAM, 800 GB SSD at $2000 / month ($2.740 / hour)
 

Highly-available five node premium Kafka plans

If you want an even higher level of reliability and performance our Premium Aiven Kafka plans are made for this. They all come with five (or more for custom plans) Kafka broker nodes. 
  • Premium-4: 1 CPU, 4 GB RAM, 200 GB SSD at $800 / month ($1.096 / hour)
  • Premium-8: 2 CPU, 8 GB RAM, 400 GB SSD at $1600 / month ($2.192 / hour)
  • Premium-16: 4 CPU, 16 GB RAM, 800 GB SSD at $3200 / month ($4.384 / hour)
Also if you need to find larger or otherwise customized plans, please don't hesitate to contact us.

 

Trying Aiven is free, no credit card required

Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans.

Go to https://aiven.io/ to get started!

We value your feedback

We are always interested in ways of making our service better. Please send your feedback and suggestions via email, Facebook, LinkedIn or using our support system.

 Cheers,

   Aiven Team