2016-11-30

Aiven in Google Tokyo & other updates

We're happy to announce that we've recently added the new Google Cloud Northeast Asia (Tokyo) region to Aiven.  All Aiven services are now available in the new region which is our fourth supported cloud in Japan and 12th cloud in Asia.

We've also been busy enhancing our Kafka service with new topic management APIs and user inteface which we hope to roll out during the next couple of weeks.  On the PostgreSQL side we've just finished the development of our connection pooling system and PGBouncer -based connection pooling will be available in Aiven after our next service update.  The use cases and benefits of these new features will be covered in upcoming blog posts.

The rest of the week the Aiven founding team is busy meeting new and old customers and partners at the Slush startup conference here in Helsinki, Finland.

Cheers,
Team Aiven

2016-10-01

PostgreSQL 9.6 now available in AWS, Azure, Google Cloud with Aiven

PostgreSQL 9.6 is now available in Aiven, bringing the latest and most advanced PostgreSQL release to all the major clouds around the world: all Amazon Web Services, DigitalOcean, Google Cloud, Microsoft Azure and UpCloud regions are supported.



9.6 is the latest version of PostgreSQL incorporating a year's development effort and introducing major improvements to handling large amounts of data with new features like parallel query and reducing the need to continuously vacuum unmodified tables.

Aiven also lets you to fork an existing PostgreSQL 9.5 database into a new PostgreSQL 9.6 service. We're also planning to introduce in-place one-click upgrades of existing PostgreSQL 9.5 databases in the near future.

Try PostgreSQL 9.6 for free in Aiven

Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!

Cheers,
Team Aiven

2016-09-23

User management improvements and new Azure regions launched

In this week's updates to Aiven we've greatly enhanced project membership management as well as launched Aiven in five new Azure regions.

 

Project membership improvements

Project membership management improvements make it easier to share the ownership and management responsibilities of an Aiven project between multiple users in one organization.  You can now promote other team members to project administrators, allowing them to invite more members to the project and to adjust billing settings.

We've also updated project invite functionality to allow inviting users who haven't yet registered their Aiven accounts.  Such users are invited to sign up to Aiven and once they've signed up the console will display pending invitations for them.  Pending invitations are shown for all users and projects in the console.

These improvements make it easier for multiple users to start collaborating in Aiven.


New Azure regions launched

We're happy to announce immediate availability of Aiven in five new Azure regions:  All Aiven services are now available in Azure Japan West (Osaka), Japan East (Tokyo), East Asia (Hong Kong), Southeast Asia (Singapore) and Brazil South (São Paulo) regions.


Pricing updates

To reflect the cost differences between different cloud providers and regions we have adjusted pricing of new services in Amazon Web Services and Azure clouds.  Pricing in most AWS regions has been increased for new plans; the prices for current Aiven users will stay unchanged until the end of 2016

We have no plans for changing Google Cloud, DigitalOcean or UpCloud pricing in the foreseeable future.

We'd also like to remind you about Aiven's unique feature allowing seamless migrations between cloud providers, making it possible to migrate services to different clouds if needed.


Trying Aiven is free, no credit card required

Our free trial program is still open: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans.

Go to https://aiven.io/ to get started!

We value your feedback

We are always interested in ways of making our service better. Please send your feedback and suggestions via email, Facebook, LinkedIn or using our support system.

2016-09-15

Aiven brings easy, powerful hosted databases to Microsoft Azure

We are proud to announce that Aiven is now available in the Microsoft Azure cloud!

The services available initially at launch in Microsoft Azure are Aiven PostgreSQL, Aiven Redis, Aiven Elasticsearch, Aiven Kafka, Aiven InfluxDB and Aiven Grafana.

Microsoft AzureMicrosoft Azure is a leading global cloud provider and what makes them special is their high number of data centers around the world, currently totaling 30+.

All Aiven services will be available in all generally available Azure regions, bringing nineteen new cloud regions to Aiven and setting the total number of supported data centers to 47, making us the cloud database provider with the widest geographic availability in the world!

The first batch of new cloud regions immediately available are from Azure North America and Europe. Azure Asia and South America regions will follow soon and will be available in the upcoming weeks.

Here's an updated world map showing our supported data center locations. The new Azure regions are the light blue ones:



Microsoft Azure provides numerous services from computing resources to higher level services like machine learning. See azure.microsoft.com for more information about their services.

The new Aiven regions that are immediately available are:

  • United States
    • Iowa - Azure: Central US
    • Virginia - Azure: East US
    • Virginia - Azure: East US 2
    • Illinois - Azure: North Central US
    • Texas - Azure: South Central US
    • California - Azure: West US
    • Washington - Azure: West US 2
    • Wyoming - Azure: West Central US
  • Canada
    • Ontario - Azure: Canada Central
    • Quebec - Azure: Canada East
  • Europe
    • Ireland - Azure: North Europe
    • Netherlands - Azure: West Europe
    • England - Azure: UK South
    • Wales - Azure: UK West
  • Asia
    • Hong Kong - Azure: East Asia (available in the coming weeks)
    • Singapore - Azure: Southeast Asia (available in the coming weeks)
    • Tokyo, Japan - Azure: Japan East (available in the coming weeks)
    • Osaka, Japan - Azure: Japan West (available in the coming weeks)
  • South America
    • Brazil - Azure: Brazil South (available in the coming weeks)
New services can be launched in these regions today and using the Aiven zero-downtime migration, it is also possible to easily migrate existing services to Azure!

All of the Aiven services offer worry-free fully automated DBaaS hosting, including offsite backups, automatic failure recovery and hardened security.

We will continue to expand our Database-as-a-Service offering in both cloud and region support and by adding more services. We are always looking for feedback on what to improve so feel free to let us know if you have ideas on what you'd love to see us support next.

Go to aiven.io to get started! Free $10 credits at registration, no credit card required. Services are billed by the hour.

Last but not least, we would like to thank all of our customers who participated in our beta testing phase!


Cheers,
    Aiven team


2016-08-18

Getting started with Aiven Kafka

Apache Kafka is a high-throughput publish-subscribe message broker, perfect for providing messaging for microservices. Aiven offers fully managed Kafka-as-a-Service in Amazon Web Services, Google Compute Engine, DigitalOcean and UpCloud (Microsoft Azure support is coming during 2016 Q3!).

Apache Kafka


Apache Kafka is a popular open-source publish-subscribe message broker. Kafka is distributed by design and offers scalability, high-throughput and fault-tolerance. Kafka excels in streaming data workloads, ingesting and serving hundreds of megabytes per second from thousands of clients.

Apache Kafka deployment example



Apache Kafka was originally developed by LinkedIn and open sourced in 2011.

Kafka is often used as a central piece in analytics, telemetry and log aggregation workloads, where it is used to capture and distribute event data at very high data rates. It can act as a communications hub for microservices for distributing work over a large cluster of nodes.

With Aiven, we use Kafka as a message bus between our cluster nodes as well as delivering telemetry, statistics and logging data. Kafka's guarantees for message delivery and fault-tolerance allows us to simplify and de-couple service components.

What is Aiven Kafka


Aiven Kafka is our fully managed Kafka service. We take care of the deployment and operational burden of running your own Kafka service, and make sure your cluster stays available, healthy and always up-to-date. We ensure your data remains safe by encrypting it both in transit and at rest on disk.

We offer multiple different plan types with different cluster sizing and capacity, and charge only based on your actual use on an hourly basis. Custom plans for deployments that are larger or have specific needs can also be requested. Aiven also makes it possible to migrate between the plans with no downtime to address changes in your requirements.

Below, I'll walk you through setting up and running with your first Aiven Kafka service.


Getting started with Aiven Kafka


Creating Aiven Kafka service is easy: just select the correct service type from the drop down menu on the new service creation dialog. You'll have the option of selecting three or five node cluster plans with the storage sizing of your choice. The larger node count allows for larger throughput or larger replica factors for mission critical data. If unsure, pick a three node cluster; you can always change the selected plan at a later time.



All Aiven services are offered over SSL encrypted connections for your protection. With Kafka, you're also required to perform client authentication with service certificates we provide. You can find and download these keys and certificates on the connection parameters section on the service details page: access key and certifications plus CA certificate you can use to verify the Aiven endpoint. Store these locally, we'll be referring back to them in code examples below (ca.crt, client.crt, client.key).




Finally, you can create the topics you'd like to use under the topics tab on the service details page. In Kafka terms, topics are logical channels that your send messages to and read them from. Topics themselves are divided into one or more partitions. Partitions can be used to handle larger read/write rates, but do note that Kafka's ordering guarantees are only valid within one partition.

When creating a topic, you can select number of partitions, number of replicas and how many hours the messages are retained in the Kafka logs before deletion. You also can increase the number of partitions at a later time.


That's it! The service is up and running and ready to capture and distribute your messages. Aiven team will take care of the operational burden of your cluster, and ensure it remains available and in use at all times. To utilize the service, we've included code examples in Python and Node.js below. Just make sure to replace the value of bootstrap_servers below with the service URL from the service details page. Also, verify that the SSL settings below point to the actual key and certificate files downloaded earlier.

Accessing Aiven Kafka in Python


Producing messages - Kafka term for sending them:

from kafka import KafkaProducer

producer = KafkaProducer(
    bootstrap_servers="getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705",
    security_protocol="SSL",
    ssl_cafile="ca.crt",
    ssl_certfile="client.crt",
    ssl_keyfile="client.key",
)

for i in range(1, 4):
    message = "message number {}".format(i)
    print("Sending: {}".format(message))
    producer.send("demo-topic", message.encode("utf-8"))

# Wait for all messages to be sent
producer.flush()

Consuming or receiving the same:

from kafka import KafkaConsumer

consumer = KafkaConsumer(
    "demo-topic",
    bootstrap_servers="getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705",
    client_id="demo-client-1",
    group_id="demo-group",
    security_protocol="SSL",
    ssl_cafile="ca.crt",
    ssl_certfile="client.crt",
    ssl_keyfile="client.key",
)

for msg in consumer:
    print("Received: {}".format(msg.value))

Output from the producer above:

$ python kafka-producer.py
Sending: message number 1
Sending: message number 2
Sending: message number 3

And the consuming side:

$ python kafka-consumer.py
Received: message number 1
Received: message number 2
Received: message number 3

 

Accessing Aiven Kafka in Node.js


Here's a Node.js example utilizing node-rdkafka module:

var Kafka = require('node-rdkafka');

var producer = new Kafka.Producer({
    'metadata.broker.list': 'getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705',
    'security.protocol': 'ssl',
    'ssl.key.location': 'client.key',
    'ssl.certificate.location': 'client.crt',
    'ssl.ca.location': 'ca.crt',
    'dr_cb': true
});

producer.connect();

producer.on('ready', function() {
    var topic = producer.Topic('demo-topic', {'request.required.acks': 1});
    producer.produce({
        message: new Buffer('Hello world!'),
        topic: topic,
    }, function(err) {
        if (err) {
            console.log('Failed to send message', err);
        } else {
            console.log('Message sent successfully');
        }
    });
});

And the consuming side:

var Kafka = require('node-rdkafka');

var consumer = new Kafka.KafkaConsumer({
    'metadata.broker.list': 'getting-started-with-kafka.htn-aiven-demo.aivencloud.com:17705',
    'group.id': 'demo-group',
    'security.protocol': 'ssl',
    'ssl.key.location': 'client.key',
    'ssl.certificate.location': 'client.crt',
    'ssl.ca.location': 'ca.crt',
});

var stream = consumer.getReadStream('demo-topic');

stream.on('data', function(data) {
    console.log('Got message:', data.message.toString());
});



Trying Aiven is free, no credit card required


Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans. The offer works for all of our services: PostgreSQL, Redis, InfluxDB, Grafana, Elasticsearch and Kafka!

Go to https://aiven.io/ to get started!


Cheers,

    Team Aiven

2016-07-22

Backing up tablespaces and streaming WAL with PGHoard

We've just released a new version of PGHoard, the PostgreSQL cloud backup tool we initially developed for Aiven and later open sourced.

Version 1.4.0 comes with the following new features:
  • Support for PostgreSQL 9.6 beta3
  • Support for backing up multiple tablespaces
  • Support for StatsD and DataDog metrics collection
  • Basebackup restoration now shows download progress
  • Experimental new WAL streaming mode walreceiver, which reads the write-ahead log data directly from the PostgreSQL server using the streaming replication protocol
  • New status API in the internal REST HTTP server
Please see our previous blog post about PGHoard for more information about the tool and a guide for deploying it.

Backing up multiple tablespaces

This is the first version of PGHoard capable of backing up multiple tablespaces. Multiple tablespaces require using the new local-tar backup option for reading files directly from the disk instead of streaming them using pg_basebackup as pg_basebackup doesn't currently allow streaming multiple tablespaces without writing them to the local filesystem.

The current version of PGHoard can utilize the local-tar backup mode only on a PG master server, PostgreSQL versions prior to 9.6 don't allow users to run the necessary control commands on a standby server without using the pgespresso extension. pgespresso also required fixes which we contributed to support multiple tablespaces - once a fixed version has been released we'll add support for it to PGHoard.

The next version of PGHoard, due out by the time of PostgreSQL 9.6 final release, will support local-tar backups from standby servers, natively when running 9.6 and using the pgespresso extension when running older versions with the latest version of the extension.

A future version of PGHoard will support backing up and restoring PostgreSQL basebackups in parallel mode when using the local-tar mode.  This will greatly reduce the time required for setting up a new standby server or restoring a system from backups.

Streaming replication support

This version adds experimental support for reading PostgreSQL's write-ahead log directly from the server using the streaming replication protocol which is also used by PostgreSQL's native replication and related tools such as pg_basebackup and pg_receivexlog. The functionality currently depends on an unmerged psycopg2 pull request which we hope to see land in a psycopg2 release soon.

While the walreceiver mode is still experimental it has a number of benefits over other methods of backing up the WAL and allows implementing new features in the future: temporary, uncompressed, files as written by pg_receivexlog are no longer needed saving disk space and I/O and incomplete WAL segments can be archived at specified intervals or, for example, whenever a new COMMIT appears in the WAL stream.

New contributors

The following people contributed their first patches to PGHoard in this release:
  • Brad Durrow
  • Tarvi Pillessaar

PGHoard in Aiven.io

We're happy to talk more about PGHoard and help you set up your backups with it.  You can also sign up for a free trial of our Aiven.io PostgreSQL service where PGHoard will take care of your backups.


Cheers,
Team Aiven

2016-07-19

New, bigger InfluxDB plans now available

We're happy to announce the immediate availability of new, bigger InfluxDB plans in Aiven. The new plans allow you to store up to 750 gigabytes of time-series data in a fully-managed InfluxDB database.

InfluxDB can be used to store time-series data from various data sources using data collection tools like Telegraf. The collected data is typically operating system and application metric data like CPU utilization and disk space usage, but we've also for example helped set up InfluxDB to host time-series data for an industrial manufacturing line where our Grafana service is used for data visualization.

Our InfluxDB Startup-4 plan, available in all AWS, Google Cloud, UpCloud and DigitalOcean regions, was expanded to 16 gigabytes of storage space and we've announced all new Startup-8, 16, 32 and 64 plans available in all AWS, Google Cloud and UpCloud regions with CPU counts ranging from 1 to 16, RAM from 4 to 64 gigabytes and storage space between 50 and 750 gigabytes.

Trying Aiven is free, no credit card required

Remember that trying Aiven is free: you will receive US$10 worth of free credits at sign-up which you can use to try any of our service plans.

Go to https://aiven.io/ to get started!


Cheers,

    Team Aiven