aht on

How to Run Metis on Top Of StackGres

Mon, 12 Feb 2024 14:00:00 +0100

How To Run Metis On Top Of StackGres

In this how-to guide, we’ll walk you through the whole process of setting up Metis on a Kubernetes cluster using StackGres.

Metis is a comprehensive database observability tool. It can track your live database activity, suggest improvements for schema, indexes, queries, configuration, and more. By onboarding your PostgreSQL server with Metis, you get access to live infrastructural metrics (CPU, memory usage) and database metrics (transactions, buffers, caches). Metis can look for anomalies in performance, index usage, or data distribution. See the documentation on how to monitor & troubleshoot your databases.

This guide is also available as a video

This post has been published with explicit permission from Metis. It's a copy of an article originally published on Metis website on February 12, 2024.

OnGres (StackGres)

StackGres is the full-stack Postgres Platform. Fully open-source software to run your own Postgres-as-a-Service on any cloud or on-prem. StackGres is a project from OnGres, the Postgres laser-focused startup (“OnGres” means “ON postGRES”).

StackGres is a Kubernetes Operator for Postgres. It allows you to create production-ready Postgres clusters in seconds. No advanced Postgres expertise is required. You can use the built-in Web Console or the high-level Kubernetes CRDs for CLI and GitOps.

With StackGres, writing a simple YAML manifest (or point-and-click on the Web Console) is all that is needed to create production-ready Postgres clusters including high availability with Patroni, replication, connection pooling, automated backups, monitoring, and centralized logs.

With support for more than 150 Postgres extensions, StackGres is the most extensible Postgres platform available. It also provides support for Babelfish for Postgres (which brings SQL Server compatibility, at the wire protocol level, SQL and T-SQL) and integrations with Citus for sharding Postgres, Timescale for time-series, Supabase and now, FerretDB.

Metis

Metis is a powerful observability tool that can track all the aspects of your databases and database servers. It integrates with your database server with the help of an agent that you need to deploy in your cluster so it can monitor the ongoing activity.

Metis is distributed as a Docker container. You can install it in any environment supporting OCI containers, including AWS ECS, Kubernetes, AKS, or GKE. This makes using Metis straightforward and lets you configure everything in seconds.

Metis connects to your database and extracts details of the database activity. This includes metrics around infrastructure (CPU, memory usage, files opened), databases (transactions, caches, temporary files), live queries (and their execution plans), configuration, schema details, index usage, and much more. Metis explains what is happening in your databases. Metis can look for anomalies and detect drifts between environments. Metis can suggest actionable performance improvements and notify you when things degrade.

Setting Up Metis On Top Of StackGres

We’ll need to install the Kubernetes cluster, then install StackGres, then configure the StackGres cluster, and then configure Metis. Instructions below are for Amazon Linux 2 x86_64 but will work the same way on other operating systems like Mac OS and Windows (with slight syntax changes for PowerShell or other shells), and on ARM64 architecture.

Installing Kubernetes With Minikube

Before installing StackGres, you will need a running Kubernetes cluster and the usual command line tools kubectl and Helm. Please refer to the respective installation pages if you don’t have these tools. As for Kubernetes, if you don’t have one you can try easily with minikube. It can be installed like this:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

sudo install minikube-linux-amd64 /usr/local/bin/minikube

This should give you a running single-node cluster in seconds (depending on your Internet connection speed). You can install it in other operating systems the way it’s described in the docs.

Keep in mind that minikube uses Docker behind the scenes. It works natively in Linux, and with a virtual machine in MacOS and Windows. Refer to installation instructions for Docker Desktop to learn how to install it on your machine.

You can now create the minikube cluster with:

This should give you the output similar to this:

* minikube v1.32.0 on Amazon 2
* Automatically selected the docker driver. Other choices: none, ssh
* Using Docker driver with root privileges
* Starting control plane node minikube in cluster minikube
* Pulling base image ...
* Creating docker container (CPUs=2, Memory=3900MB) ...
* Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
- Generating certificates and keys ...
- Booting up control plane ...
- Configuring RBAC rules ...
* Configuring bridge CNI (Container Networking Interface) ...
* Verifying Kubernetes components...
- Using image gcr.io/k8s-minikube/storage-provisioner:v5
* Enabled addons: storage-provisioner, default-storageclass
* Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

When you’re done with the examples, you can easily delete the cluster with:

Let’s now install StackGres.

Installing StackGres

The best way to install StackGres is through the official Helm chart. You can install it in this way:

curl -LO https://get.helm.sh/helm-v3.14.0-linux-amd64.tar.gz

tar -zxvf helm-v3.14.0-linux-amd64.tar.gz

sudo mv linux-amd64/helm /usr/local/bin/helm

Follow this page to install Helm on your machine.

For our particular setup, we use the following Helm commands:

helm repo add stackgres-charts https://stackgres.io/downloads/stackgres-k8s/stackgres/helm/

helm install --create-namespace --namespace stackgres stackgres-operator stackgres-charts/stackgres-operator

To confirm that the operator is running while also waiting for setup to complete, run the following commands:

kubectl wait -n stackgres deployment -l group=stackgres.io --for=condition=Available

kubectl get pods -n stackgres -l group=stackgres.io

As you run the first kubectl command, it should wait for the successful deployment, and the second command will list the pods running in the stackgres namespace.

NAME READY STATUS RESTARTS AGE
stackgres-operator-57bff75d-xlnfp 1/1 Running 1 (2m36s ago) 2m58s

Creating a StackGres Cluster

Here, we’ll create an SGCluster configured to fit Metis requirements. We’re going to use some resources created inline. Metis requires the pg_stat_statements extension, a separate user, and permissions to monitor databases.

Let’s create a password for the user that will get initialized based on the SQL command:

kubectl -n stackgres create secret generic metis-user-password-secret --from-literal=metis-create-user-sql="create user metis password 'admin123'"

Let’s now create a script that will create a database named metis and a user metis:

cat << EOF | kubectl apply -f -
apiVersion: stackgres.io/v1
kind: SGScript
metadata:
 namespace: stackgres
 name: cluster-scripts
spec:
 scripts:
 - name: create-metis-user
 scriptFrom:
 secretKeyRef:
 name: metis-user-password-secret
 key: metis-create-user-sql
 - name: grant-monitor-to-metis
 script: |
 GRANT pg_monitor TO metis
 - name: create-metis-database
 script: |
 CREATE DATABASE metis OWNER metis
 - name: grant-conntent-to-metis-database
 script: |
 GRANT CONNECT ON DATABASE metis TO metis
 - name: create-pgstatstatements-extension
 database: metis
 script: |
 CREATE EXTENSION IF NOT EXISTS pg_stat_statements;
EOF

We can see the script has five parts. First, we create the user with a password and store the password in the Kubernetes secret. Next, we give pg_monitor permissions to the user. Next, we create the database. Then, we let the user connect to the database. Finally, we create the pg_stat_statements extension in the metis database.

We are now ready to create the Postgres cluster:

cat << EOF | kubectl apply -f -
apiVersion: stackgres.io/v1
kind: SGCluster
metadata:
 namespace: stackgres
 name: cluster
spec:
 postgres:
 version: '15.0'
 instances: 1
 pods:
 persistentVolume:
 size: '5Gi'
 managedSql:
 scripts:
 - sgScript: cluster-scripts
EOF

It should take a few seconds to a few minutes for the cluster to be up and running:

kubectl get pods -n stackgres

NAME READY STATUS RESTARTS AGE
cluster-0 6/6 Running 0 74s

Likewise, a database named metis must exist and be owned by the same user:

kubectl -n stackgres exec -it cluster-0 -c postgres-util -- psql -l metis

 List of databases
Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges
--------+----------+----------+---------+---------+------------+-----------------+--------------------
metis | metis | UTF8 | C.UTF-8 | C.UTF-8 | | libc | =Tc/metis
...

Deploying Metis

We can now add monitoring of the database to Metis. Go to Metis and click Deploy:

Select Postgres and click Next:

Metis instructs us how to configure the user and grant necessary permissions. We already did that. Click Next.

Metis asks for the connection string now:

Let’s extract the server name:

kubectl get svc -n stackgres

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cluster ClusterIP 10.99.23.208 <none> 5432/TCP,5433/TCP 2m34s
cluster-config ClusterIP None <none> <none> 2m34s
cluster-primary ExternalName <none> cluster.stackgres.svc.cluster.local <none> 2m34s
cluster-replicas ClusterIP 10.103.149.23 <none> 5432/TCP,5433/TCP 2m34s
cluster-rest ClusterIP 10.104.238.166 <none> 8008/TCP 2m34s
stackgres-operator ClusterIP 10.108.192.220 <none> 443/TCP 34m
stackgres-restapi ClusterIP 10.103.146.192 <none> 443/TCP 33m

The service name is cluster.stackgres.svc.cluster.local. The final URL should be like this:

postgresql://metis:admin123@cluster.stackgres.svc.cluster.local:5432/postgres?sslmode=disable

Click Next. Finally, click on Helm and copy the script to deploy Metis.

Extend the script with --namespace stackgres and run it. Your API key will be different as its a specific API key to be used with your project:

helm repo add metis https://charts.metisdata.io

helm install mmc metis/mmc-chart --namespace stackgres --set API_KEY=YOURAPIKEY --set-json CONNECTION_STRING='[{ "uri":"postgresql://metis:admin123@cluster.stackgres.svc.cluster.local:5432/metis?sslmode=disable" }]'

When you execute the script, you should get the following:

NAME: mmc
LAST DEPLOYED: Tue Jan 23 11:24:14 2024
NAMESPACE: stackgres
STATUS: deployed
REVISION: 1
TEST SUITE: None

We can verify that it works with:

kubectl get pods -n stackgres

NAME READY STATUS RESTARTS AGE
cluster-0 6/6 Running 0 25m
mmc-deployment-58dbb76fcc-vnnd5 1/1 Running 0 13s

You can now go to Metis Monitoring and see that the new host has been added:

When you click on the host, you should see the properties and databases:

You can also click on database metis and get all the details:

Notice that many charts are empty as it’s a completely new database.

Monitoring Live Database Activity

Let’s now add some data and run some queries in the database.

kubectl -n stackgres exec -it cluster-0 -c postgres-util -- psql --username=metis --dbname=metis -c "CREATE TABLE orders(region VARCHAR(100), amount INT, product VARCHAR(100))"

kubectl -n stackgres exec -it cluster-0 -c postgres-util -- psql --username=metis --dbname=metis -c "INSERT INTO orders(region, amount, product)VALUES ('EU', 10, 'Product1'), ('EU', 20, 'Product2'), ('US', 1, 'XYZ'), ('JP', 10, 'ABC')"

kubectl -n stackgres exec -it cluster-0 -c postgres-util -- psql --username=metis --dbname=metis -c "SELECT region, product, (SELECT SUM(amount) AS total_sales FROM orders AS o2 WHERE o2.region = o.region GROUP BY region)
FROM orders AS o
WHERE region IN (
 SELECT region
 FROM orders
 GROUP BY region
 HAVING SUM(amount) > (
 SELECT SUM(amount)
 FROM orders
 ) / 10
)"

We create table orders, insert some data, and then run a query that extracts the rows with some aggregation. This is the output:

region | product | total_sales
--------+----------+-------------
EU | Product1 | 30
EU | Product2 | 30
JP | ABC | 10
(3 rows)

You can now go to Metis and check the Table Sizes widget. It shows the orders table:

You can also go to Top Queries widget and notice that new queries are flowing to the database:

Metis captures the live activity of your database and can suggest how to improve things.

Conclusion

Throughout this article, we have explored the process of setting up a Kubernetes cluster, deploying the StackGres operator, and running Metis on top of it. We have covered steps such as creating an SGCluster, and monitoring basic database operations with Metis.

If you’re eager to try Metis and experience its capabilities firsthand, we encourage you to try it out in your own Kubernetes environment, through the StackGres operator.

Here’s the Stackgres runbook and Metis Guide to get you started.

If you encounter any issue or just wish to say “Hi!” and tell us how Metis works on StackGres, you may drop a line at the:

‍

How to Run FerretDB on Top of StackGres

Wed, 16 Aug 2023 12:05:34 +0100

In this how-to guide, we’ll walk you through the whole process of setting up FerretDB on a Kubernetes cluster using StackGres.

As an open-source MongoDB alternative, FerretDB translates MongoDB wire protocols to SQL using a PostgreSQL backend. And by using the StackGres operator, you can easily deploy and manage PostgreSQL instances, as well as access all the management features you need.

So if you’re exploring a scalable and open-source MongoDB alternative on your Kubernetes cluster, then this article on FerretDB and StackGres should pique your interest.

This post has been published with explicit permission from FerretDB. It's a copy of an article originally published on FerretDB's website on July 13, 2023.

OnGres (StackGres)

StackGres is the full-stack Postgres Platform. A fully open source software to run your own Postgres-as-a-Service on any cloud or on-prem. StackGres is a project from OnGres, the Postgres laser-focused startup (“OnGres” means “ON postGRES”).

StackGres is a Kubernetes Operator for Postgres. It allows you to create production-ready Postgres clusters in seconds. No advanced Postgres expertise required. You can use the built-in Web Console or the high level Kubernetes CRDs for CLI and GitOps.

FerretDB

FerretDB is the defacto open-source replacement for MongoDB with the popular and reliable PostgreSQL as the database backend. What FerretDB does is convert MongoDB wire protocols in BSON format into JSONB in PostgreSQL. See this article to learn more about how this works.

Despite MongoDB’s popularity as an open-source database and its popularity among developers, after the switch from open source license to SSPL (get the full story on that here!), it was important to restore MongoDB workloads back to open-source so users can have complete control of their data without vendor lock-in.

Built on an ever-reliable PostgreSQL database backend, you can host FerretDB anywhere or run it locally on your own machine. Another significant advantage of FerretDB is that you get to use the same syntax and commands as you’re used to in MongoDB. Besides, you can also query it with SQL in PostgreSQL (in some cases, you may also need intricate knowledge of JSONB for more advanced queries).

At present, FerretDB is compatible with the most common MongoDB use cases and plans on improving and adding more features as needs arise. Plus, we’ve just recently released FerretDB version 1.5.0, which includes beta support for SQLite backend.

Setting up FerretDB on top of StackGres

Before installing StackGres, you will need a running Kubernetes cluster and the usual command line tools kubectl and Helm. Please refer to the respective installation pages if you don’t have these tools.

As for Kubernetes, if you don’t have one you can try easily with K3s. It can be installed with a single command line as in:

curl -sfL https://get.k3s.io | sh -

This should give you a running single-node cluster in seconds (depending on your Internet connection speed).

Keep in mind that K3s is not available natively on macOS. If you want to run it on macOS, you’ll have to use a virtual machine or a Docker container running Linux.

First, you need to install Docker Desktop for Mac from the Docker website. After installing Docker Desktop, go to preferences and navigate to the Kubernetes tab to “Enable Kubernetes”, then click “Apply & Restart”.

Fortunately, K3D is a tool that makes it easy to run K3s inside Docker, which works on macOS. You can install K3D using Homebrew and if successful, create a Kubernetes cluster:

brew install k3d &&k3d cluster create <cluster-name> --image rancher/k3s:v1.25.9-k3s1

Then you can get the configuration so that you can connect and operate with it via both kubectl and Helm:

mkdir -p ~/.kube/; sudo k3s kubectl config view --raw >> ~/.kube/config

For macOS users, using the command above might create some issues so it’s better to use this one below:

k3d kubeconfig get <cluster-name> > ~/.kube/config

Once you are done you can uninstall k3s if you wish with sudo /usr/local/bin/k3s-uninstall.sh.

Installing StackGres

The best way to install StackGres is through the official Helm chart. Here’s the installation guide in the official docs.

For our particular setup, we use the following Helm commands:

helm repo add stackgres-charts https://stackgres.io/downloads/stackgres-k8s/stackgres/helm/
helm install --create-namespace --namespace stackgres stackgres-operator stackgres-charts/stackgres-operator

To confirm that the operator is running while also waiting for setup to complete, run the following commands:

kubectl wait -n stackgres deployment -l group=stackgres.io --for=condition=Available
kubectl get pods -n stackgres -l group=stackgres.io

As you run the first kubectl command, it should wait for the successful deployment, and the second command will list the pods running in the stackgres namespace.

NAME READY STATUS RESTARTS AGE
stackgres-operator-c4c6b4bcd-trsgp 1/1 Running 0 4m50s
stackgres-restapi-6986cc8997-lfwql 2/2 Running 0 4m49s

Creating a StackGres Cluster

Here, we’ll create an SGCluster configured to fit FerretDB’s requirements. Resources for the example are available in the apps-on-stackgres GitHub repository.

Please clone the repository and navigate to the examples/ferretdb directory, which contains all the files referenced in this guide.

To properly group all related resources together, let’s first create a namespace:

kind: Namespace
apiVersion: v1
metadata:
name: ferretdb

From the apps-on-stackgres GitHub repository on your local machine, navigate to the examples/ferretdb folder within your terminal and apply the configurations specified in the 01-namespace.yaml file to your Kubernetes cluster using this command:

kubectl apply -f 01-namespace.yaml

During startup, FerretDB will try to configure the search_path parameter in PostgreSQL.

However, PgBouncer, the Postgres sidecar deployed by StackGres by default, does not support this.

You can either choose to disable PgBouncer’s sidecar (not recommended) or customize PgBouncer’s connection pooling configuration so it ignores this parameter:

apiVersion: stackgres.io/v1
kind: SGPoolingConfig
metadata:
name: sgpoolingconfig1
namespace: ferretdb
spec:
pgBouncer:
pgbouncer.ini:
pgbouncer:
ignore_startup_parameters: extra_float_digits,search_path

Create with:

kubectl apply -f 02-sgpoolingconfig.yaml

FerretDB uses one or more Postgres databases, and requires them to be created and owned by a given user. In this guide, we will create one database, with one user and a unique password, but we will not be using a Potgres superuser for this.

We plan to use the StackGres' SGScript facility to create, manage and apply SQL scripts in the database automatically.

First, we’ll start by creating a Secret containing the SQL command that will generate a random password for the user.

#!/bin/sh
PASSWORD="$(dd if=/dev/urandom bs=1 count=8 status=none | base64 | tr / 0)"
kubectl -n ferretdb create secret generic createuser \
--from-literal=sql="create user ferretdb with password '"${PASSWORD}"'"

./03-createuser_secret.sh

The next step is to create SGScript, which contains two scripts: a script for creating a password-protected user by obtaining the SQL literal from this Secret, and another one to create the database, owned by this user, and configured to FerretDB’s requirements:

apiVersion: stackgres.io/v1
kind: SGScript
metadata:
name: createuserdb
namespace: ferretdb
spec:
scripts:
- name: create-user
scriptFrom:
secretKeyRef:
name: createuser
key: sql
- name: create-database
script: |
  create database ferretdb owner ferretdb encoding 'UTF8' locale 'en_US.UTF-8' template template0;

kubectl apply -f 04-sgscript.yaml

We are now ready to create the Postgres cluster:

apiVersion: stackgres.io/v1
kind: SGCluster
metadata:
namespace: ferretdb
name: postgres
spec:
postgres:
version: '15'
instances: 1
pods:
persistentVolume:
size: '5Gi'
configurations:
sgPoolingConfig: sgpoolingconfig1
managedSql:
scripts:
- sgScript: createuserdb

kubectl apply -f 05-sgcluster.yaml

It should take a few seconds to a few minutes for the cluster to be up and running:

kubectl -n ferretdb get pods
NAME READY STATUS RESTARTS AGE
postgres-0 6/6 Running 0 16m

Likewise, a database named ferretdb must exist and be owned by the same user:

kubectl -n ferretdb exec -it postgres-0 -c postgres-util -- psql -l ferretdb
List of databases
Name | Owner | Encoding | Collate | Ctype | ICU Locale | Locale Provider | Access privileges
-----------+----------+----------+-------------+-------------+------------+-----------------+-----------------------
ferretdb | ferretdb | UTF8 | en_US.UTF-8 | en_US.UTF-8 | | libc |
...

Deploying FerretDB

FerretDB itself is a stateless application, and as such, so we can just use the standard Deployment pattern (for easy scaling) with a Service to deploy it:

apiVersion: apps/v1
kind: Deployment
metadata:
name: ferretdb-dep
namespace: ferretdb
labels:
app: ferretdb
spec:
replicas: 1
selector:
matchLabels:
app: ferretdb
template:
metadata:
labels:
app: ferretdb
spec:
containers:
- name: ferretdb
image: ghcr.io/ferretdb/ferretdb
ports:
- containerPort: 27017
env:
- name: FERRETDB_POSTGRESQL_URL
value: postgres://postgres/ferretdb
---
apiVersion: v1
kind: Service
metadata:
name: ferretdb
namespace: ferretdb
spec:
selector:
app: ferretdb
ports:
- name: mongo
protocol: TCP
port: 27017
targetPort: 27017

kubectl apply -f 06-ferretdb.yaml

Note the line where we pass the FERRETDB_POSTGRESQL_URL environment variable to FerretDB’s container, set with the value postgres://postgres/ferretdb: the second postgres on the string is the Service name that StackGres exposes pointing to the primary instance of the created cluster, which is named after the SGCluster’s name; and ferretdb is the name of the database.

If all goes well, you should see the pod up & running. To test it, we need to run a MongoDB client. For example, we can use kubectl run to run a mongosh image:

kubectl -n ferretdb run mongosh --image=rtsp/mongosh --rm -it -- bash

FerretDB exposes the same Postgres database, usernames, and passwords over the MongoDB wire protocol. In that case, we have access to the user, password, and database that were created previously using SGScript.

When the terminal prompt appears, type the command:

mongosh mongodb://ferretdb:${PASSWORD}@${FERRETDB_SVC}/ferretdb?authMechanism=PLAIN

where ${PASSWORD} represents the randomly generated password for the ferretdb user in Postgres:

kubectl -n ferretdb get secret createuser --template '{{ printf "%s\n" (.data.sql | base64decode) }}'

and ${FERRETDB_SVC} is the address and port exposed by the FerretDB Service (10.43.94.52:27017 in the example below):

kubectl -n ferretdb get svc ferretdb
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ferretdb ClusterIP 10.43.94.52 <none> 27017/TCP 10m

QuickStart Example

Once the mongosh command is executed, you can try inserting and querying data:

ferretdb> db.test.insertOne({a:1})
{
acknowledged: true,
insertedId: ObjectId("646dca174663396264d4bfeb")
}
ferretdb> db.test.find()
[ { _id: ObjectId("646dca174663396264d4bfeb"), a: 1 } ]

If you are curious, you can see how data was materialized on the Postgres database:

kubectl -n ferretdb exec -it postgres-0 -c postgres-util -- psql ferretdb
psql (15.1 (OnGres 15.1-build-6.18))
Type "help" for help.
ferretdb=# set search_path to ferretdb;
SET
ferretdb=# \dt
List of relations
Schema | Name | Type | Owner
----------+-----------------------------+-------+----------
ferretdb | _ferretdb_database_metadata | table | ferretdb
ferretdb | test_afd071e5 | table | ferretdb
(2 rows)
ferretdb=# table test_afd071e5;
_jsonb
-----------------------------------------------------------------------------------------------------------------------------
{"a": 1, "$s": {"p": {"a": {"t": "int"}, "_id": {"t": "objectId"}}, "$k": ["_id", "a"]}, "_id": "646dca174663396264d4bfeb"}
(1 row)

Creating a Meteor example with FerretDB on top of Stackgres

To test and run our entire setup locally, we can use port forwarding with kubectl command, which should forward all connections from our local machine port to the ferretdb service’s port 27017 in the cluster.

kubectl port-forward svc/ferretdb 27017:27017 -n ferretdb

In a new terminal, run this command to be sure your connection is working. Note that this requires that you have mongosh installed.

mongosh 'mongodb://ferretdb:${PASSWORD}@localhost:27017/ferretdb?authMechanism=PLAIN'

To test our database locally, we’ll be executing commands from a Meteor application.

If you don’t have Meteor installed, you can install it by following the steps in their documentation. Then create a project using

meteor create <app-name>
cd <app-name>
meteor npm install

Be sure to replace <app-name> with that of your project.

Without making any changes, on startup, the function will insert documents with the fields title, url, createdAt as long as there are none present.

async function insertLink({ title, url }) {
await LinksCollection.insertAsync({ title, url, createdAt: new Date() });
}
Meteor.startup(async () => {
// If the Links collection is empty, add some data.
if (await LinksCollection.find().countAsync() === 0) {
await insertLink({
title: 'Do the Tutorial',
url: 'https://www.meteor.com/tutorials/react/creating-an-app',
});
await insertLink({
title: 'Follow the Guide',
url: 'https://guide.meteor.com',
});
await insertLink({
title: 'Read the Docs',
url: 'https://docs.meteor.com',
});
await insertLink({
title: 'Discussions',
url: 'https://forums.meteor.com',
});
}
// We publish the entire Links collection to all clients.
// In order to be fetched in real-time to the clients
Meteor.publish("links", function () {
return LinksCollection.find();
});
});

Once installed, you can run it locally by connecting the MongoDB URL with Meteor, which then sets the environment variable for the MongoDB connection string. This way, Meteor will use an external MongoDB database instead of starting its own.

MONGO_URL='mongodb://username:password@localhost:27017/mydatabase' meteor

You can check out these documents in the Postgres database.

ferretdb-# \dt
List of relations
Schema | Name | Type | Owner
----------+-----------------------------+-------+----------
ferretdb | _ferretdb_database_metadata | table | ferretdb
ferretdb | links_e9ca9aee | table | ferretdb
ferretdb | test_afd071e5 | table | ferretdb
(3 rows)
ferretdb=# table links_e9ca9aee;
_jsonb
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
{"$s": {"p": {"_id": {"t": "string"}, "url": {"t": "string"}, "title": {"t": "string"}, "createdAt": {"t": "date"}}, "$k": ["_id", "title", "url", "createdAt"]}, "_id": "7NmpEw6k7z7HNTJpf", "url": "https://www.meteor.com/tutorials/react/creating-an-app", "title": "Do the Tutorial", "createdAt": 1687291371079}
{"$s": {"p": {"_id": {"t": "string"}, "url": {"t": "string"}, "title": {"t": "string"}, "createdAt": {"t": "date"}}, "$k": ["_id", "title", "url", "createdAt"]}, "_id": "f3D3wB25KGduKzbsc", "url": "https://guide.meteor.com", "title": "Follow the Guide", "createdAt": 1687291371148}
{"$s": {"p": {"_id": {"t": "string"}, "url": {"t": "string"}, "title": {"t": "string"}, "createdAt": {"t": "date"}}, "$k": ["_id", "title", "url", "createdAt"]}, "_id": "rdJH59NGgnonSSe5o", "url": "https://docs.meteor.com", "title": "Read the Docs", "createdAt": 1687291371159}
{"$s": {"p": {"_id": {"t": "string"}, "url": {"t": "string"}, "title": {"t": "string"}, "createdAt": {"t": "date"}}, "$k": ["_id", "title", "url", "createdAt"]}, "_id": "zoGgwqbofdGDvPyj4", "url": "https://forums.meteor.com", "title": "Discussions", "createdAt": 1687291371164}
(4 rows)

Conclusion

With FerretDB, you can seamlessly store and access your data, enabling the flexibility and freedom of using MongoDB’s document model and syntax without the vendor lock-in.

Throughout this article, we have explored the process of setting up a Kubernetes cluster, deploying the StackGres operator, and running FerretDB on top of it. We have covered steps such as creating an SGCluster, and performing basic database operations in FerretDB.

If you’re eager to try FerretDB and experience its capabilities firsthand, we encourage you to try it out in your own Kubernetes environment, through the StackGres operator.

Here’s the Stackgres runbook and FerretDB installation guide to get you started.

If you encounter any issue or just wish to say “Hi!” and tell us how FerretDB works on StackGres, you may drop a line at the:

Easily Running Babelfish for PostgreSQL on Kubernetes

Mon, 15 Nov 2021 20:05:34 +0100

Easily Running Babelfish for PostgreSQL on Kubernetes

TL;DR

Babelfish for PostgreSQL (“Babelfish” in short) is an open source project created by Amazon AWS that adds SQL Server compatibility on top of Postgres. It was made public a couple of weeks ago both as a managed service and as an open source project. Using the latter involves as of today compiling it from source, which requires some effort and expertise. To contribute a better user’s experience, the upcoming StackGres 1.1.0 release has added Babelfish, making it trivially easy to run Babelfish on Kubernetes.

If, for any reason, you don’t have a Kubernetes cluster handy; and/or you are not familiar with Kubernetes, please jump first to the Appendix in the last section, where I show how to get up & running a lightweight Kubernetes cluster in 1 minute.

Babelfish for PostgreSQL

Around one year ago, Amazon surprised us all by announcing Babelfish for PostgreSQL, a project that would bring a SQL Server compatibility layer on top of Postgres. Babelfish would become both an AWS managed service (on Aurora); as well as an open source project! I then blogged about it, knowing that this was a disruption point for Postgres. Babelfish enables Postgres to reach out to many other use cases, users and Communities: the SQL Server ecosystem.

Adding yet another capability to Postgres reflects on the thoughts shared by Stephen O’Grady on a recent post, A Return to the General Purpose Database. Postgres is not only a feature-full relational database; but with its extensions, it’s also a time-series database; a sharded database; a graph database; and now, also a SQL Server-compatible database. Postgres is, and will be, the unifying database for almost every imaginable database workload.

At StackGres we were almost literally eating our finger’s nails, waiting for Babelfish to be finally published as open source. It finally happened a couple of weeks ago. Since then, our team has been working tirelessly to give you an easy way to run Babelfish on Kubernetes, by integrating Babelfish in StackGres. Kudos to the whole team for such an amazing job!

Not only are we making Babelfish available on Kubernetes; but also giving to the Community the first (as far as we know) mechanism to run the open source Babelfish version without having to go through the (somehow involved) process of compiling it from the source code. With StackGres, you can go from zero to Babelfish in one command to install StackGres; and a 15-lines YAML to create a Babelfish-enabled cluster. It can’t hardly get easier than this.

Install StackGres 1.1.0-beta1 with Babelfish support

Installation is just one command. The recommended installation method is with Helm:

$ helm install --namespace stackgres --create-namespace stackgres \
 https://stackgres.io/downloads/stackgres-k8s/stackgres/1.1.0-beta1/helm/stackgres-operator.tgz

In some seconds / a minute you should have StackGres installed. If you see the StackGres ascii-art logo, you are ready to go!

Create a Babelfish cluster

Creating a Babelfish cluster is not so much different from creating a simple Postgres cluster. The few differences will be highlighted. Follow the instructions on either of the next two sections ("Using kubectl" or “Using the Web Console"), depending on your preferences. You can use either method interchangeably.

First, create a namespace:

$ kubectl create namespace notmssql

Using kubectl

Create the following 15-line file bbf.yaml with the following content:

kind: SGCluster
apiVersion: stackgres.io/v1
metadata:
namespace: notmssql
name: bbf
spec:
instances: 1
postgres:
version: 'latest'
flavor: babelfish
pods:
persistentVolume:
size: '5Gi'
nonProductionOptions:
enabledFeatureGates: [ "babelfish-flavor" ]

The main differences between creating a Babelfish cluster and a regular (vanilla Postgres) cluster are:

The .spec.postgres.flavor field must be set to babelfish. If unset, it is assumed the default “vanilla” Postgres flavor. Also note that the version: latest field automatically resolves to different versions, depending on the flavor (14.0 for the default Postgres flavor and 13.4 for Babelfish).
Babelfish is in preview mode in StackGres. It is not considered production ready. As such, you need to explicitly enable a feature gate named babelfish-flavor or cluster creation would be rejected by StackGres. This is what the two last lines do.

Finally apply it with:

$ kubectl apply -f bbf.yaml

In around a minute, you should get your Babelfish cluster running on StackGres! It wasn’t hard, was it? You can check that the cluster is up and running:

$ kubectl -n notmssql get pods
NAME READY STATUS RESTARTS AGE
bbf-0 6/6 Running 0 11m

Using the Web Console

StackGres comes bundled with a fully-featured Web Console, that it’s installed by default. For simplicity, let’s use the port-forwarding mechanism of kubectl to expose it on a port local to your computer:

$ WEBC=`kubectl --namespace stackgres get pods -l "app=stackgres-restapi" -o jsonpath="{.items[0].metadata.name}"`
$ kubectl --namespace stackgres port-forward "$WEBC" 8443:9443

Open in your web browser the address https://localhost:8443. You will see a warning (expected, the certificate is self-signed, you may also bring your own with custom Helm parameters). The default username is admin. The password is generated randomly and can be queried with the following command:

$ kubectl -n stackgres get secret stackgres-restapi --template '{{ printf "%s\n" (.data.clearPassword | base64decode) }}'

Select the notmssql namespace in the dropdown. And proceed to create a new SGCluster. Select the Babelfish flavor (you will see that the feature gate “Babelfish Flavor” becomes enabled). It should look like the following screenshot:

Connecting to your Babelfish cluster via the TDS protocol

Babelfish speaks the TDS protocol, like SQL Server. By default it listens in the same TCP port, 1433, and this is also what StackGres exposes.

To easily get connected using the TDS protocol, this StackGres release also includes with the postgres-util sidecar of every pod the command line utility usql, a database client that supports many databases, including SQL Server. By default, StackGres installation creates a database named babelfish, owned by a superuser named babelfish, that is initialized to be connected via the TDS protocol, with all the required Babelfish extensions already loaded. The password is generated randomly and can be obtained from the secret named after the cluster name:

$ kubectl -n notmssql get secret bbf --template '{{ printf "%s" (index .data "babelfish-password" | base64decode) }}'

Knowing the password, we can trivially connect to Babelfish with usql, using the SQL Server protocol, and run some example T-SQL queries:

$ kubectl -n notmssql exec -it bbf-0 -c postgres-util -- usql --password ms://babelfish@localhost
Enter password:
Connected with driver sqlserver (Microsoft SQL Server 12.0.2000.8, , Standard Edition)
Type "help" for help.
ms:babelfish@localhost=> select @@version;
version
----------------------------------------------------------------------
Babelfish for PostgreSQL with SQL Server Compatibility - 12.0.2000.8+
Nov 3 2021 09:55:42 +
Copyright (c) Amazon Web Services +
PostgreSQL 13.4 for Babelfish OnGres Inc. on x86_64-pc-linux-gnu
(1 row)
ms:babelfish@localhost=> create schema sch1;
CREATE SCHEMA
ms:babelfish@localhost=> create table [sch1].test (
ms:babelfish@localhost(> pk int primary key identity(1,1),
ms:babelfish@localhost(> text varchar(50),
ms:babelfish@localhost(> t datetime);
CREATE TABLE
ms:babelfish@localhost=> insert into [sch1].test (text, t) values ('hi', getdate());
INSERT 1
ms:babelfish@localhost=> select * from [sch1].test;
pk | text | t
----+------+-------------------------
1 | hi | 2021-11-14T23:26:05.45Z
(1 row)

Above are shown commands using the T-SQL syntax, and a connection over the TDS protocol, with specific SQL Server data types and functions. But there’s no SQL Server, only Postgres! It would be interesting to see how this is seen by Postgres? Let’s give it a try!

$ kubectl -n notmssql exec -it bbf-0 -c postgres-util -- psql babelfish
psql (13.4 OnGres Inc.)
Type "help" for help.
babelfish=# \d master_sch1.test
Table "master_sch1.test"
Column | Type | Collation | Nullable | Default
--------+-------------------+-----------------------+----------+------------------------------
pk | integer | | not null | generated always as identity
text | sys."varchar"(50) | bbf_unicode_cp1_ci_as | |
t | sys.datetime | | |
Indexes:
"test_pkey" PRIMARY KEY, btree (pk)
babelfish=# table master_sch1.test;
pk | text | t
----+------+-------------------------
1 | hi | 2021-11-14 23:26:05.449
(1 row)

That was interesting! You can see the obvious differences between the different syntax and data types.

Finally, let’s try to access Babelfish from a GUI tool. Firstly, let’s use port-forward to expose our pod’s port 1433 on our own laptop:

$ kubectl --namespace notmssql port-forward bbf-0 1433:1433

Now you can use GUI tools to connect as if you had SQL Server on your laptop. Some may not work yet, as Babelfish progresses. Below is a screenshot of how I connected with DBeaver:

Just please note to use the default’s master value for the Database/Schema. Querying the metadata table gave some errors, but general operation works well, and data is perfectly shown. Enjoy running T-SQL on your Babelfish for PostgreSQL.

Conclusion

Please be aware that this is a beta version of the upcoming 1.1.0 release, and as such it is not as polished as a GA release. Even on 1.1.0GA, Babelfish will not be declared “production ready”. Use it at your own risk.

However, it is quite interesting to start exploring Babelfish, given that StackGres makes it so simple to use. No need to go over the somehow involved compilation process that the current open source Babelfish release requires. Just a few commands, and you get a fully working Babelfish version on Kubernetes.

We’re eager to get feedback and ideas from you all. Please join our Slack and/or Discord channels and let you know what you think about Babelfish, StackGres, and chat about any other topic related to Postgres on Kubernetes, if you want.

Appendix: If you don’t have Kubernetes, get it in 1 minute

Not totally familiar with Kubernetes yet still want to easily try Babelfish? Keep reading, you just need one additional minute.

Run on your laptop:

$ curl -sfL https://get.k3s.io | sh -

This will get you running with K3s, a lightweight, certified Kubernetes distribution by SUSE Rancher. K3s will get started as a service on your system (managed by systemd; you can later stop with sudo systemctl stop k3s and/or uninstall with /usr/local/bin/k3s-uninstall.sh). To get access to kubectl, you can get it via the k3s binary:

$ sudo k3s kubectl
$ alias kubectl="sudo k3s kubectl" # optional

To check that the k3s cluster is running, you can run:

$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

You are ready to go!

StackGres 1.0.0: Open Source Postgres-aaS with 120+ Extensions

Thu, 28 Oct 2021 20:57:02 +0200

StackGres 1.0.0: Open Source Postgres-aaS with 120+ Extensions

TL;DR

We’re announcing StackGres 1.0.0 GA. StackGres is an Open Source Postgres-as-a-Service that runs on any Kubernetes environment. StackGres, today, is the Postgres platform with the most Postgres extensions available: 120 as of today. Many more to come in the future.

Why Kubernetes?

At OnGres we have been providing Postgres Professional Services for years. We love IaC (Infrastructure as Code). One of our realizations was that it was next to impossible to create a “single IaC package” that would contain Postgres and its Stack of components, that would work on any environment. There are just so many different compute, storage and networking components, with different APIs.

So we asked ourselves: is there any “abstraction layer” for distributed infrastructure? The answer was Kubernetes. Kubernetes APIs allowed us to create a single deployable Postgres package, that contains Postgres itself and all its necessary Stack, and run –almost– anywhere.

StackGres, a feature-rich Postgres Platform

StackGres contains the full Stack of Postgres components needed for production:

High availability and automated failover, using Patroni. Managed DNS endpoints for the primary (rw) and replicas (ro, load balanced) connections.
Built-in, enabled by default connection-pooling.
Configurable automated backups with retention policies.
Prometheus metrics exporter, with auto-binding. Customized Postgres metrics, Grafana dashboards and Postgres alerts.
Expertly tuned default configurations –which you can optionally override.

StackGres: innovating the Postgres Platform

So far, the features mentioned may be “table stakes” as of today. We wanted to provide more. StackGres has introduced significant innovations –while keeping the same Postgres core boring– such as:

Using Envoy to proxy all Postgres traffic. In collaboration with the Envoy Community, we developed the Postgres filter for Envoy. StackGres uses it to add additional monitoring (by inspecting the wire protocol) and to terminate SSL!
A fully-featured Web Console. It supports everything StackGres can do. And yes, it has a dark mode ;)

A system to push all Postgres and Patroni logs from all pods to a central server. This central server is a separate, StackGres-managed Postgres database with Timescale to support large volume and high velocity logging. Query your logs with SQL or from the Web Console!
Fully automated “Day 2” DBA operations, including: minor and major version upgrades, controlled cluster restart, container version upgrades, vacuum, repack and even benchmarks!
A system to dynamically load Postgres extensions into the containers. This allows us to ship lighter containers, while supporting potentially hundreds of extensions –120 as of now, with many more to come in the future.

Easy of installation and use

Production Postgres requires a fair amount of Postgres expertise. One of our goals is to “democratize production Postgres”, and make it accessible to almost anyone. StackGres is designed to be easy to operate.

Visit StackGres Installation page for a quick primer on how to install StackGres either via kubectl or Helm (Helm Chart also published to ArtifactHub, please star it!). There is more detailed information in the Production Installation section of the documentation.

For this post we will use Helm installation. Add the StackGres Helm repository:

helm repo add stackgres-charts https://stackgres.io/downloads/stackgres-k8s/stackgres/helm/

And install with Helm into a dedicated namespace:

helm install --namespace stackgres --create-namespace \
 stackgres stackgres-charts/stackgres-operator

The above command will normally take 0-2 minutes. Now let’s create our first cluster. If you prefer to do it all from the Web Console, skip to the next section. Create a simple YAML file named demo.yamlwith the following content:

apiVersion: stackgres.io/v1
kind: SGCluster
metadata:
name: sgdemo
spec:
postgres:
version: 'latest'
instances: 1
pods:
persistentVolume:
size: '5Gi'

Then kubectl apply the file:

kubectl apply -f demo.yaml

The easiest way to connect is to run psql within the postgres-util container, an administration container that is always started with your StackGres pods:

kubectl exec -it sgdemo-0 -c postgres-util -- psql

You may also connect from any other pod within the cluster. Note that the default username is postgres and the password is randomly generated and written in a secret named as the cluster, with the key superuser-password.

Adding an extension!

StackGres supports as of today more than 120 extensions, with many more to come in the future.

As mentioned, StackGres loads extensions dynamiycally. This means that save a few that are required for internal operation, your cluster will contain by default no other extension. You can check by running the select * from pg_available_extensions query from within psql. This is good! Containers are lighter and are exposed to fewer potential security problems.

But now let’s say you want to use your favorite extension, for example citext. This is a very useful extension to deal with text in a case insensitive manner. To add this extension to the live container, you just need to edit the .spec.postgres section of the SGCluster as in:

...
spec:
postgres:
version: 'latest'
extensions:
- name: 'citext'
...

(you can either edit the original YAML file and then do kubectl apply -f demo.yaml or directly via kubectl edit sgcluster demo)

In a few seconds, the extension will be available!. You may also do it from the Web Console by editing the cluster and selecting the extension(s) that you want to lead from the Web Console. Now, to use the extension, you just need to run in psql, as with any Postgres extension:

create extension citext;

To release the resources, simply run kubectl delete -f demo.yaml.

Using the Web Console

As mentioned, StackGres comes with a fully-featured Web Console. Let’s check it out. Let’s identify the pod name where the Web Console is running and do a port-forward to expose it in our laptop:

WEBC=`kubectl --namespace stackgres get pods -l "app=stackgres-restapi" -o jsonpath="{.items[0].metadata.name}"`
kubectl --namespace stackgres port-forward "$WEBC" 8443:9443

For production usage, it is recommended to use a LoadBalancer as part of the installation Helm parameters or setup an Ingress controller. Check the Installation via Helm section of the documentation for more information.

Open in your web browser the address https://localhost:8443/. You will see a warning (expected, the certificate is self-signed, you may also bring your own with custom Helm parameters). The default username is admin. The password is generated randomly and can be queried with the following command:

kubectl -n stackgres get secret stackgres-restapi --template '{{ printf "%s\n" (.data.clearPassword | base64decode) }}'

Select a namespace to work on (you may use default, or create one with kubectl create namespace demo). On the left pane hover over StackGres Clusters and click on the (+) icon to create a new cluster. You may create a simple cluster as in the following example:

Once it is creating, you may click on the name to see the details about the cluster. Check the View Connection Info link to the top right of the cluster information to see quick commands on how to connect to your new cluster.

Next steps: tutorial

Want to go deeper? Check our tutorial, which explores the main StackGres features in a little bit more detail.

Please drop us a note and let us know what you think in our Slack and/or Discord channels. Also write us there if you find any bug or want to become a collaborator.