This runbook will show you how to perform a volume downsize. The usual operation is to extend a volume, but in some cases you might have over-dimensioned your volumes and might need to downsize your volumes, in order to reduce costs.
Assume you have a StackGres cluster with:
3ongres-dbongres-db20Gi$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.11:7433 | Leader | running | 3 | |
| ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | | running | 3 | 0 |
+-------------+----------------+--------+---------+----+-----------+
Verify the PVC’s:
$ kubectl get pvc -n ongres-db
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 162m
ongres-db-data-ongres-db-0 Bound pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f 20Gi RWO gp2-data 11m
ongres-db-data-ongres-db-1 Bound pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44 20Gi RWO gp2-data 10m
ongres-db-data-ongres-db-2 Bound pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO gp2-data 4m47s
Assuming the disk size is over-dimensioned, and you need to perform a downsize to 15Gi.
Perform a switchover to the pod with the higher index number (ongres-db-2).
Execute:
kubectl exec -it -n ongres-db ongres-db-0 -c patroni -- patronictl switchover
Master [ongres-db-0]:
Candidate ['ongres-db-1', 'ongres-db-2'] []: ongres-db-2
When should the switchover take place (e.g. 2021-01-15T16:40 ) [now]:
Current cluster topology
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.11:7433 | Leader | running | 3 | |
| ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | | running | 3 | 0 |
+-------------+----------------+--------+---------+----+-----------+
Are you sure you want to switchover cluster ongres-db, demoting current primary ongres-db-0? [y/N]:y
2021-01-15 15:41:11.93457 Successfully switched over to "ongres-db-2"
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.11:7433 | | stopped | | unknown |
| ongres-db-1 | 10.0.0.10:7433 | | running | 3 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | Leader | running | 3 | |
+-------------+----------------+--------+---------+----+-----------+
Now, check the cluster state:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.11:7433 | | running | 4 | 0 |
| ongres-db-1 | 10.0.0.10:7433 | | running | 4 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | |
+-------------+----------------+--------+---------+----+-----------+
Edit the StackGres cluster volume definition to the new size:
kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/pods/persistentVolume/size", "value": "15Gi" }]'
You’ll get the following message:
sgcluster.stackgres.io/ongres-db patched
The operator does not validate the change of the persistent volume size: whether a volume can be expanded or shrunk is up to Kubernetes and the StorageClass implementation. Since the volume claim templates of a StatefulSet can not be changed, the operator recreates the StatefulSet with the new volume size, keeping the existing Pods and PersistentVolumeClaims. The existing PersistentVolumeClaims keep their size, only the ones created from now on get the new size.
Verify that the stateful set now has the new volume size:
$ kubectl describe sts -n ongres-db ongres-db | grep -i capacity
Capacity: 15Gi
Edit the replica size to 1:
$ kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/instances", "value": 1 }]'
Once you decrease the replicas, you’ll see something like:
$ kubectl get pods -n ongres-db
NAME READY STATUS RESTARTS AGE
distributedlogs-0 2/2 Running 0 3h4m
ongres-db-2 6/6 Running 0 27m
Proceed to delete the unused PVCs ongres-db-data-ongres-db-0 and ongres-db-data-ongres-db-1:
$ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-0
persistentvolumeclaim "ongres-db-data-ongres-db-0" deleted
$ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-1
persistentvolumeclaim "ongres-db-data-ongres-db-1" deleted
This will release the persistent volumes and then you can proceed to delete them:
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO Retain Bound ongres-db/ongres-db-data-ongres-db-2 gp2-data 32m
pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO Delete Bound ongres-db/distributedlogs-data-distributedlogs-0 standard 3h10m
pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f 20Gi RWO Retain Released ongres-db/ongres-db-data-ongres-db-0 gp2-data 39m
pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44 20Gi RWO Retain Released ongres-db/ongres-db-data-ongres-db-1 gp2-data 38m
Delete the disks with Released status:
$ kubectl delete pv pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f
persistentvolume "pvc-a2aa5198-c553-4e0d-a1e1-914669abb69f" deleted
$ kubectl delete pv pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44
persistentvolume "pvc-c724b2bf-cf17-4f57-a882-3a5da6947f44" deleted
Increase the replica size to 2:
$ kubectl patch sgclusters -n ongres-db ongres-db --type='json' -p '[{ "op": "replace", "path": "/spec/instances", "value": 2 }]'
Now, your cluster will have 2 pods:
$ kubectl get pods -n ongres-db
NAME READY STATUS RESTARTS AGE
distributedlogs-0 2/2 Running 0 3h15m
ongres-db-0 6/6 Running 0 49s
ongres-db-2 6/6 Running 0 37m
Check again the cluster state:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl list
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.12:7433 | | running | 4 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | |
+-------------+----------------+--------+---------+----+-----------+
And the new pod will have the new disk size:
$ kubectl get pvc -n ongres-db
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 3h17m
ongres-db-data-ongres-db-0 Bound pvc-37d96872-b132-4a89-a579-d87f8cf1fa92 15Gi RWO gp2-data 2m47s
ongres-db-data-ongres-db-2 Bound pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148 20Gi RWO gp2-data 39m
Perform another switchover, this time to node ongres-db-0:
$ kubectl exec -it -n ongres-db ongres-db-2 -c patroni -- patronictl switchover
Master [ongres-db-2]:
Candidate ['ongres-db-0'] []: ongres-db-0
When should the switchover take place (e.g. 2021-01-15T17:12 ) [now]:
Current cluster topology
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.12:7433 | | running | 4 | 0 |
| ongres-db-2 | 10.0.6.9:7433 | Leader | running | 4 | |
+-------------+----------------+--------+---------+----+-----------+
Are you sure you want to switchover cluster ongres-db, demoting current primary ongres-db-2? [y/N]: y
2021-01-15 16:12:57.14561 Successfully switched over to "ongres-db-0"
+ Cluster: ongres-db (6918002883456245883) -------+----+-----------+
| Member | Host | Role | State | TL | Lag in MB |
+-------------+----------------+--------+---------+----+-----------+
| ongres-db-0 | 10.0.7.12:7433 | Leader | running | 4 | |
| ongres-db-2 | 10.0.6.9:7433 | | stopped | | unknown |
+-------------+----------------+--------+---------+----+-----------+
This will delete the pod ongres-db-2 and create the pod ongres-db-1
NAME READY STATUS RESTARTS AGE
distributedlogs-0 2/2 Running 0 3h19m
ongres-db-0 6/6 Running 0 4m51s
ongres-db-1 6/6 Running 0 41s
You can proceed to delete the PVC and PV of ongres-db-2
$ kubectl delete pvc -n ongres-db ongres-db-data-ongres-db-2
persistentvolumeclaim "ongres-db-data-ongres-db-2" deleted
$ kubectl delete pv pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148
persistentvolume "pvc-5124b9d2-ec35-46d7-9eda-7543d9ed7148" deleted
Now, your cluster will have the new, reduced disk size:
$ kubectl get pvc -n ongres-db
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
distributedlogs-data-distributedlogs-0 Bound pvc-9bab7a68-a209-4d9a-93f7-871a217a28b1 50Gi RWO standard 3h24m
ongres-db-data-ongres-db-0 Bound pvc-37d96872-b132-4a89-a579-d87f8cf1fa92 15Gi RWO gp2-data 9m21s
ongres-db-data-ongres-db-1 Bound pvc-46c1433b-26e8-422c-aecf-145b1bb5aac1 15Gi RWO gp2-data 5m11s