Resizing PVC(kube-prom-stack)

prometheusで下記エラーが発生したのでPVCを拡張する。

ts=2022-05-19T17:00:05.979Z caller=head.go:518 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: corruption in head chunk file /prometheus/chunks_head/000105: head chunk file doesn't include enough bytes to read the chunk header - required:27537424, available:27537408, file:105"
ts=2022-05-19T17:02:04.664Z caller=db.go:824 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"

拡張手順

Pod(prometheus-kube-prom-stack-kube-prome-prometheus-0)の停止

kubectl -n monitoring scale sts prometheus-kube-prom-stack-kube-prome-prometheus --replicas 0

kubectl editでPVCサイズ拡張

kubectl -n monitoring edit pvc prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0

spec:resources:storageを変更(5Gi->8Gi)

spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: do-block-storage
  volumeMode: Filesystem
  volumeName: pvc-329d74f1-3e0f-4395-9063-64905805e5be

helm chart(prometheus:prometheusSpec:storageSpec)変更。

prometheus:
...
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: do-block-storage
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 8Gi

helm upgrading

helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack --version "32.4.0" --namespace monitoring -f dok8s-prom-stack-values.yaml

pvcステータス確認、8Giに拡張されている。

kubectl -n monitoring get pvc prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0
NAME                                                                                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0   Bound    pvc-329d74f1-3e0f-4395-9063-64905805e5be   8Gi        RWO            do-block-storage   10d

DigtitalOcean Volumeステータス確認、Sizeが8GiBになっている。

doctl compute volume ls pvc-329d74f1-3e0f-4395-9063-64905805e5be
ID                                      Name                                        Size     Region    Filesystem Type    Filesystem Label    Droplet IDs    Tags
ce26988f-d0ba-11ec-9815-0a58ac14a124    pvc-329d74f1-3e0f-4395-9063-64905805e5be    8 GiB    sgp1      ext4                                   [298622529]    k8s:38b25270-16ca-4cea-b545-886b87cb9bc9

コメントを残す

メールアドレスが公開されることはありません。 が付いている欄は必須項目です