prometheusで下記エラーが発生したのでPVCを拡張する。
ts=2022-05-19T17:00:05.979Z caller=head.go:518 level=error component=tsdb msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: corruption in head chunk file /prometheus/chunks_head/000105: head chunk file doesn't include enough bytes to read the chunk header - required:27537424, available:27537408, file:105"
ts=2022-05-19T17:02:04.664Z caller=db.go:824 level=error component=tsdb msg="compaction failed" err="compact head: persist head block: populate block: write chunks: preallocate: no space left on device"
拡張手順
Pod(prometheus-kube-prom-stack-kube-prome-prometheus-0)の停止
kubectl -n monitoring scale sts prometheus-kube-prom-stack-kube-prome-prometheus --replicas 0
kubectl editでPVCサイズ拡張
kubectl -n monitoring edit pvc prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0
spec:resources:storageを変更(5Gi->8Gi)
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8Gi
storageClassName: do-block-storage
volumeMode: Filesystem
volumeName: pvc-329d74f1-3e0f-4395-9063-64905805e5be
helm chart(prometheus:prometheusSpec:storageSpec)変更。
prometheus:
...
prometheusSpec:
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: do-block-storage
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 8Gi
helm upgrading
helm upgrade kube-prom-stack prometheus-community/kube-prometheus-stack --version "32.4.0" --namespace monitoring -f dok8s-prom-stack-values.yaml
pvcステータス確認、8Giに拡張されている。
kubectl -n monitoring get pvc prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0 Bound pvc-329d74f1-3e0f-4395-9063-64905805e5be 8Gi RWO do-block-storage 10d
DigtitalOcean Volumeステータス確認、Sizeが8GiBになっている。
doctl compute volume ls pvc-329d74f1-3e0f-4395-9063-64905805e5be
ID Name Size Region Filesystem Type Filesystem Label Droplet IDs Tags
ce26988f-d0ba-11ec-9815-0a58ac14a124 pvc-329d74f1-3e0f-4395-9063-64905805e5be 8 GiB sgp1 ext4 [298622529] k8s:38b25270-16ca-4cea-b545-886b87cb9bc9