Debug School

Cover image for OpenShift Storage Config
Suyash Sambhare
Suyash Sambhare

Posted on

OpenShift Storage Config

You can expand and modify your cluster to your specifications, including storage settings, after installing OpenShift Container Platform. Containers use temporary local storage, often known as ephemeral storage, by default. There is a lifespan restriction on the ephemeral storage. You need to set up persistent storage if you want the data to be kept for a long time. You can use one of the following techniques to configure storage:

  • Dynamic provisioning: By designing and building storage classes that manage various levels of storage, including storage access, you can dynamically provision storage on-demand.
  • Static provisioning: A cluster can access existing storage by using Kubernetes persistent volumes. Different device configurations and mount options can be supported using static provisioning.

Metrics are unstable when stored in files with the ReadWriteMany (RWX) access mode. If you use file storage, do not enable RWX access mode on any persistent volume claims (PVCs) set up for use with metrics. Review the recommended storage solution for logging in the section Configuring persistent storage for the log store. Using NFS storage as a persistent volume or NAS, such as Gluster, might cause data corruption. As a result, OpenShift Container Platform Logging does not support NFS for Elasticsearch storage or the LokiStack log store. For each log store, only one persistent volume type is allowed. The OpenShift Container Platform’s PVs or PVC does not consume Object storage. Apps must work with the object storage REST API.

Specific application storage recommendations

There are some performance issues when using the NFS server on Red Hat Enterprise Linux (RHEL) as a storage backend for key services. This contains OpenShift Container Registry and Quay, Prometheus for storage monitoring, and Elasticsearch for storage logging. Therefore, utilizing RHEL NFS to back up PVs utilized by core services is not recommended. Other NFS implementations in the marketplace may not have these concerns. You can find out specific NFS implementation vendor for more information on any testing that may have been performed on these OpenShift Container Platform fundamental components.

Registry

  • The technology used for storage must provide read-after-write consistency.
  • RWX access mode need not be supported.
  • Block storage is the second most used type of storage technology, behind object storage.
  • When deploying OpenShift image registry clusters with production workloads, file storage is not advised.

Scaled registry

  • The storage technology which is to be used must support the RWX access mode.
  • The storage technology in question must be able to provide read-after-write consistency.
  • The ideal storage technology is object storage.
  • Red Hat OpenShift Data Foundation (ODF), Amazon Simple Storage Service (Amazon S3), Google Cloud Storage (GCS), Microsoft Azure Blob Storage, and OpenStack Swift are all compatible.
  • Object storage must be S3 or Swift compliant.
  • The sole modifiable technology for non-cloud platforms, such as vSphere and bare metal installs, is file storage.
  • Block storage is not customizable.
  • OpenShift Container Platform supports network file system (NFS) storage. However, using NFS storage with a scaled registry can result in recognized difficulties.

Metrics

  • Block storage is the most common type of storage technology.
  • Object storage is not customizable.
  • File storage is not recommended for a hosted metrics cluster deployment that includes production workloads.

Logging

  • Loki Operator: The preferred storage technology is S3 compatible Object storage. Block storage is not configurable.
  • OpenShift Elasticsearch Operator: The preferred storage technology is block storage. Object storage is not supported.

Storage

Dynamic provisioning

Dynamic Provisioning allows you to generate storage volumes as needed, reducing the need for cluster administrators to pre-provision storage.
The StorageClass resource object specifies and categorizes storage that can be requested, as well as providing a way to pass specifications for dynamically provisioned storage on demand. StorageClass objects can also be used to manage various degrees of storage and access to the storage. Cluster Administrators (cluster-admin) and Storage Administrators (storage-admin) define and generate the StorageClass objects that users can request without knowing anything about the underlying storage volume sources.

This feature and the ability for administrators to provision a cluster with persistent storage are made possible by the OpenShift Container Platform persistent volume framework. Additionally, consumers can request those resources using the framework without needing to understand the underlying infrastructure.
The OpenShift Container Platform supports a wide range of storage types as persistent volumes. Some forms of storage are produced dynamically using the built-in provider and plugin APIs, but all of them can be statically provided by an admin.

Defining a storage class

StorageClass objects are currently globally scoped and may only be created by cluster or storage administrators.
The Cluster Storage Operator may install a default storage class depending on the platform being used. The Operator owns and controls this storage class. It cannot be deleted or updated except to define annotations and labels. If you want different behaviour, you must create a custom storage class.

Basic StorageClass object definition

The following resource shows the parameters and default values that you use to configure a storage class.

kind: StorageClass # The API object type.
apiVersion: storage.k8s.io/v1 # The current apiVersion.
metadata:
  name: <storage-class-name> # The name of the storage class.
  annotations: # Annotations for the storage class.
    storageclass.kubernetes.io/is-default-class: 'true'
    ...
provisioner: kubernetes.io/aws-ebs # The type of provisioner associated with this storage class.
parameters: # The parameters required for the specific provisioner; this will change from plug-in to plug-in. 
  type: gp3
Enter fullscreen mode Exit fullscreen mode

Storage class annotations

Set a storage class as the cluster-wide default, add the following annotation to your storage class metadata: storageclass.kubernetes.io/is-default-class: "true"

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
Enter fullscreen mode Exit fullscreen mode

This allows any permanent volume claim (PVC) that does not specify a specific storage class to be automatically provisioned using the default storage class. However, your cluster can have many storage classes, but only one can be the default storage class.
The beta annotation storageclass.beta.kubernetes.io/is-default-class is still functional, however it will be deleted in a future release.

Include a storage class description, use the following annotation in your storage class metadata: kubernetes.io/description: My Storage Class Description

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubernetes.io/description: My Storage Class Description
Enter fullscreen mode Exit fullscreen mode

AWS Elastic Block Store (EBS) object definition

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: <storage-class-name> # Name of the storage class. The persistent volume claim uses this storage class for provisioning the associated persistent volumes.
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 # Select from io1, gp3, sc1, st1. The default is gp3.
  iopsPerGB: "10" # Only for io1 volumes. I/O operations per second per GiB. The AWS volume plugin multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS.
  encrypted: "true" # Denotes whether to encrypt the EBS volume. Valid values are true or false.
  kmsKeyId: key value # The full ARN of the key to use when encrypting the volume. If none is supplied, but encrypted is set to true, then AWS generates a key.
  fsType: ext4 # File system that is created on dynamically provisioned volumes. This value is copied to the fsType field of dynamically provisioned persistent volumes, and the file system is created when the volume is mounted for the first time. The default value is ext4.
Enter fullscreen mode Exit fullscreen mode

Modifying the default storage class

Use the following procedure to change the default storage class. If you have two defined storage classes, gp3 and standard, and you want to change the default storage class from gp3 to standard.

oc get storageclass
NAME                 TYPE
gp3 (default)        kubernetes.io/aws-ebs 
standard             kubernetes.io/aws-ebs
Enter fullscreen mode Exit fullscreen mode

For the desired storage class, set the storageclass.kubernetes.io/is-default-class annotation to true by running the following command:

oc patch storageclass standard -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

You can have different default storage classes for a limited period. However, you should verify that just one default storage class exists in the end.
When multiple default storage classes are present, any persistent volume claim (PVC) requesting the default storage class (pvc.spec.storageClassName=nil) receives the most recently created default storage class, regardless of its default status, and the administrator receives an alert in the alerts dashboard indicating that there are multiple default storage classes, MultipleDefaultStorageClasses.
Remove the default storage class setting from the previous default storage class.
For the old default storage class, change the value of the storageclass.kubernetes.io/is-default-class annotation to false by running the following command:

oc patch storageclass gp3 -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

$ oc get storageclass

NAME                 TYPE
gp3                  kubernetes.io/aws-ebs
standard (default)   kubernetes.io/aws-ebs
Enter fullscreen mode Exit fullscreen mode

Ref: https://docs.openshift.com/container-platform/4.16/post_installation_configuration/post-install-storage-configuration.html

Top comments (0)