Skip to content

Minio

What is it?

Minio is a high-performance, S3 compatible object store. It can be deployed on a wide variety of platforms, and it comes in multiple flavors.

Why this guide?

This guide is a very quick way of deploying the community version of Minio in order to quickly setup a fully standalone Object Store, in an OpenShift Cluster. This can then be used for various prototyping tasks that require Object Storage.

Note that nothing in this guide should be used in production-grade environments. Also, Minio is not included in RHOAI, and Red Hat does not provide support for Minio.

Pre-requisites

  • Access to an OpenShift cluster
  • Namespace-level admin permissions, or permission to create your own project

Deploying Minio on OpenShift

Create a Data Science Project (Optional)

If you already have your own Data Science Project, or OpenShift project, you can skip this step.

  1. If your cluster already has Red Hat OpenShift AI installed, you can use the Dashboard Web Interface to create a Data Science project.
  2. Simply navigate to Data Science Projects
  3. And click Create Project

  4. Choose a name for your project (here, Showcase) and click Create:

    alt_text

  5. Make sure to make a note of the Resource name, in case it's different from the name.

Log on to your project in OpenShift Console

  1. Go to your cluster's OpenShift Console:

    alt_text

  2. Make sure you use the Administrator view, not the developer view.

  3. Go to Workloads then Pods, and confirm the selected project is the right one

    alt_text

  4. You now have a project in which to deploy Minio

Deploy Minio in your project

  1. Click on the + ("Import YAML") button:

    alt_text

  2. Paste the following YAML in the box, but don't press ok yet!: 

    ---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: minio-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  volumeMode: Filesystem
---
kind: Secret
apiVersion: v1
metadata:
  name: minio-secret
stringData:
  # change the username and password to your own values.
  # ensure that the user is at least 3 characters long and the password at least 8
  minio_root_user: minio
  minio_root_password: minio123
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: minio
spec:
  replicas: 1
  selector:
    matchLabels:
      app: minio
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: minio
    spec:
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: minio-pvc
      containers:
        - resources:
            limits:
              cpu: 250m
              memory: 1Gi
            requests:
              cpu: 20m
              memory: 100Mi
          readinessProbe:
            tcpSocket:
              port: 9000
            initialDelaySeconds: 5
            timeoutSeconds: 1
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          terminationMessagePath: /dev/termination-log
          name: minio
          livenessProbe:
            tcpSocket:
              port: 9000
            initialDelaySeconds: 30
            timeoutSeconds: 1
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 3
          env:
            - name: MINIO_ROOT_USER
              valueFrom:
                secretKeyRef:
                  name: minio-secret
                  key: minio_root_user
            - name: MINIO_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: minio-secret
                  key: minio_root_password
          ports:
            - containerPort: 9000
              protocol: TCP
            - containerPort: 9090
              protocol: TCP
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: data
              mountPath: /data
              subPath: minio
          terminationMessagePolicy: File
          image: >-
            quay.io/minio/minio:RELEASE.2023-06-19T19-52-50Z
          args:
            - server
            - /data
            - --console-address
            - :9090
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      schedulerName: default-scheduler
  strategy:
    type: Recreate
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
---
kind: Service
apiVersion: v1
metadata:
  name: minio-service
spec:
  ipFamilies:
    - IPv4
  ports:
    - name: api
      protocol: TCP
      port: 9000
      targetPort: 9000
    - name: ui
      protocol: TCP
      port: 9090
      targetPort: 9090
  internalTrafficPolicy: Cluster
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app: minio
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: minio-api
spec:
  to:
    kind: Service
    name: minio-service
    weight: 100
  port:
    targetPort: api
  wildcardPolicy: None
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: minio-ui
spec:
  to:
    kind: Service
    name: minio-service
    weight: 100
  port:
    targetPort: ui
  wildcardPolicy: None
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect

  3. By default, the size of the storage is 20 GB. (see line 11). Change it if you need to.

  4. If you want to, edit lines 21-22 to change the default user/password.
  5. Press Create.

  6. You should see:

    alt_text

  7. And there should now be a running minio pod:

    alt_text

  8. As well as two minio routes:

    alt_text

  9. The -api route is for programmatic access to Minio

  10. The -ui route is for browser-based access to Minio
  11. Your Minio Object Store is now deployed, but we still need to create at least one bucket in it, to make it useful.

Creating a bucket in Minio

Log in to Minio

  1. Locate the minio-ui Route, and open its location URL in a web browser:

  2. When prompted, log in

    • if you kept the default values, then:
    • user: minio
    • pass: minio123

    alt_text

  3. You should now be logged into your Minio instance.

Create a bucket

  1. Click on Create a Bucket

    alt_text

  2. Choose a name for your bucket (for example mybucket) and click Create Bucket:

    alt_text

  3. Repeat those steps to create as many buckets as you will need.

Create a matching Data Connection for Minio

  1. Back in RHOAI, inside of your Data Science Project, Click on Add data connection:

    alt_text

  2. Then, fill out the required field to match with your newly-deployed Minio Object Storage

    alt_text

  3. You now have a Data Connection that maps to your mybucket bucket in your Minio Instance.

  4. This data connection can be used, among other things
    • In your Workbenches
    • For your Model Serving
    • For your Pipeline Server Configuration

Validate

To test if everything is working correctly, you can access the workbench associated with your Data Connection and run the following commands (i.e., inside a Jupyter notebook):

  1. Install and import MinIO Python Client SDK

    !pip install minio

    from minio import Minio
from minio.error import S3Error
import  os
import datetime

  2. Access Data Connection properties as environment variables:

    # MinIO client doesn't like URLs with procotol/schema, so use
# yourendpoint.com instead of https://yourtendpoint.com
AWS_S3_ENDPOINT = os.getenv("AWS_S3_ENDPOINT")
AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")
AWS_S3_BUCKET = os.getenv("AWS_S3_BUCKET")

  3. Create the MinIO client

    # Create the MinIO client
client = Minio(
    AWS_S3_ENDPOINT,
    access_key=AWS_ACCESS_KEY_ID,
    secret_key=AWS_SECRET_ACCESS_KEY,
    secure=True  # Set to True if you are using HTTPS
)

  4. Test the connection by listing all buckets

    #List all buckets
try:
    buckets = client.list_buckets()
    for bucket in buckets:
        print(bucket.name, bucket.creation_date)
except S3Error as e:
    print("Error occurred: ", e)

  5. Create a sample local file

    # Create File
FILE_ON_DISK = 'file.txt'

file = open(f"{FILE_ON_DISK}", "w")
file.write('Hello there %s recorded at %s.\n' % (FILE_ON_DISK, datetime.datetime.now()))
file.close()

  6. Upload a file to MinIO

    # Upload a File 
file_path = FILE_ON_DISK
object_name = 'target-file.txt'

try:
    client.fput_object(AWS_S3_BUCKET, object_name, file_path)
    print(f"'{object_name}' is successfully uploaded as object to bucket '{bucket_name}'.")
except S3Error as e:
    print("Error occurred: ", e)

  7. Download a file from MinIO

    # Download a file 
object_name = 'target-file.txt'
file_path = 'file-froms3.txt'

try:
 client.fget_object(AWS_S3_BUCKET, object_name, file_path)
 print(f"'{object_name}' is successfully downloaded to '{file_path}'.")
except S3Error as e:
 print("Error occurred: ", e)

  8. List objects in our bucket

    # Download a file 
object_name = 'target-file.txt'
file_path = 'file-froms3.txt'

try:
    client.fget_object(AWS_S3_BUCKET, object_name, file_path)
    print(f"'{object_name}' is successfully downloaded to '{file_path}'.")
except S3Error as e:
    print("Error occurred: ", e)

For more complete and detailed information about MinIO Python Client SDK usage, please check the official documentation.

Notes and FAQ

  • As long as you are using the Route URLs, a Minio running in one namespace can be used by any other application, even running in another namespace, or even in another cluster altogether.

Uninstall instructions:

This will completely remove Minio and all its content. Make sure you have a backup of the things your need before doing so!

  1. Track down those objects created earlier:

    alt_text

  2. Delete them all.