Walkthrough: Deploying a Flask app with Redis Queue (RQ) Workers and Dashboard using Kubernetes

In this article, we walkthrough the steps to deploy a simple Flask app together with Redis Queue (RQ), a minimalistic job queue built on Redis, using Kubernetes to provision and manage the necessary deployments and services for our stack.

../../images/rq-dashboard.png

1   Getting Started

Throughout this guide, we use a local single-node Kubernetes cluster that is provisioned and managed by Minikube. The methods described herein can be trivially adapted to Kubernetes cluster on other cloud platforms such as Amazon AWS ECS, Google Container Engine (GKE). The key difference is that the Minikube Kubernetes cluster does't support LoadBalancer's, so services must be exposed with NodePort. Both AWS EC2 and GKE support LoadBalancer's, so if you are using either of these platforms, or any others that support it, you should specify LoadBalancer as the service type instead of NodePort.

That said, I still highly recommend using a local Minikube Kubernetes cluster for more efficient development workflows, since it saves you having to wait ~10 minutes for load balancers to come alive, as is the case with AWS EC2. Let's go ahead and get started by bringing up our local Kubernetes cluster:

$ minikube start
Starting local Kubernetes cluster...
Kubernetes is available at https://192.168.99.101:443.

Let's create and navigate to the directory for the files relevant to this deployment:

$ mkdir flask-redis-queue
$ cd flask-redis-queue

This guide assumes some familiarity with Flask integrated with RQ and will focus more on provisioning the various components using Docker and Kubernetes. The post Implementing a Redis Task Queue from the Flask by Example blog post series provides a good starting point and should get you up to speed.

2   Deploy the Redis master

2.1   redis-master-deployment.yaml

$ kubectl run redis-master --image=redis --replicas=1 --port=6379 \
>                          --labels='app=redis,role=master,tier=backend' \
>                          --requests='cpu=100m,memory=100Mi' \
>                          --dry-run --output=yaml > redis-master-deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: redis
    role: master
    tier: backend
  name: redis-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
      role: master
      tier: backend
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
        role: master
        tier: backend
    spec:
      containers:
      - image: redis
        name: redis-master
        ports:
        - containerPort: 6379
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
status: {}
$ kubectl create -f redis-master-deployment.yaml
deployment "redis-master" created

2.2   redis-master-service.yaml

$ kubectl expose deployment redis-master --selector='app=redis,role=master,tier=backend' \
>                                        --dry-run --output=yaml > redis-master-service.yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: redis
    role: master
    tier: backend
  name: redis-master
spec:
  ports:
  - port: 6379
    protocol: TCP
    targetPort: 6379
  selector:
    app: redis
    role: master
    tier: backend
status:
  loadBalancer: {}
$ kubectl create -f redis-master-service.yaml
service "redis-master" created

3   Deploy the Flask web app

$ mkdir web-flask

3.1   requirements.txt

$ cat > web-flask/requirements.txt
flask
redis
rq

3.2   settings.py

$ $EDITOR web-flask/settings.py
import os

REDIS_HOST = os.environ['REDIS_MASTER_SERVICE_HOST'] \
          if os.environ.get('GET_HOSTS_FROM', '') == 'env' else 'redis-master'
REDIS_PORT = 6379

3.3   app.py

$ $EDITOR web-flask/app.py
from flask import Flask, jsonify, request
from redis import StrictRedis
from rq import Queue

from random import randrange

from settings import REDIS_HOST, REDIS_PORT


app = Flask(__name__)

q = Queue(connection=StrictRedis(host=REDIS_HOST, port=REDIS_PORT))


@app.route('/')
def get_randrange():

    if 'stop' in request.args:

        stop = int(request.args.get('stop'))
        start = int(request.args.get('start', 0))
        step = int(request.args.get('step', 1))

        job = q.enqueue(randrange, start, stop, step, result_ttl=5000)

        return jsonify(job_id=job.get_id())

    return 'Stop value not specified!', 400


@app.route("/results")
@app.route("/results/<string:job_id>")
def get_results(job_id=None):

    if job_id is None:
        return jsonify(queued_job_ids=q.job_ids)

    job = q.fetch_job(job_id)

    if job.is_failed:
        return 'Job has failed!', 400

    if job.is_finished:
        return jsonify(result=job.result)

    return 'Job has not finished!', 202

if __name__ == '__main__':
    # Start server
    app.run(host='0.0.0.0', port=8080, debug=True)

3.4   Dockerfile

$ $EDITOR web-flask/Dockerfile

Attention!

Not suitable for production!

FROM python:3.5.1-onbuild

EXPOSE 8080
CMD ["python", "app.py"]
$ docker build -t tiao/web-flask-rq:v1 web-flask
$ kubectl run web-flask --image=tiao/web-flask-rq:v1 --replicas=1 --port=8080 \
>                       --labels='app=flask,tier=frontend' \
>                       --requests='cpu=100m,memory=100Mi' \
>                       --env="GET_HOSTS_FROM=dns" \
>                       --dry-run --output=yaml > web-flask-deployment.yaml

3.5   web-flask-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: flask
    tier: frontend
  name: web-flask
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flask
      tier: frontend
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flask
        tier: frontend
    spec:
      containers:
      - env:
        - name: GET_HOSTS_FROM
          value: dns
        image: tiao/web-flask-rq:v1
        name: web-flask
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
status: {}
$ kubectl create -f web-flask-deployment.yaml
deployment "web-flask" created
$ kubectl expose deployment web-flask --selector='app=flask,tier=frontend' --type=NodePort \
>                                     --dry-run --output=yaml > web-flask-service.yaml

3.6   web-flask-service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: flask
    tier: frontend
  name: web-flask
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: flask
    tier: frontend
  type: NodePort
status:
  loadBalancer: {}
$ kubectl create -f web-flask-service.yaml
You have exposed your service on an external port on all nodes in your
cluster.  If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30321) to serve traffic.

See http://releases.k8s.io/release-1.2/docs/user-guide/services-firewalls.md for more details.
service "web-flask" created

4   Deploy the Redis Queue (RQ) workers

$ mkdir rq-worker

4.1   Dockerfile

FROM tiao/web-flask-rq:v1

CMD ["rq", "worker", "--config", "settings"]
$ docker build -t tiao/rq-worker:v1 rq-worker
$ kubectl run rq-worker --image=tiao/rq-worker:v1 --replicas=5 \
>                       --labels="app=rq,role=worker,tier=backend" \
>                       --requests="cpu=100m,memory=100Mi" \
>                       --env="GET_HOSTS_FROM=dns" \
>                       --dry-run --output=yaml > rq-worker-deployment.yaml

4.2   rq-worker-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: redis
    role: worker
    tier: backend
  name: rq-worker
spec:
  replicas: 5
  selector:
    matchLabels:
      app: redis
      role: worker
      tier: backend
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: redis
        role: worker
        tier: backend
    spec:
      containers:
      - env:
        - name: GET_HOSTS_FROM
          value: dns
        image: tiao/rq-worker:v1
        name: rq-worker
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
status: {}
$ kubectl create -f rq-worker-deployment.yaml
deployment "rq-worker" created

5   Testing it out

We make make use of JSONPath support in the kubectl tool to query the NodePort for our web-flask service:

$ kubectl get service web-flask --output='jsonpath={.spec.ports[0].NodePort}'
30321%
$ port=$(kubectl get service web-flask --output='jsonpath={.spec.ports[0].NodePort}')

We construct the address for ease of reference later on:

$ address="$(minikube ip):$port"
$ echo $address
192.168.99.101:31637
$ open "http://${address}/?start=23&stop=31"
../../images/flask-rq-job.png
$ curl "http://${address}/?start=41&stop=45"
{
  "job_id": "cc31bdcd-ad31-41ce-b516-2b90cd92f2a1"
}
$ curl "http://${address}/?start=41&stop=45" | jq '.job_id'
"cc31bdcd-ad31-41ce-b516-2b90cd92f2a1"
$ curl "http://${address}/results/cc31bdcd-ad31-41ce-b516-2b90cd92f2a1"
{
  "result": 43
}
$ curl "http://${address}/results/cc31bdcd-ad31-41ce-b516-2b90cd92f2a1" | jq '.result'
43
$ curl "http://${address}/?start=53&stop=45" | jq '.job_id'
"252b14a4-4a9e-45eb-8834-9e2078fb94ed"
$ curl "http://${address}/results/252b14a4-4a9e-45eb-8834-9e2078fb94ed"
Job has failed!%

6   RQ Dashboard (Optional)

$ mkdir rq-dashboard

6.1   requirements.txt

$ echo rq-dashboard > rq-dashboard/requirements.txt

6.2   Dockerfile

$ $EDITOR rq-dashboard/Dockerfile
FROM python:3.5.1-onbuild

EXPOSE 9181
CMD ["rq-dashboard", "--port", "9181", \
                     "--redis-host", "redis-master", \
                     "--redis-port", "6379"]
$ docker build -t tiao/rq-dashboard:v1 rq-dashboard
$ kubectl run rq-dashboard --image=tiao/rq-dashboard:v1 --replicas=1 --port=9181 \
>                          --labels='app=rq,role=dashboard,tier=frontend' \
>                          --requests='cpu=100m,memory=100Mi' \
>                          --env="GET_HOSTS_FROM=env" \
>                          --dry-run --output=yaml > rq-dashboard-deployment.yaml

6.3   rq-dashboard-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: rq
    role: dashboard
    tier: frontend
  name: rq-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rq
      role: dashboard
      tier: frontend
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: rq
        role: dashboard
        tier: frontend
    spec:
      containers:
      - env:
        - name: GET_HOSTS_FROM
          value: env
        image: tiao/rq-dashboard:v1
        name: rq-dashboard
        ports:
        - containerPort: 9181
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
status: {}
$ kubectl create -f rq-dashboard-deployment.yaml
deployment "rq-dashboard" created
$ kubectl expose deployment rq-dashboard --selector='app=rq,role=dashboard,tier=frontend' --type=NodePort \
>                                        --dry-run --output=yaml > rq-dashboard-service.yaml

6.4   rq-dashboard-service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: rq
    role: dashboard
    tier: frontend
  name: rq-dashboard
spec:
  ports:
  - port: 9181
    protocol: TCP
    targetPort: 9181
  selector:
    app: rq
    role: dashboard
    tier: frontend
  type: NodePort
status:
  loadBalancer: {}
$ kubectl create -f rq-dashboard-service.yaml
You have exposed your service on an external port on all nodes in your
cluster.  If you want to expose this service to the external internet, you may
need to set up firewall rules for the service port(s) (tcp:30645) to serve traffic.

See http://releases.k8s.io/release-1.2/docs/user-guide/services-firewalls.md for more details.
service "rq-dashboard" created
$ open "http://$(minikube ip):$(kubectl get service rq-dashboard --output='jsonpath={.spec.ports[0].NodePort}')"
../../images/rq-dashboard-failed.thumbnail.png
$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
redis-master-2576299852-iwf15   1/1       Running   0          52m
rq-dashboard-1288919851-n10qs   1/1       Running   0          20m
rq-worker-3416405364-bekby      1/1       Running   0          45m
rq-worker-3416405364-cecxu      1/1       Running   0          45m
rq-worker-3416405364-lnxha      1/1       Running   0          45m
rq-worker-3416405364-nc474      1/1       Running   0          45m
rq-worker-3416405364-ztmuq      1/1       Running   0          45m
web-flask-338777398-21dli       1/1       Running   0          47m

6.5   Alternative Solution: Integrate the Dashboard in our Flask app

diff --git a/web-flask/app.py b/web-flask/app.py
index af6ec48..ec7c1cd 100644
--- a/web-flask/app.py
+++ b/web-flask/app.py
@@ -4,12 +4,16 @@ from rq import Queue

 from random import randrange

-from settings import REDIS_HOST, REDIS_PORT
-
+import rq_dashboard
+import settings

 app = Flask(__name__)
+app.config.from_object(rq_dashboard.default_settings)
+app.config.from_object(settings)
+app.register_blueprint(rq_dashboard.blueprint, url_prefix='/dashboard')

-q = Queue(connection=StrictRedis(host=REDIS_HOST, port=REDIS_PORT))
+q = Queue(connection=StrictRedis(host=settings.REDIS_HOST,
+                                 port=settings.REDIS_PORT))
diff --git a/web-flask/requirements.txt b/web-flask/requirements.txt
index 17dba2a..fcb5d58 100644
--- a/web-flask/requirements.txt
+++ b/web-flask/requirements.txt
@@ -1,3 +1,4 @@
 flask
 redis
 rq
+rq-dashboard
$ docker build -t tiao/web-flask-rq:v2 web-flask
diff --git a/web-flask-deployment.yaml b/web-flask-deployment.yaml
index a4524e6..2dc5356 100644
--- a/web-flask-deployment.yaml
+++ b/web-flask-deployment.yaml
@@ -24,7 +24,7 @@ spec:
       - env:
         - name: GET_HOSTS_FROM
           value: dns
-        image: tiao/web-flask-rq:v1
+        image: tiao/web-flask-rq:v2
         name: web-flask
         ports:
         - containerPort: 8080
$ kubectl apply -f web-flask-deployment.yaml
deployment "web-flask" configured
$ kubectl get deployment web-flask --output='jsonpath={.spec.template.spec.containers[*].image}'
tiao/web-flask-rq:v2%
$ open "http://${address}/dashboard"
../../images/rq-dashboard-failed-flask.thumbnail.png

Comments

Comments powered by Disqus