OpenStack on Kubernetes (GitHub)



Hi! all, Welcome to my project. I hope this will be useful thing. 🙂

p.s. And I hope that some guy help me. If you are interested in this, please contact me. thank you.


  • OaaS is “OpenStack as a Service”.
  • MAINTAINER: Jung-In.Jung (
  • 2018-06-20 ~ Now


  • Auto-Provisioning OpenStack, Based on Kubernetes(k8s).
  • Scalaing for LB/HA.
  • Supoort Provisioning Almost OpenStack Releases. (Newton/Ocata/Pike/Queens/…)
  • Simple and Dynamic Configuration.
  • Auto Failover/Failback.

Components Diagram


Networking Diagram

OaaS Netowkring


  • Ocata
    • memcached (Completed)
    • rabbitmq (Completed)
    • mongodb (Completed)
    • etcd (Completed)
    • galera (Completed)
    • haproxy (Completed)
    • keystone (Completed)
    • glance (Completed)
    • nova (Completed)
    • neutron (Completed)
    • cinder (Completed)
    • heat (Completed)
    • ceilometer-central (Completed)
    • aodh (Completed)
    • horizon (Completed)
  • Integration of OpenStack Releases.
    • (TODO: Planning…)


System Env. & Arch.


  • General Kubernetes Cluster.
  • All k8s Worker nodes have to sync Time (e.g. chrony, ntp)
  • k8s worker nodes for neutron-server/nova-compute need to load openvswitch/ebtables/ip_vs kernel module.
    • Run contents of on compute/network role(label) woker nodes.
  • Quorum PODs (Replica is have to 2n+1)
    • galera-etcd
    • galera
    • rabbitmq


(Note) We have tested this tutorial with 6 VMs on VirtualBox Env. If u want, any env is possible. (eg. physical machines)

(Note) We use NFS for cinder backend storage for simple tutorial, but soon we will change to ceph back-end storage.

Spec. of Physical Host

  • Processor: Intel Core i5-6500 (3.2GHz)
  • Memory: 32GB
  • Storage: SSD (2TB)
  • NIC: Intel(R) Ethernet Connection (5) I219-LM

Spec. of each VM

(Note) Maybe, you need big RAM. if not, reduce number of replicas.

  • CentOS-7 x86_64
  • 2 EA vCPUs
  • vRAM: 2GB ~ 6GB
  • 100 GB vDisk
  • 1 EA NIC

Spec. of k8s

This is versions of packages that i have tested.

  • docker-ce-18.06.0.ce-3.el7.x86_64
  • kubernetes-cni-0.6.0-0.x86_64
  • kubectl-1.11.1-0.x86_64
  • kubelet-1.11.1-0.x86_64
  • kubeadm-1.11.1-0.x86_64

VM OS Env.

# cat /etc/hosts k8s-master k8s-node01 k8s-node02 k8s-node03 k8s-node04 k8s-node05

Deploy Tutorial

Node Labels (Role)

(!) Network(neutron-sever) worker nodes must be separated/dedicated. (network=true)

[k8s-master]# kubectl get node --show-labels

k8s-master   Ready     master    8d        v1.11.1
k8s-node01   Ready     <none>    8d        v1.11.1   controller=true,compute=true,nfs-server=true
k8s-node02   Ready     <none>    8d        v1.11.1   controller=true,compute=true
k8s-node03   Ready     <none>    8d        v1.11.1   controller=true,compute=true
k8s-node04   Ready     <none>    8d        v1.11.1   network=true
k8s-node05   Ready     <none>    8d        v1.11.1   network=true

default configs (eg. password)

check main env file, src-ocata/configMap-env-common.yaml

apiVersion: v1
kind: ConfigMap
  name: env-common
  DISCOVERY_SERVICE: "etcd-client:2379"
  CLUSTER_NAME: "mariadb_galera_ss"
  #MYSQL_DATABASE: "mydatabase"
  #MYSQL_USER: "myuser"
  MYSQL_PASSWORD: "passw0rd"
  K8S_MONGO_USER: "admin"
  K8S_MONGO_PASS: "passw0rd"
  K8S_KEYSTONE_DB_PASS: "passw0rd"
  K8S_GLANCE_DB_PASS: "passw0rd"
  K8S_CINDER_DB_PASS: "passw0rd"
  K8S_NEUTRON_DB_PASS: "passw0rd"
  K8S_NOVA_DB_PASS: "passw0rd"
  K8S_GNOCCHI_DB_PASS: "passw0rd"
  K8S_AODH_DB_PASS: "passw0rd"
  K8S_HEAT_DB_PASS: "passw0rd"
  K8S_NFS_SERVER_IP_ETC_KEY: "k8s-oaas-nfs-server-ip-address"

Required Docker Images.

  • call518/oaas-init-container
  • call518/oaas-nfs-server
  • call518/oaas-etcd
  • call518/oaas-galera
  • call518/oaas-memcached
  • call518/oaas-rabbitmq
  • call518/oaas-mongodb
  • call518/oaas-haproxy
  • call518/oaas-zookeeper
  • call518/oaas-ocata

Initiate Deploying OpenStack

[k8s-master]# git clone [here]

[k8s-master]# cd OpenStack-on-Kubernetes/src-ocata

[k8s-master]# ./

Result Deploying

[k8s-master]# kubectl get all -o wide

NAME                           READY     STATUS    RESTARTS   AGE       IP             NODE
pod/cinder-0                   1/1       Running   0          38m   k8s-node03
pod/cinder-1                   1/1       Running   0          6m   k8s-node01
pod/etcd0                      1/1       Running   0          38m   k8s-node02
pod/etcd1                      1/1       Running   0          38m   k8s-node03
pod/etcd2                      1/1       Running   0          38m   k8s-node01
pod/galera-0                   1/1       Running   1          38m   k8s-node02
pod/galera-1                   1/1       Running   0          17m   k8s-node03
pod/galera-2                   1/1       Running   1          33m   k8s-node01
pod/glance-0                   1/1       Running   0          38m   k8s-node01
pod/glance-1                   1/1       Running   0          10m   k8s-node03
pod/haproxy-7b567f67d8-mxm7v   1/1       Running   3          38m   k8s-node03
pod/horizon-6965547f56-kgc44   1/1       Running   0          38m   k8s-node02
pod/keystone-0                 1/1       Running   0          38m   k8s-node02
pod/keystone-1                 1/1       Running   0          13m   k8s-node03
pod/memcached-0                1/1       Running   0          38m   k8s-node02
pod/memcached-1                1/1       Running   0          38m   k8s-node03
pod/memcached-2                1/1       Running   0          38m   k8s-node01
pod/neutron-server-0           1/1       Running   0          38m   k8s-node01
pod/neutron-server-1           1/1       Running   0          5m   k8s-node03
pod/nfs-server                 1/1       Running   0          38m   k8s-node01
pod/nova-compute-0             1/1       Running   0          38m    k8s-node04
pod/nova-compute-1             1/1       Running   0          38m    k8s-node05
pod/nova-server-0              1/1       Running   0          38m   k8s-node02
pod/nova-server-1              1/1       Running   0          4m   k8s-node01
pod/rabbitmq-0                 1/1       Running   0          38m   k8s-node03
pod/rabbitmq-1                 1/1       Running   0          35m   k8s-node01
pod/rabbitmq-2                 1/1       Running   0          35m   k8s-node02

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                       AGE       SELECTOR
service/cinder                ClusterIP   None             <none>        8776/TCP                                                      38m       app=cinder
service/etcd-client           ClusterIP    <none>        2379/TCP                                                      38m       app=etcd
service/etcd0                 ClusterIP    <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd0
service/etcd1                 ClusterIP     <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd1
service/etcd2                 ClusterIP   <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd2
service/galera                ClusterIP   None             <none>        3306/TCP                                                      38m       app=galera
service/glance                ClusterIP   None             <none>        9292/TCP,9191/TCP                                             38m       app=glance
service/haproxy-galera        ClusterIP   None             <none>        3306/TCP                                                      38m       app=haproxy
service/haproxy-stats         NodePort   <none>        9000:30090/TCP                                                38m       app=haproxy
service/horizon               NodePort    <none>        80:30080/TCP                                                  38m       app=horizon
service/keystone              ClusterIP   None             <none>        5000/TCP,35357/TCP                                            38m       app=keystone
service/kubernetes            ClusterIP        <none>        443/TCP                                                       2d        <none>
service/memcached             ClusterIP   None             <none>        11211/TCP                                                     38m       app=memcached
service/neutron-server        ClusterIP   None             <none>        9696/TCP                                                      38m       app=neutron-server
service/nova-compute          ClusterIP   None             <none>        8774/TCP,8775/TCP,6080/TCP                                    38m       app=nova-compute
service/nova-server           NodePort     <none>        8774:30177/TCP,8778:30246/TCP,8775:31964/TCP,6080:30068/TCP   38m       app=nova-server
service/rabbitmq              ClusterIP   None             <none>        5672/TCP,4369/TCP,25672/TCP                                   38m       app=rabbitmq
service/rabbitmq-management   ClusterIP   None             <none>        15672/TCP                                                     38m       app=rabbitmq

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES                 SELECTOR
deployment.apps/haproxy   1         1         1            1           38m       haproxy      call518/oaas-haproxy   app=haproxy
deployment.apps/horizon   1         1         1            1           38m       horizon      call518/oaas-ocata     app=horizon

NAME                                 DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES                 SELECTOR
replicaset.apps/haproxy-7b567f67d8   1         1         1         38m       haproxy      call518/oaas-haproxy   app=haproxy,pod-template-hash=3612392384
replicaset.apps/horizon-6965547f56   1         1         1         38m       horizon      call518/oaas-ocata     app=horizon,pod-template-hash=2521103912

NAME                              DESIRED   CURRENT   AGE       CONTAINERS       IMAGES
statefulset.apps/cinder           2         2         38m       cinder           call518/oaas-ocata
statefulset.apps/galera           3         3         38m       galera           call518/oaas-galera
statefulset.apps/glance           2         2         38m       glance           call518/oaas-ocata
statefulset.apps/keystone         2         2         38m       keystone         call518/oaas-ocata
statefulset.apps/memcached        3         3         38m       memcached        call518/oaas-memcached
statefulset.apps/neutron-server   2         2         38m       neutron-server   call518/oaas-ocata
statefulset.apps/nova-compute     2         2         38m       nova-compute     call518/oaas-ocata
statefulset.apps/nova-server      2         2         38m       nova-server      call518/oaas-ocata
statefulset.apps/rabbitmq         3         3         38m       rabbitmq         call518/oaas-rabbitmq

Open Horizon Dashboard

In browser, http://[One_of_worker_nodes]:30080


(Note) Maybe, needed more worker nodes…


[k8s-master]# kubectl scale --replicas=4 statefulset.apps/cinder
[k8s-master]# kubectl scale --replicas=5 statefulset.apps/galera (must 2n+1)



Horizon Login


Horizon Overview


Horizon Instances

Network Topology

Horizon Network Topology

Instance Web Console

Horizon VM Console


  • Re-Configuration Flat-IP Ranbe for EXT-NET.
  • (Done) Simplify initContainer Check-Processing.
  • Integration of All of Provision Source. /w Template and ETc…



Active Repository (GitHub)

Mirrored Repository (GitLab)


Sponsored Post Learn from the experts: Create a successful blog with our brand new courseThe Blog is excited to announce our newest offering: a course just for beginning bloggers where you’ll learn everything you need to know about blogging from the most trusted experts in the industry. We have helped millions of blogs get up and running, we know what works, and we want you to to know everything we know. This course provides all the fundamental skills and inspiration you need to get your blog started, an interactive community forum, and content updated annually.

Distributed Virtual Routing – SNAT

Assaf Muller

Where Am I?
Overview and East/West Traffic
Floating IPs

SNAT vs Floating IPs

A quick reminder about two NAT types used in Neutron.

  1. SNAT refers to source NAT, or, changing the source address of packets as they leave the external device of a router. This is used for traffic originating from VMs that have no floating IP attached. A router is allocated a single IP address from the external network which is shared across all VMs connected to all subnets the router is connected to. Sessions are differentiated according to the full tuple of (source IP, destination IP, source port, destination port). This is typically known as ‘PAT’, or port address translation in the networking world.
  2. Floating IPs, sometimes called DNAT (Destination NAT) in Neutronland, implement a much simpler form of NAT, a 1:1 private to public address translation. You can assign a VM a floating IP and

View original post 1,016 more words

Distributed Virtual Routing – Floating IPs

Assaf Muller

Where Am I?
Overview and East/West Traffic
* Floating IPs

In The Good Old Days

Legacy routers provide floating IPs connectivity by performing 1:1 NAT between the VM’s fixed IP and its floating IP inside the router namespace. Additionally, the L3 agent throws out a gratuitous ARP when it configures the floating IP on the router’s external device. This is done to advertise to the external network that the floating IP is reachable via the router’s external device’s MAC address. Floating IPs are configured as /32 prefixes on the router’s external device and so the router answers any ARP requests for these addresses. Legacy routers are of course scheduled only on a select subgroup of nodes known as network nodes.

Things Are About to Get Weird

In the DVR world, however, things are very different. This is going to get very complicated very fast so let’s understand how and

View original post 1,341 more words