Author Archives: call518

OpenStack on Kubernetes (GitHub)



Hi! all, Welcome to my project. I hope this will be useful thing. 🙂

p.s. And I hope that some guy help me. If you are interested in this, please contact me. thank you.


  • OaaS is “OpenStack as a Service”.
  • MAINTAINER: Jung-In.Jung (
  • 2018-06-20 ~ Now


  • Auto-Provisioning OpenStack, Based on Kubernetes(k8s).
  • Scalaing for LB/HA.
  • Supoort Provisioning Almost OpenStack Releases. (Newton/Ocata/Pike/Queens/…)
  • Simple and Dynamic Configuration.
  • Auto Failover/Failback.

Components Diagram


Networking Diagram

OaaS Netowkring


  • Ocata
    • memcached (Completed)
    • rabbitmq (Completed)
    • mongodb (Completed)
    • etcd (Completed)
    • galera (Completed)
    • haproxy (Completed)
    • keystone (Completed)
    • glance (Completed)
    • nova (Completed)
    • neutron (Completed)
    • cinder (Completed)
    • heat (Completed)
    • ceilometer-central (Completed)
    • aodh (Completed)
    • horizon (Completed)
  • Integration of OpenStack Releases.
    • (TODO: Planning…)


System Env. & Arch.


  • General Kubernetes Cluster.
  • All k8s Worker nodes have to sync Time (e.g. chrony, ntp)
  • k8s worker nodes for neutron-server/nova-compute need to load openvswitch/ebtables/ip_vs kernel module.
    • Run contents of on compute/network role(label) woker nodes.
  • Quorum PODs (Replica is have to 2n+1)
    • galera-etcd
    • galera
    • rabbitmq


(Note) We have tested this tutorial with 6 VMs on VirtualBox Env. If u want, any env is possible. (eg. physical machines)

(Note) We use NFS for cinder backend storage for simple tutorial, but soon we will change to ceph back-end storage.

Spec. of Physical Host

  • Processor: Intel Core i5-6500 (3.2GHz)
  • Memory: 32GB
  • Storage: SSD (2TB)
  • NIC: Intel(R) Ethernet Connection (5) I219-LM

Spec. of each VM

(Note) Maybe, you need big RAM. if not, reduce number of replicas.

  • CentOS-7 x86_64
  • 2 EA vCPUs
  • vRAM: 2GB ~ 6GB
  • 100 GB vDisk
  • 1 EA NIC

Spec. of k8s

This is versions of packages that i have tested.

  • docker-ce-18.06.0.ce-3.el7.x86_64
  • kubernetes-cni-0.6.0-0.x86_64
  • kubectl-1.11.1-0.x86_64
  • kubelet-1.11.1-0.x86_64
  • kubeadm-1.11.1-0.x86_64

VM OS Env.

# cat /etc/hosts k8s-master k8s-node01 k8s-node02 k8s-node03 k8s-node04 k8s-node05

Deploy Tutorial

Node Labels (Role)

(!) Network(neutron-sever) worker nodes must be separated/dedicated. (network=true)

[k8s-master]# kubectl get node --show-labels

k8s-master   Ready     master    8d        v1.11.1
k8s-node01   Ready     <none>    8d        v1.11.1   controller=true,compute=true,nfs-server=true
k8s-node02   Ready     <none>    8d        v1.11.1   controller=true,compute=true
k8s-node03   Ready     <none>    8d        v1.11.1   controller=true,compute=true
k8s-node04   Ready     <none>    8d        v1.11.1   network=true
k8s-node05   Ready     <none>    8d        v1.11.1   network=true

default configs (eg. password)

check main env file, src-ocata/configMap-env-common.yaml

apiVersion: v1
kind: ConfigMap
  name: env-common
  DISCOVERY_SERVICE: "etcd-client:2379"
  CLUSTER_NAME: "mariadb_galera_ss"
  #MYSQL_DATABASE: "mydatabase"
  #MYSQL_USER: "myuser"
  MYSQL_PASSWORD: "passw0rd"
  K8S_MONGO_USER: "admin"
  K8S_MONGO_PASS: "passw0rd"
  K8S_KEYSTONE_DB_PASS: "passw0rd"
  K8S_GLANCE_DB_PASS: "passw0rd"
  K8S_CINDER_DB_PASS: "passw0rd"
  K8S_NEUTRON_DB_PASS: "passw0rd"
  K8S_NOVA_DB_PASS: "passw0rd"
  K8S_GNOCCHI_DB_PASS: "passw0rd"
  K8S_AODH_DB_PASS: "passw0rd"
  K8S_HEAT_DB_PASS: "passw0rd"
  K8S_NFS_SERVER_IP_ETC_KEY: "k8s-oaas-nfs-server-ip-address"

Required Docker Images.

  • call518/oaas-init-container
  • call518/oaas-nfs-server
  • call518/oaas-etcd
  • call518/oaas-galera
  • call518/oaas-memcached
  • call518/oaas-rabbitmq
  • call518/oaas-mongodb
  • call518/oaas-haproxy
  • call518/oaas-zookeeper
  • call518/oaas-ocata

Initiate Deploying OpenStack

[k8s-master]# git clone [here]

[k8s-master]# cd OpenStack-on-Kubernetes/src-ocata

[k8s-master]# ./

Result Deploying

[k8s-master]# kubectl get all -o wide

NAME                           READY     STATUS    RESTARTS   AGE       IP             NODE
pod/cinder-0                   1/1       Running   0          38m   k8s-node03
pod/cinder-1                   1/1       Running   0          6m   k8s-node01
pod/etcd0                      1/1       Running   0          38m   k8s-node02
pod/etcd1                      1/1       Running   0          38m   k8s-node03
pod/etcd2                      1/1       Running   0          38m   k8s-node01
pod/galera-0                   1/1       Running   1          38m   k8s-node02
pod/galera-1                   1/1       Running   0          17m   k8s-node03
pod/galera-2                   1/1       Running   1          33m   k8s-node01
pod/glance-0                   1/1       Running   0          38m   k8s-node01
pod/glance-1                   1/1       Running   0          10m   k8s-node03
pod/haproxy-7b567f67d8-mxm7v   1/1       Running   3          38m   k8s-node03
pod/horizon-6965547f56-kgc44   1/1       Running   0          38m   k8s-node02
pod/keystone-0                 1/1       Running   0          38m   k8s-node02
pod/keystone-1                 1/1       Running   0          13m   k8s-node03
pod/memcached-0                1/1       Running   0          38m   k8s-node02
pod/memcached-1                1/1       Running   0          38m   k8s-node03
pod/memcached-2                1/1       Running   0          38m   k8s-node01
pod/neutron-server-0           1/1       Running   0          38m   k8s-node01
pod/neutron-server-1           1/1       Running   0          5m   k8s-node03
pod/nfs-server                 1/1       Running   0          38m   k8s-node01
pod/nova-compute-0             1/1       Running   0          38m    k8s-node04
pod/nova-compute-1             1/1       Running   0          38m    k8s-node05
pod/nova-server-0              1/1       Running   0          38m   k8s-node02
pod/nova-server-1              1/1       Running   0          4m   k8s-node01
pod/rabbitmq-0                 1/1       Running   0          38m   k8s-node03
pod/rabbitmq-1                 1/1       Running   0          35m   k8s-node01
pod/rabbitmq-2                 1/1       Running   0          35m   k8s-node02

NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                       AGE       SELECTOR
service/cinder                ClusterIP   None             <none>        8776/TCP                                                      38m       app=cinder
service/etcd-client           ClusterIP    <none>        2379/TCP                                                      38m       app=etcd
service/etcd0                 ClusterIP    <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd0
service/etcd1                 ClusterIP     <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd1
service/etcd2                 ClusterIP   <none>        2379/TCP,2380/TCP                                             38m       etcd_node=etcd2
service/galera                ClusterIP   None             <none>        3306/TCP                                                      38m       app=galera
service/glance                ClusterIP   None             <none>        9292/TCP,9191/TCP                                             38m       app=glance
service/haproxy-galera        ClusterIP   None             <none>        3306/TCP                                                      38m       app=haproxy
service/haproxy-stats         NodePort   <none>        9000:30090/TCP                                                38m       app=haproxy
service/horizon               NodePort    <none>        80:30080/TCP                                                  38m       app=horizon
service/keystone              ClusterIP   None             <none>        5000/TCP,35357/TCP                                            38m       app=keystone
service/kubernetes            ClusterIP        <none>        443/TCP                                                       2d        <none>
service/memcached             ClusterIP   None             <none>        11211/TCP                                                     38m       app=memcached
service/neutron-server        ClusterIP   None             <none>        9696/TCP                                                      38m       app=neutron-server
service/nova-compute          ClusterIP   None             <none>        8774/TCP,8775/TCP,6080/TCP                                    38m       app=nova-compute
service/nova-server           NodePort     <none>        8774:30177/TCP,8778:30246/TCP,8775:31964/TCP,6080:30068/TCP   38m       app=nova-server
service/rabbitmq              ClusterIP   None             <none>        5672/TCP,4369/TCP,25672/TCP                                   38m       app=rabbitmq
service/rabbitmq-management   ClusterIP   None             <none>        15672/TCP                                                     38m       app=rabbitmq

NAME                      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES                 SELECTOR
deployment.apps/haproxy   1         1         1            1           38m       haproxy      call518/oaas-haproxy   app=haproxy
deployment.apps/horizon   1         1         1            1           38m       horizon      call518/oaas-ocata     app=horizon

NAME                                 DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES                 SELECTOR
replicaset.apps/haproxy-7b567f67d8   1         1         1         38m       haproxy      call518/oaas-haproxy   app=haproxy,pod-template-hash=3612392384
replicaset.apps/horizon-6965547f56   1         1         1         38m       horizon      call518/oaas-ocata     app=horizon,pod-template-hash=2521103912

NAME                              DESIRED   CURRENT   AGE       CONTAINERS       IMAGES
statefulset.apps/cinder           2         2         38m       cinder           call518/oaas-ocata
statefulset.apps/galera           3         3         38m       galera           call518/oaas-galera
statefulset.apps/glance           2         2         38m       glance           call518/oaas-ocata
statefulset.apps/keystone         2         2         38m       keystone         call518/oaas-ocata
statefulset.apps/memcached        3         3         38m       memcached        call518/oaas-memcached
statefulset.apps/neutron-server   2         2         38m       neutron-server   call518/oaas-ocata
statefulset.apps/nova-compute     2         2         38m       nova-compute     call518/oaas-ocata
statefulset.apps/nova-server      2         2         38m       nova-server      call518/oaas-ocata
statefulset.apps/rabbitmq         3         3         38m       rabbitmq         call518/oaas-rabbitmq

Open Horizon Dashboard

In browser, http://[One_of_worker_nodes]:30080


(Note) Maybe, needed more worker nodes…


[k8s-master]# kubectl scale --replicas=4 statefulset.apps/cinder
[k8s-master]# kubectl scale --replicas=5 statefulset.apps/galera (must 2n+1)



Horizon Login


Horizon Overview


Horizon Instances

Network Topology

Horizon Network Topology

Instance Web Console

Horizon VM Console


  • Re-Configuration Flat-IP Ranbe for EXT-NET.
  • (Done) Simplify initContainer Check-Processing.
  • Integration of All of Provision Source. /w Template and ETc…



Active Repository (GitHub)

Mirrored Repository (GitLab)



An Overview of Link Aggregation and LACP

The Network Way - Nir Yechiel's blog

The concept of Link Aggregation (LAG) is well known in the networking industry by now, and people usually consider it as a basic functionality that just works out of the box. With all of the SDN hype that’s going on out there, I sometimes feel that we tend to neglect some of the more “traditional” stuff like this one. As with many networking technologies and protocols, things may not just work out of the box, and it’s important to master the details to be able to design things properly, know what to expect to (i.e., what the normal behavior is) and ultimately being able to troubleshoot in case of a problem.

The basic concept of LAG is that multiple physical links are combined into one logical bundle. This provides two major benefits, depending on the LAG configuration:

  1. Increased capacity – traffic may be balanced across the member links to provide…

View original post 1,297 more words

Distributed Virtual Routing – SNAT

Assaf Muller

Where Am I?
Overview and East/West Traffic
Floating IPs

SNAT vs Floating IPs

A quick reminder about two NAT types used in Neutron.

  1. SNAT refers to source NAT, or, changing the source address of packets as they leave the external device of a router. This is used for traffic originating from VMs that have no floating IP attached. A router is allocated a single IP address from the external network which is shared across all VMs connected to all subnets the router is connected to. Sessions are differentiated according to the full tuple of (source IP, destination IP, source port, destination port). This is typically known as ‘PAT’, or port address translation in the networking world.
  2. Floating IPs, sometimes called DNAT (Destination NAT) in Neutronland, implement a much simpler form of NAT, a 1:1 private to public address translation. You can assign a VM a floating IP and…

View original post 1,016 more words