netgo

In this post, I will install the kubernetes with external etcd. Most of instrcution come from here. I will follow step by step as the beginner.

1. Pre-requsite

At first I need to prepare the hosts for Kubernetes components. In my case, I have 5 nodes, one for master, one for external etcd and three for workers. After I prepare these nodes. I need to update "/etc/hostname" and "/etc/hosts" files propertly. For example, I have to change hostname to "m1" as master node.

After this, I need to log-out and log-in again.

Kubernetes use Docker engine. Therefore, I need to install Docker. I will follow this instruction. In my case, I will run this kubernetes over Ubuntu 16.04. This is overview.

I need to check docker info. It will be like below. I need to check "Cgroup Driver: cgroupfs" as default.

From this instruction, I will change this driver from "cgroupsfs" to "systemd". This change affect lots of issue during Kubernetes installation. Therefore I need to remember this.

After this, Cgroup Driver will be changed 

2. Install kubelet, kubeadm and kubectl

Kubelet (daemon on each node has the role to handle conatiner), kubeadm (daemon on each node has the role to setup master and workers) and kubectl (daemon on each node has the role to manage the cluster) are necessary components.

After installation, I need to check the directories which are created during the installation. "/etc/kubernetes/" and "/var/lib/kubelet/pki/" and "/etc/systemd/system/kubelet.service.d/" are created. 

To work Kubernetes normally, Kubelet work normally at first. I can check the status with "systemctl status kubelet". At this moment, it is not activate.

To see more details, I will look at the logs in "/var/log/syslog". From logs, I can check I need "config.yaml" file. Keep this issue and go to next step. This file will be created with "kubectl init" command later.

I need to match the cgroup driver with the value "systemd". In the instruction, there is the comment like below.

However there are no "/etc/default/kubelet" and "/var/lib/kubelet/kubeadm-flags.env". So I can not update this with this instruction. Please look at the result from "systemctl status kubelet". I can see "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf". This is the loaded file when kubelet is started.

I will add the red text, Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd". And restart the kubelet with "systemctl daemon-reload" and "systemctl restart kubelet". However, it is still not activated due to no config.yaml.

3. Install external etcd cluster

I have alread posted in here. To create the etcd cluster, It is not easy. However, I can make these with this instruction easliy. This etcd will be created as one of container application which are managed by Kubernetes. On the external etcd node, I have to run command below.

 Please note that there is no option for cgroup drivers. Therefore, I need to change something.

The file name start with 20-etcd-service-manager.conf due to "20", so it has higher priority rather than 10-kubeadm.conf. After running this, I can check the "systemctl status kubelet" and "/var/log/syslog". The kubelet does not works with some error. This error is different from above. The first error above is happend after 10-kubeadm.conf loading. However this error is happend during 20-etcd-service-manager.conf loading. So I have to solve this error at first.

There are some error and warning messages. To solve this, I have to create "kubeadmcfg.yaml" from this instruction. In my case, I will difference file name like below

"controlPlaneEndpoint: 147.75.94.251:6443" is master IP address and Port for API server. This parameter solve this warning "No api server defined - no events will be sent to API server.". 147.75.92.69 ip address is external etcd node interface IP addresss which is announced for outside. In this yaml file, some options are included. Therefore, I have to revise 20-etcd-service-manager.conf file like below. 

Even if I added more options in yaml file, I do not handle root cause why the kubelet fail. In the syslog, "failed to run Kubelet: Running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false" is the main reason. To solve this, there are 3 option.

"swapoff -a" is temporary option. Comment "swap" in "/etc/fstab" is permenant option. Adding " --fail-swap-on=false" in 20-etcd-service-manager.conf is another permenant option. After then, the status of kubelet will be like this.

Now, I am ready to deploy etcd container deployment. I need certificate and private key for etcd cluster to work well. These values are located in "/etc/kubernetes/pki/etcd". However, you do not have this directory at this time. As the instruction, these file will be generated with command below.

If you are want to make cluster of etcd nodes, I have copy these files on each etcd nodes. However, it is not enough to work well. I need more certiface and key.

after then, serveral key will be generated. In fact, some of key is not necessary if you run single external etcd node. Now, I can run etcd container with command below

I can also check the health status.

4. Install master nodes

There are 2 instructions for master node, single master and cluser masters. To setup the master node, I need configuration file. I will refer from this instruction. This is the sample configuration yaml file what I have

With this configuration above, I can run "kubeadm init". Please note that "mode: ipvs" will change the kube-proxy mode to work with "ipvsadm" kerneal module. In the post what I wrote, It will be undertand how kube-proxy work with this part. This is overview to install and load kernal module

Sometimes, some module are not loaded. So I need to do manually with "modeprobe". If these module are not loaed when kubeadm init is started, I can see this message "kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh]" 

Now, it will load the file "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf".

With this configuration I can run "kubeadm init" 

before run kubeadm init, I need to "swapoff -a". I can get some error message without "swapoff -a"

Anyway, I will get this result when I make master node succefully.

I need more steps. First I configure for "kubectl" enable.

After this, I can use "kubectl" command. Sometimes, master status can be "NotReady". At this time, I should check the status of pods with "kubectl get pods --all-namespace"

From this instruction, I need to POD network to solve this issue.

To install POD network, look at this instuction.

In my case, I will use "Flannel".

And then, the master status is normal

Oh It is work now. See status with more information

There is something strange. "cgroupDriver: cgroupfs" is strange. It should be "systemd", becuase I update "Environment="KUBELET_EXTRA_ARGS=--cgroup-driver=systemd" in "/etc/systemd/system/kubelet.service.d/10-kubeadm.conf". I think it is not work. I will update for this

I will run "kubeadm reset" and "kubeadm init" again. Now

However, this is not all of things. I have alread made external etcd. I have not configure for this yet. Default, etcd conatiner is located on each master node. They are not clustered.

From this instruction, there are serveral step for this. At first I need to copy the certificate and key from etcd node.

Now, I need to update my configuration file

Now, I will run "kubeadm reset" and "kubeadm init" again. Please note that the certificate and key will be remove when I run kubeadm reset. Therefore, I need to copy again from etcd node

After I copy the certificate and key, I can check the status again

etcd is not existed on master node. 

5. Setup the Worker nodes

It is almost same. I need docker engin and kubeadm, kubectl, kubelet. I have already explain about these above. After creating master node, I can see the message like below

First is for adding master node, second is for adding worker node. I need some information such as token.

With these values, I can run command on each worker node. I will check the status of the cluster.

Now I made the kubernetes cluster.

Reference

[ 1 ] https://kubernetes.io/docs/setup/independent/install-kubeadm/#installing-kubeadm-kubelet-and-kubectl

[ 2 ] https://docs.docker.com/install/linux/docker-ce/ubuntu/

[ 3 ] https://kubernetes.io/docs/setup/cri/

[ 4 ] https://kubernetes.io/docs/setup/independent/setup-ha-etcd-with-kubeadm/ 

[ 5 ] https://createnetech.tistory.com/17?category=679927

[ 6 ] https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[ 7 ] https://kubernetes.io/docs/setup/independent/high-availability/

[ 8 ] https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta1

[ 9 ] https://kubernetes.io/docs/setup/independent/troubleshooting-kubeadm/#coredns-or-kube-dns-is-stuck-in-the-pending-state

[ 10 ] https://kubernetes.io/docs/setup/independent/high-availability/#external-etcd-nodes

[ 11 ] https://createnetech.tistory.com/36?category=672585

[ 12 ] https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#is-kube-proxy-writing-iptables-rules

How install oracle 11g on Windows 2012 R2?  (Enable archive log and Supplemental log)?

Recently, I need to learn the oracle to migrate data from on-prem to aws. In fact, I am not the DBA, therefore, I can not handle deep dive. I hope the I can help someone like me who is the first time about oracle.

1. Pre-requsite.

I have Windows 2012 R2 which supports oracle database 11g. For successful installation, I need the ".Netframe 3.5" installed. In server manager. Click "Add Roles and Features" at first.

In Features, choose the ".NET Framework 3.5 Feature" and Install this.

After then, I am ready to install oracle database.

2. Install oracle database.

The reason why I select Windows is simple to consist rather than Linux. I will follow two instructions mainly. https://www.youtube.com/watch?v=YgFfjLHY-wE is helpful for me. At the beginning, I need installation file. I can download from oralce site.

In my case, I will download "Microsoft Windows (X64)". There are two files. After extract both of files, I need to merge and make together in same directory. I copied and pasted whole directory from secondary to first.

After merge, I can see the directory inside. There is the "setup" icon. Click it.

Click Run and Start Installation for oracle database.

Depends on the system environments, Installer make warnning messages like this. However, I can ignore this, Click "yes"

Uncheck "I wish to receive security updates via My Oracle support", because I do not need this feature. It does not matter for next step. In email field, I can leave with empty. Just click "Next"

After Next, the warning message come. However it is also not matter for next step. Just click "yes".

Choose "Create and configure as a database". Because this is first installtion on this server. Click "Next".

In my case, my operating system is windows 2012 R2 server. Therefore, I will choose "Sever class".

I will install with standalone mode. Choose "single instance database installation" and click "Next".

Choose "Advanced Install" and Next.

This step is more important rather than before. In this insturction, there are some explain about the fields. I overview about below. 

Select "Langurage"

I will select "Standard Edition One" (Please note I can select what I want as the oracle database engine).

I select my custom home directory like below.

Select configuration type. In my case, I will choose genera purpose.

"Global database name" is not SID. However, I choosed single instance mode. Becuase of this, the SID can be same as Global database name. However, I can make different in Advanced mode.

I will adjust some paramters for memory, characters set, security and sample schema.

I will UTF-8 for character sets.

I will not make sample schema. Un-check "Create database with sample schema"

During this installation, Oracle Enterprise Manager will be installed, even if I do not want. Just click Next

Select File system. Usually it is better to seperate from root disk.

This is the back up configuration. I don't want this at this time.

When I select "Typical Installation". There are Administrative password which is default password for schema (=users). 

However, In Advanced mode, I can set password individually like below.

If you use simple password, the warnning message happen like below. In my case, this is test oracle database. I will ignore this. Click yes.

This is summary, Click Finish to installation.

During installation, I can see the configuration assistant process like below.

During the installation, warnning message like below can be occure. In this case, I need to set variable in windows.

I will handle this later. At this time, just click next. I registered the "Administrative password" before. I have told it will be used as default password. However, I have another change to set the password for each schema (=user).

Click Password Management. I will set individual password for "SYS" and "SYSTEM". If you want more, you can set the password more. In oracle database, there are serveral sample test schema such as "HR" and "SCOTT".

Continue with Yes.

3. Access database

Now, I can access database. At this time, I can use sqlplus which is installed with.

I can access with SYSTEM schema account like below

4. Verify the oracle environment for enterprise manager.

During the installation, I see warning message below. Because of this, I need to check environment variables.

In this page, I need to check the status with "emctl status dbconsole". 

In this case, I need to set environment value. Properties > Advanced system settings.

In Advanced tap, there are environment value button.

I will add some values in User vriables for Administrator.

For the first issue, I will add "ORACLE_UNQNAME". 

I will reference the value from db_unique_name. In fact, there is value when environment is not set. run "select name,db_unique_name from v$database;".

I will try again "emctl status dbconsole". At this time, there is "Not found" error. 

Go into the directory. Thre is no localhost. Instead, 11.0.0.131 is existed.

To resolve this, I will add "ORACLE_HOSTNAME" like below.

After then, It will be like below. This is normal. However, It look like not perfect. "EM Daemon is not running" is happen.

I followed this video. To make success, I have to start service manually. At this time, I can not start after right button.

Find the file below. In emd properites file, "agentTZRegion=+06:00" should be adjusted with OS time.

I adjust the time 

Now, I can use this.

5. Oralce database stop, start and restart

In this instruction, there are several way to stop and start. In this part, I will overview sqlplus to stop and start.

6. Enable/Disable Archive log Mode

Sometime, it is necessary to enable this feature. It is utilized to synchronize databases with log-miner. For this, I will follow this instruction. run "archive log list". The result is displaied. It is not enabled default.

I need to shutdown with "shutdown immediate". My status is changed to "nolog".

I have to change the status to "mount", which I can change configuration on. Run "startup mount".

Run "alter database archivelog;" to enable

After then, I open the database "alter database open;"

If you want to disable, I use "alter database noarchivelog;"

7. Enable/Disable supplemental logging 

This action should be done in database. To check the status, run "select supplemental_log_data_min from v$database;"

To enable and disable, run query like below.

This is the basic configuration for the installation. 

Reference 

[ 1 ] https://www.youtube.com/watch?v=YgFfjLHY-wE

[ 2 ] https://medium.com/@Dracontis/how-to-install-oracle-11g-database-on-windows-server-2012-a4d30d4dc727

[ 3 ] https://www.oracle.com/technetwork/database/enterprise-edition/downloads/index.html

[ 4 ] https://docs.oracle.com/cd/E25178_01/server.1111/e10897/install.htm

[ 5 ] http://www.dba-oracle.com/t_oracle_unqname.htm

[ 6 ] http://blog.mclaughlinsoftware.com/2012/08/23/whats-oracle_unqname/

[ 7 ] https://www.youtube.com/watch?v=Y8i41J7aLwo

[ 8 ] https://docs.oracle.com/cd/B14117_01/win.101/b10113/admin.htm#CDCEJADC

[ 9 ] http://www.oracledistilled.com/oracle-database/backup-and-recovery/enabledisable-archive-log-mode-10g11g/

How to user the Oracle View and Joins (Inner and outer)

Recently, I have been studying about the Oracle Database. In this post, I wrote about the basic concept. In this post, I will handle about View and Joins.

1. Pre-requisite

I will create 2 tables like below.

After run these command, I can get the tables with datas

Now, I am ready to start.

2. Inner and  Right, Left, Full Joins

I will follow this instructions. In this instructions, It has more detail. In this post, I will handle the way how to use this. Joins are one of filters with tables. Therefore, I can extract the matched data from tables.

2-1. Inner Join

As the figure, It shows the result after Inner Join. I can extract the matched data as intersection. This is the command. 

This command means that I can extract intersection from both of "MYADMIN.users" table and "MYADMIN.lessons" table.

2-2. Left outer Join

With this option, Left side of tables are the source. This source start to compare data of the Right side of tables. When condition is matched. It extract matched data from both of table. However, the condition is not matched, the data from the source can be obtained but the data from right side of table inserted as the "<null>"

2-3. Right outer Join

This is reverse concept of the left outer join above. At this time, left side of tables is the source. Therefore, the value could be "<null>" from right side of tables, when condition does not match.

2-4. Full outer Join

With this option, left side and right side can be source. If the column from left side which does not exist in right side, this is the source. If the column from right side which does not exist in left side, this is also the source.  

3. Views

I will follow this instruction for this part. Views is the virtual table which is not existed in real. Sometime, I need to combine or make complex database query. In this case, this views is quite useful. For example, I will create view with the query of full outer join.

I have already told that the view is not real table. Thus, I can not find out the information from "user_tables".

However, I can see the view information from "user_views". In the "TEXT", I can see the query which I used to create.

I think "the views is belonged to the schema (=user), even if it is virtual table". I can give some grant option like below.

After login again with "MYUSER" account, I can show the data from this view also.

However, I can not show any detail information with "user_views" or "user_table".

Because, it only show the data whch is created by "schema". Therefore, if you want to see, you need to search through grant option like below.

If I want delete (drop) the view, Use this command below

Reference 

[ 1 ] https://www.techonthenet.com/oracle/joins.php

[ 2 ] https://createnetech.tistory.com/32

[ 3 ] https://www.techonthenet.com/oracle/views.php