Posts
Add certificates to existing gateway with non-synch resources
I started one of my certificates using cert-manager and letsencrypt without root domain in one product system. Later I tried to add root domain to the certificate as our customer requested withou success at first day. I figured out how to do that next morning. Here are my the success one and several failed attempts.
Here some background information of this system: gitops + argocd
Successful solution Considering that the app argocd is in manual synchronization mode, I can do that changes in place in kubernetes instead of the git repo.
Posts
recover from mysql XA transaction issues
In the past 1 week, I encountered an issue that I couldn’t add a column to a table in one of our production database. I thought restart would resolve the issue. After a restart, however the issue still persist. I checked logs in error/mysql-error-running.log in rds and noticed there are two transactions in prepared state after recovery. In the first gooogle search result page, How to Deal with XA Transactions Recovery caught my attention.
Posts
Do you still need to care ChatOps
ChatOps has its place, even more prominent with AI and rich integration with other systems.
I learnt ChatOps in the past when I first heard hubot. Later GitOps and Kubernetes grabbed my attention. I didn’t see how ChatOps fit in GitOps until I saw this 湾区日报是如何运作的? today after I implemented an integration with telegram bot APIs.
What Are the Benefits of ChatOps? enumerates the benefits. In my eyes I saw other benefits or more benefits in productivity when considing the full potentials of ChatOps.
Posts
Reflection on my choice of cicd in the past
TLDR: my CICD choice: argo-cd and argo-workflow
Since I chose argocd and github action as our CICD pipeline two years ago, I rarely thought about other choices and the reason I made the choices. Recently I leant from several sources that jenkins is still widely used, it arouse my interest on this subject again. I reconsidered my choice and layed out the reasons here for my future references.
I didn’t have a good impression with java in my past java projects.
Posts
Istio troubleshooting in new scenario that one service to expose into multiple domains and multiple ingress gateways
Since my first time to configure istio with proxy protocol supports in aws, istio and envoyproxy changed a lot. In the past several days, I was exposed to a different scenario that one service to expose into multiple domains.
Things are getting not straighforward, I struggled to make it success yesterday. Today another layer of complexity was added into my cluster: an extra ingressgateway for some services.
For multiple ingress gateways, I followed this article, however my setup is more complex than that.
Posts
Make a S3 bucket public
Turn off "Block public access (bucket settings)" Access control list (ACL): give Everyone (public access) object lists and bucket ACL read. Policy Policy to attach to a role, so the role can be used to upload resources and put the ACL on the uploaded objects.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::demo" }, { "Sid": "VisualEditor1", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::demo/*" }, { "Sid": "VisualEditor2", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::demo-public" }, { "Sid": "VisualEditor3", "Effect": "Allow", "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject", "s3:PutObjectAcl" ], "Resource": "arn:aws:s3:::demo-public/*" } ] } Uploading command line bash scripts aws s3 sync --acl public-read local-dir/ s3://demo-public/test/ C# source code using (var client = new AmazonS3Client(RegionEndpoint.
Posts
Create another user and grant proper permissions in aws rds mysql
When I tried to create an user for application instead of giving out the master user and its password. I got "Error Code: 1045. Access denied for user 'mysqladmin'@'%' (using password: YES)" when I run "GRANT ALL PRIVILEGES ON demodb.* TO 'appadmin'@'%';".
I tried the method in How do I create another master user for my Amazon RDS DB instance that is running MySQL?, it is for creating another master user with more permissions than an app db user.
Posts
istio virtual service with tls - Connection reset by peer
I got following similar errors when setup my istio clusters.
Mark bundle as not supporting multiuse 301 istio 301 or 404 error:02FFF036:system library:func(4095):Connection reset by peer * Trying 20.190.14.28:443... * TCP_NODELAY set * Connected to kiali.example.com (20.190.14.28) port 443 (#0) * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /home/ng/anaconda3/ssl/cacert.pem CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * OpenSSL SSL_connect: Connection reset by peer in connection to kiali.
Posts
Reflection on kubernete usage
Today I explained the current infrastructure fleet configurations to our devops team, I have the thought to make a reflection on the things or lessons I learnt in the past and think about what should I do in the next.
The technologies and tools I used in my projects is as following. There are still a lot of things to learn considering the list in my list and the huge ecosystem of kubernetes.
Posts
Resize volumes when PVCs and PVs are okay and the size of file systems in pods doesn't change
Here is an issue with aws-ebs-csi-driver: The size of file system doesn’t change when pvc is expanded. I got the same issue when I tried to do the Curl elk in pods to delete indices this afternoon. I got the message "resize2fs 1.44.5 (15-Dec-2018) open: No such file or directory while opening /dev/nvme1n1" as well when I tried to resize the file system /dev/nvme1n1 in my pod.
As the issue is about csidriver, it is not in the the result of running command "kubectl get csidriver" on my cluster.