DNS issues with AWS EC2 images and systemd-resolved
Some DNS issues I faced with latest EC2 Ubuntu images and systemd-resolved.
Posts by Year
2020
Kops Kubernetes Cluster upgrades
Some notes and rules on upgrades to the Kubernetes clusters with Kops I’ve adopted during more than 3 years of working with Kops and Kubernetes. I always fol...
Kubernetes NFS shared storage in AWS with EFS
AWS efs-provisioner plugin
2019
Building Multi Architecture Golang Docker Images with GitLab CI/CD
The motivation is to provide Docker images for use with the AWS EC2 A1 Instances that deliver significant cost savings and are ideally suited for scale-out a...
Encrypted DNS with BIND and DNSCrypt
DNSCrypt is a protocol that authenticates communications between a DNS client and a DNS resolver. It works by encrypting all DNS traffic between the user and...
Kops Kubernetes upgrade to etcd-v3 with etcd-manager and calico-v3
This is to document a procedure I followed during Kubernetes cluster upgrade from 1.10 to latest 1.12 with kops. I’ve been using kops for all our test and pr...
Etcd cluster member recovery in Kubernetes
This is a process I followed to recover one of the etcd masters that was broken after unsuccessful kops upgrade. Login to one of the healthy etcd cluster mem...
2018
GitLab CI/CD Multi-project Pipelines for Terraform and AWS
I use Terraform to provision our AWS infrastructure. Each production and staging environment gets provisioned in its own VPC and each service is clustered or...
Ad blocking with BIND DNS
There are couple of options to block ads in Bind DNS server like ad block Zone files or RPZ (Response Policy Zones).
Kubernetes HPA Autoscaling with Custom Metrics
The initial Horizontal Pod Autoscaler was limited in features and it only supported scaling deployments based on CPU metrics. The most recent Kubernetes rele...
Overlay SDN with VxLAN, BGP-EVPN and FRR
In BGP based control plane for Vxlan, E-VPN plays the role of a distributed controller for layer-2 network virtualization. BGP is the routing protocol of the...
GitLab CI/CD Pipelines for Kubernetes clusters
GitLab is a versatile open source (CE edition) tool that provides Git stile project repository, CI/CD pipelines and private Container Image Registry for the ...
Disk encryption at rest with LUKS and AWS SSM and KMS services in Systemd
Implementing disk encryption-at-rest in secure and automated way can be challenging. After we are done with the disk encryption we are often faced with the p...
ZFS NAS with NFS and Samba on ROCK64 ARM SBC
The old home NAS I built about 3 years ago died on me suddenly. It was a mini-ITX AMD board powered by freeNAS with 2 x 1TB Seagate drives in ZFS mirror. Sin...
Traefik with etcdv2 backend and Let’s Encrypt SSL certificates on Kubernetes
This post is an extension of a previous one Kubernetes cluster step-by-step: Services and Load Balancing about Traefik and its usage in Kubernetes. This time...
2017
HAProxy DDOS protection and API rate limiting
HAProxy is great reverse proxy and load balancer but can also be used for DDOS protection and rate limiting with great success. The below configuration provi...
Kubernetes RBAC Authorization and LDAP Authentication with Tokens using API Webhook and kube-ldap-authn
I’ve been looking for unified authentication solution that will work across all our Kubernetes cluster. Most specifically a solution that would utilize our e...
Build NGINX 1.10.3 from package sources with added Naxsi, PageSpeed, LDAP and GeoIP on Ubuntu-16.04 Xenial
The Nginx packages in Ubuntu Xenial do not come with some modules that are one of the most important when setting up Nginx for production use, like LDAP, Nax...
Automated SSL Certificates management with HAProxy, Consul and Lets Encrypt on AWS
Let’s Encrypt has quickly become a standard in obtaining and managing TLS certificates. It is a service provided by the Internet Security Research Group (ISR...
Secure Squid3 Reverse Proxy with SSL and LDAP Authentication on Ubuntu-16.04 Xenial
As described on it’s website Direct SSL/TLS connection, Squid can be used for SSL termination in reverse proxy mode. The SSL is not enabled by default in the...
Hghly Available Caching Cluster with Varnish and HAProxy in AWS
Varnish is a smart caching reverse-proxy and web application accelerator. According to its documentation Varnish Cache is really fast. It typically speeds up...
Pacemaker VM cluster fencing in Proxmox PVE with fence_pve
We can use the fence_pve agent to fence/stonith peers in Pacemaker cluster running on VM’s in Proxmox PVE host(s). This works and has been tested on Ubuntu-1...
Apache Traffic Server as Caching Reverse Proxy
Apache Traffic Server is a high-performance web proxy cache that improves network efficiency and performance by caching frequently-accessed information at th...
HAProxy dynamic backends with Consul and Consul Template in AWS
Consul has been part of our infrastructure for almost two years now. Each of our VPCs gets Consul cluster installed and configured via Terraform and Ansible ...
Geo Location with HAProxy
Often there might be need to allow, block or redirect users based on the country or continent they come from. This is how to do it with HAProxy.
Kubernetes cluster step-by-step: Services and Load Balancing
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: Kubernetes Add-ons
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: Kubelet, Kube-scheduler and Kube-controller-manager
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: Kube-apiserver with Keepalived and HAProxy for HA
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: FlannelD
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: ETCD
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: Binaries, Certificates, Kubeconfig and Tokens
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Kubernetes cluster step-by-step: Nodes System Setup
The purpose of this exercise is to create local Kubernetes cluster for testing deployments. It will be deployed on 3 x VMs (Debian Jessie 8.8) nodes which wi...
Linux Container Basics
Containers are nothing but isolated groups of processes running on a single host. That isolation leverages several underlying technologies built into the Lin...
Lets Encrypt and DANE
For quite some time I’ve been using certificate issued by StartSSL CA for my personal website. It’s for free and the recent refresh of their web portal they ...
Docker Private Registry with S3 backend on AWS
Our current Docker Hub Registry at https://hub.docker.com provides for a single private repository. This means all our private images must be stored there wh...
GitLab Server with LDAP and S3 backend
This is a procedure that enables S3 as backend storage for a GitLab Image Registry with LDAP for secure access and user authentication.
Kubernetes - Exposing External Services to Pods via Consul
Using Ingresses and Services of various types we can expose the k8s cluster services for use outside the cluster. Now we need to do the opposite, let our Pod...
Kubernetes shared block storage with external GlusterFS backend
There are many options available in Kubernetes when it comes to shared storage. I’m using here a GlusterFS cluster as backend for the shared storage in a k8s...
Kubernetes shared storage with S3 backend
There are many options available in Kubernetes when it comes to shared storage. I’m using a S3 bucket as backend for the shared storage in a k8s cluster in A...
Kubernetes Cluster External Services
On This Page Previously created Service works nice but only if we have ALL our services deployed as containers which, at least at the beginning, ...
Kubernetes Applications and Services
On This Page In my previous post Kubernetes Cluster in AWS with Kops I deployed a Kubernetes cluster with fully private topology (subnets and DNS...
Kubernetes Cluster in AWS with Kops
Kubernetes is a platform for deploying and managing containers. It is production-grade, open-source infrastructure for the deployment, scaling, management, a...
Docker ECS cluster with Terraform
Introduction
OpenATTIC 2-node cluster setup
OpenATTIC is an opensource converged storage that I think has a great potential to become a unified SDS for virtualization platforms. It offers features like...
PostgreSQL High Availibility with Pacemaker
Setting up PostgreSQL synchronous or asynchronous replication cluster with Pacemaker is described in couple of resources like the official Pacemaker site PgS...
2016
Highly Available NAT/Proxy EC2 instance with Keepalived
The two EC2 instances we are using as GW are launched in different AZ’s and are running Ubuntu-16.04. Each instance has one primary and one secondary IP atta...
Hosting static Website with S3 bucket and CloudFront CDN
Amazon AWS offers convenient way for hosting static website via S3 bucket providing CDN caching and SSL encryption using CloudFront.
Proxmox device hot-plugging in guests
This should be pretty straightforward, adding:
HA Features in Proxmox PVE cluster and final words
At the end, some testing of the High Availability fatures in PVE 4.2 on node and VM/LXC level.
Cluster Networking for Multi-tenant isolation in Proxmox with OpenVSwitch
This is probably the most complex part of the setup. It involves network configuration of the cluster in a way that the instances running on different nodes ...
Adding ZFS over iSCSI shared storage to Proxmox
PVE-4.2 has built in support for ZFS over iSCSI for several targets among which is Solaris COMSTAR. I built a ZFS VM appliance based on OmniOS (Solaris) and ...
Adding RBD (CEPH) remote cluster storage to Proxmox
There is a 3 node CEPH cluster running on the office virtualization server that is external to PVE. The latest PVE though has built in support for CEPH using...
PostgreSQL Confluence DB replication with Bucardo
In cases where we can’t use the built-in PostgreSQL replication facility, like for example Confluence DB which has replication protection, Bucardo is very ef...
Adding iSCSI shared volume to Proxmox to support Live Migration
We will use Multipath for link HA and improved performance. Install the needed packages first:
Adding DRBD shared volumes to Proxmox to support Live Migration
The plan is to create 2 resources in Primary/Primary mode. The first one r0 will be used to store disk images for VM’s running on proxmox01 and r1 for the VM...
Adding GlusterFS shared storage to Proxmox to support Live Migration
To be able to move VM’s from one cluster member to another their root, and in fact any other attached disk, needs to be created on a shared storage. PVE has ...
Proxmox clustering and nested virtualization
The motivation for creating this setup is the possibility of having Encompass private virtualization cloud deployed in any third party infrastructure provide...
Duplicity encrypted backups to Amazon S3
Duplicity is a tool for creating bandwidth-efficient, incremental, encrypted backups. It backs directories by producing encrypted tar-format volumes and uplo...
Building VPC with Terraform in Amazon AWS
On This Page Terraform is a tool for automating infrastructure management. It can be used for a simple task like managing single application inst...
ZFS storage with OmniOS and iSCSI
On This Page The following setup of iSCSI shared storage on cluster of OmniOS servers was later used as ZFS over iSCSI storage in Proxmox PVE, se...
Using data source to mitigate lack of intermediate variables and interpolation
Just something I dug out in the Terraform forum and would like to keep as a reminder for the future. Terraform will not allow us to do something like this:
File System sync with Csync2 and Lsyncd
In this scenario we are migration from old 2.x to a new 3.0 Nexus instance in EC2 and we need to keep the new and old Nexus instances in sync until the migra...
Replacing GlusterFS failed node
In the following scenario the node 10.66.4.225 has become unresponsive and has been terminated. This leaves us with the following state on the cluster:
Clustering with Pacemaker, DRBD and GFS2 on Bare-Metal servers in SoftLayer
On This Page SoftLayer is IBM company providing cloud and Bare-Metal hosting services. We are going to setup a cluster of Pacemaker, DRBD and GFS...
Route53 Zone export
You need to have your local system ready for AWS access.
Tomcat9, ECDSA/ECC (Elliptic Curve) Certificates and HTTP/2
Tomcat9 brings bunch of new features of which support for HTTP/2 and multiple certificates per Virtual Host via SNI extension are most important ones…
ActiveMQ Master/Slave KahaDB on OCFS2 shared file system
During my tests of shared storage clusters I wondered if ActiveMQ supports file locking on OCFS2 file system which I used on couple of occasions. While looki...
Highly Available iSCSI ALUA (Asymetric Logical Unit Access) Storage with Pacemaker and DRBD in Dual-Primary mode - Part2
On This Page This is continuation of the Highly Available iSCSI ALUA Storage with Pacemaker and DRBD in Dual-Primary mode series. We have setup t...
Highly Available iSCSI ALUA (Asymetric Logical Unit Access) Storage with Pacemaker and DRBD in Dual-Primary mode - Part1
On This Page I already wrote a post on this topic so this is kind of extension or variation of the setup described here Highly Available iSCSI St...
LXC on Debian
A diary of a process of setting up LXC containers and networking on Debian.
Highly Available iSCSI Storage with SCST, Pacemaker, DRBD and OCFS2 - Part2
On This Page This is continuation of the Highly Available iSCSI Storage with SCST, Pacemaker, DRBD and OCFS2 series. We have setup the HA backing...
Highly Available iSCSI Storage with SCST, Pacemaker, DRBD and OCFS2 - Part1
On This Page SCST the generic SCSI target subsystem for Linux, allows creation of sophisticated storage devices from any Linux box. Those devices...
S3 buckets as file system storage
s3fs is a direct mapping of S3 to a file system paradigm. Files are mapped to objects. File system meta-data (e.g. ownership and file modes) are stored insid...
2015
HAProxy Load Balancer with sticky sessions
HAProxy is highly customizable and function reach software load balancer. The below section outlines the installation and configuration of HAProxy as https l...
EBS volumes with LUKS encryption
Encrypting data at rest provides protection of sensitive information stored on EBS volumes. When taking snapshots of encrypted volumes the snapshots are encr...
Resolving GlusterFS conflicts using extended attributes metadata
This is a walk through example of resolution of conflict created as result of split-brain.
GlusterFS metadata split brain recovery
While investigating an error related to failed documents I came across following error in the GlusterFS healing daemon log file:
GlusterFS internals
GlusterFS stores metadata info in extended attributes which is supported and enabled by default in the XFS file system we use for the bricks. This is differe...
HAProxy dynamically adjust server weight using external agent
Trying to utilize HAProxy-1.5/1.6 agent-check feature, see HAProxy documentation, I wrote this small script to check Tomcat system load and return back some ...
Consul Service Discovery, Distributed KV Store and much more
Consul is completely distributed, highly available service discovery tool that can scale to thousands of nodes and services across multiple datacenters. In a...
Centralized logs collection with Logstash, ElasticSearch and Kibana in Amazon AWS
On This Page Logstash is a tool for managing events and logs. It is very useful for collecting, parsing and storing logs for later use like for e...
Setting up Encompass Maintenance Page with ELB, S3, Route53 and CloudFront
This is for the environments we have ELB (Elastic Load Balancer) instead of HAProxy. The idea is to host the maintenance page as static website in S3 bucket ...
Setting up Encompass Maintenance Page with HAProxy and NodeJS http-server
At Encompass we use HAProxy as a load balancer due to its speed, stability and welth of features. This is how we set our maintenance page to be servered by H...
Horde Groupware Webserver
On This Page Horde Groupware Webserver Edition is a free, enterprise ready, browser based communication suite. Users can read, send and organize ...
HAProxy OCSP stapling
The Online Certificate Status Protocol (OCSP) is an Internet protocol used for obtaining the revocation status of an X.509 digital certificate. It is used b...
Web site statistics with AWStats
Awstats (Apache Web Statistics) is powerful and highly customizable tool for collecting web site statistics. The purpose of this document is to show one way ...
PostgreSQL Archiving Replication
In this mode PostgreSQL replicates the WAL archive logs.
PostgreSQL Streaming Replication
Streaming replication means the changes are synchronously applied from the master to the slave(s).
SRIOV Enhanced Networking in AWS EC2 on Ubuntu-14.04 HVM
The latest EC2 generation of HVM instances makes use of the Enhanced Networking, utilizing the ixgbevf e1000 Gigabit Virtual Function Network Driver which pr...
Content upload to Joomla! website hosted in AWS EC2
We have a Joomla! website hosted by clustered services on couple of EC2 instances. The document root resides on shared storage provided by GlusterFS. We need...
Server side PHP caching in Joomla! with Repcached
Caching provides significant performance speed up since reading data from the memory is much faster then reading it from the database or disk, especially if ...
Managing system resources with Cgroups and Ansible
Sometimes we need to limit particular resource usage for some process, utility or group of processes in order to prioritize or limit their usage. One way to ...
2014
Dynamic DNS (DDNS) in AWS for internal zones
The following configuration will resolve the internal (meaning inside VPC) domain queries for encompasshost.com and forward all other queries to the default ...
Deploying Encompass In Docker Containers
At the beginning, just a short summery of how we can start using out container images.
Container Networking
The previous related post Building custom Docker images and configuring with Ansible talked about creating our own customized images and running our applicat...
Building custom Docker images and configuring with Ansible
On This Page Due to ever rising popularity of Docker this page will provide a walk through process of building custom Encompass Docker images and...
Host-based IDS with Snort, Barnyard2 and Snorby in AWS
On This Page Snort is open source network-based intrusion detection system (NIDS) that has the ability to perform real-time traffic analysis and ...
MySQL SSL and client certificates authentication
First lets create a small Camel database with couple of tables on our server.mydomain.com host using the following script:
HAProxy dynamic backend updates with Ansible
Due to some ELB limitations that did not play well with our user case like limited session timeout to 17 minutes, lack of multizone balancing, url rewriting ...
Monitoring and data collection with Logstash, Graphite, Grafana and Statsd
On This Page We have already setup our centralized log collection system based on Logstash as described in this article Centralized logs collecti...
OpenStack Icehouse Multi-node Installation with Ceph backend for Cinder and Glance
On This Page This is a standard Installation of OpenStack Icehouse on 3 x VM nodes: Controller, Compute and Networking. Later I decided to create...
MySQL High Availability and Load Balancing with Keepalived
What we want to achieve here is have a MySQL HA two nodes cluster in Master-Master mode and load balance the instances using as less hardware as possible. Th...
MySQL Circular Replication
Setting the MySQL in Master-Master mode means in case of an instance failure the other one will transparently take over the client connections avoiding the n...
Secure Nginx with Naxsi, SSL, GeoIP and Google Page Speed modules on Debian/Ubuntu
We will use the latest stable version of nginx-naxsi which has XSS (Cross Site Scripting) protection via Naxsi module. We will also build and install this De...
Nginx LDAP module on Debian/Ubuntu
Nginx by default contains the core modules needed which makes it light and lean web server. Any additional stuff needed have to be recompiled and added as mo...
Ceph cluster on Ubuntu-14.04
As pointed on its home page, Ceph is a unified, distributed storage system designed for performance, reliability and scalability. It provides seamless access...
GlusterFS orphaned GFID hard links
Orphaned GlusterFS GFID’s are hard links under the $BRICK/.glusterfs directory that point to an inode of a file that has been removed manually, outside of th...
LDAP replication for Directory High-Availability
As said before, once the users and services rely on the LDAP server for providing credentials and permissions the LDAP server becomes crucial part of any set...
Unified Linux Login, Privileges and Home Directory with OpenLDAP and NFS/automount
On This Page Maintaining users, shared file systems and authentication in centralized manner is one of the biggest challenges for a organization ...
VIP(EIP) fail over with Keepalived in Amazon VPC across availability zones
This example covers VIP failover in AWS VPC across AZ’s with Keepalived. The main problem in AWS is that this provider is blocking the multicast traffic in t...
Ansible summary
Summary of Ansible features.
ODROID-U3 as wireless router
ODROID-U3 is tiny SBC from Hardkernel packing quad-core CPU and 2GB of RAM.
Resolving GlusterFS split brain
I was running a load test against our Staging stack the other day and noticed that application broke down at around 100 users under Siege. Checking the logs ...
Securing Logstash to Redis communication with Stunnel
Logstash is meant for private LAN usage since it doesn’t offer any kind of encryption support. If we need to ship sensitive data across WAN’s, like between A...
IPSEC VPN tunnel setup between two Amazon VPC’s with OpenSwan and EC2 NAT instances
With services running in multiple VPC’s sooner or later a need will arise for secure clustering of instances across regions. This is especially important in ...
IPSec VPN server setup in Amazon VPC with OpenSwan
On This Page The access to our Amazon VPC’s atm is based on ssh key pairs. While this is working fine and is pretty much secure it requires thoug...
2013
Tomcat7 clustering and session replication in AWS
This was a POC for Tomcat clustering and session replication in AWS. It has been set up and tested on a pair of EC2 instances (ip-172-31-13-11 and ip-172-31-...
Joining Linux and Mac workstation to Windows AD Domain
The Likewise package can be used to join Mac and Linux boxes to Windows AD domain. The company has been acquired by Beyond Trust couple of years ago and is n...
Windows Active Directory with SAMBA4
On This Page Setting up an Active Directory server for company domain is a must in these days. It provides centralized management of user rights ...
Build MongoDB with SSL support
The free source version of MongoDB 2.x does not come with SSL support. To enable it we need to build it from source with --ssl option at compile time or use ...
JVM and GC tuning for 64 bit Java 6 applications
On This Page This article describes the options used in our production Amazon AWS servers for JVM and GC tuning. It also gives a short overview o...
MongoDB Replica Set setup
The replica set will consist of 3 nodes (given with their host names) created and hosted in Amazon EC2: ip-172-31-16-61 (PRIMARY), ip-172-31-16-62 (SECONDARY...