Kubernetes Agent DNS issues

jfurey · February 16, 2022, 11:03am

I have attempted to follow the instruction here

in an attempt to configure a test agent to run in our Kubernetes cluster

However we can’t get it working, I dug into the issue and I found the following
when the image startups up I’m met with this rather obscure error

Using Agent Alias: My First k8s Agent
Marking Agent as temporary...
Failed obtaining agent configuration!
Error code: 
Request ID: N/A
Error message: N/A

some debug info from the container itself while it’s running

/opt/testproject/agent $ cat /etc/os-release 
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"

/opt/testproject/agent $ curl -v -s -i -X POST "https://api.testproject.io/v2/agents/config/v2/agents/config" -H "accept: application/json" -H "Authorization: <redacted>" 
* Could not resolve host: api.testproject.io
* Closing connection 0

it’s not starting due it a dns error Could not resolve host: api.testproject.io

I see the agent images “testproject/agent:latest” are based on alpine 3.15
which has a known issue with DNS not working correctly inside Kubernetes

see :

github.com/kubernetes-sigs/kind

DNS not working on Alpine <= 3.10

opened 04:52PM - 15 Aug 21 UTC

paolomainardi

kind/bug kind/external

**What happened**: All is started by having a problem with an nginx container… having this conf: ``` location ~* /sites/default/files/(.*) { set $bucket_base_url http://drupal-minio.default.svc.cluster.local:9000/bucket/public; set $file_path $1; resolver kube-dns.kube-system.svc.cluster.local; proxy_hide_header Cache-Control; add_header Cache-Control "public,max-age=3600"; proxy_pass $bucket_base_url/$file_path$is_args$args; } ``` The offending part is: `resolver kube-dns.kube-system.svc.cluster.local;` It crashes because it cannot resolve the host. This nginx image is `nginx:1.17.6-alpine` which is based on alpine `3.10`, running some smoke tests it confirms the behaviour: ``` ❯ kubectl run --rm --attach --restart Never --image alpine:3.10 alpine -- nslookup kube-dns.kube-system.svc.cluster.local If you don't see a command prompt, try pressing enter. nslookup: can't resolve 'kube-dns.kube-system.svc.cluster.local': Try again pod "alpine" deleted pod default/alpine terminated (Error) ~ 14s ❯ kubectl run --rm --attach --restart Never --image alpine:3.11 alpine -- nslookup kube-dns.kube-system.svc.cluster.local If you don't see a command prompt, try pressing enter. Error attaching, falling back to logs: unable to upgrade connection: container alpine not found in pod alpine_default Server: 10.96.0.10 Address: 10.96.0.10:53 Name: kube-dns.kube-system.svc.cluster.local Address: 10.96.0.10 pod "alpine" deleted ❯ kubectl run --rm --attach --restart Never --image nginx:1.17.6-alpine nginx-alpine -- nslookup kube-dns.kube-system.svc.cluster.local If you don't see a command prompt, try pressing enter. nslookup: can't resolve 'kube-dns.kube-system.svc.cluster.local': Try again pod "nginx-alpine" deleted pod default/nginx-alpine terminated (Error) ~ 6s ❯ kubectl run --rm --attach --restart Never --image nginx:1.21.1-alpine nginx-alpine -- nslookup kube-dns.kube-system.svc.cluster.local Server: 10.96.0.10 Address: 10.96.0.10:53 *** Can't find kube-dns.kube-system.svc.cluster.local: No answer Name: kube-dns.kube-system.svc.cluster.local Address: 10.96.0.10 pod "nginx-alpine" deleted ``` **How to reproduce it (as minimally and precisely as possible)**: The cluster is just a plain cluster created with `kind create cluster --name cluster-name --wait 60s --image kindest/node:v1.19.11@sha256:07db187ae84b4b7de440a73886f008cf903fcf5764ba8106a9fd5243d6f327` The commands are reported below. **Anything else we need to know?**: **Environment:** - kind version: (use `kind version`): `kind v0.11.1 go1.16.4 linux/amd64` - Kubernetes version: (use `kubectl version`): ```Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.11", GitCommit:"c6a2f08fc4378c5381dd948d9ad9d1080e3e6b33", GitTreeState:"clean", BuildDate:"2021-05-27T23:47:11Z", GoVersion:"go1.15.12", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.21) and server (1.19) exceeds the supported minor version skew of +/-1 ``` - Docker version: (use `docker info`): ``` Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., v0.9.1-beta3) buildx: Build with BuildKit (Docker Inc., v0.6.1-docker) scan: Docker Scan (Docker Inc., v0.8.0) Server: Containers: 42 Running: 1 Paused: 0 Stopped: 41 Images: 93 Server Version: 20.10.8 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2 Default Runtime: runc Init Binary: docker-init containerd version: e25210fe30a0a703442421b0f60afac609f950a3 runc version: v1.0.1-0-g4144b63 init version: de40ad0 Security Options: apparmor seccomp Profile: default Kernel Version: 5.11.0-25-generic Operating System: Ubuntu 21.04 OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.32GiB Name: spark-carbon-cto ID: XZHS:42UR:QTQV:VROZ:I7QE:T7VO:TXNG:DLAT:PEFI:OIMI:4QVP:XEWS Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` - OS (e.g. from `/etc/os-release`): ``` NAME="Ubuntu" VERSION="21.04 (Hirsute Hippo)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 21.04" VERSION_ID="21.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=hirsute UBUNTU_CODENAME=hirsute ```

I have confirmed that the API call works from a Debian based image inside the same cluster.
Could we possibly get a Debian based agent image so we can get this running in a kubernetes cluster?

gershon.alvais1 · February 17, 2022, 9:34am

Hello,
Can’t confirm the same.
Tested on Kubernetes v1.22.5 from Docker Desktop.
Tested on Kubernetes v 1.21 AWS Eks
Seem The described issue is more related to kind

/opt/testproject/agent $ cat /etc/os-release
NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.15.0
PRETTY_NAME="Alpine Linux v3.15"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://bugs.alpinelinux.org/"
/opt/testproject/agent $ cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
/opt/testproject/agent $

jfurey · February 17, 2022, 1:16pm

I cannot replicate it in minikube locally either running.
however I can replicate it reliably using alpine:3.15 running in our GKE clusters ( k8s 1.22.6-gke.300)
funnily enough older version of alpine do work. I have tested connectivity from an alpine:3.1 image and it works fine

Anyway, Alpine images have a long history of being problematic in kubernetes. it’s even called out in the google docs themselves when troubleshooting dns problems