TestProject Forum

Docker Agent Error Occurs in GKE which has Containerd based VM

As you might know, Kubernetes is no longer supporting Docker runtime.
Therefore using Container-Optimized OS with Containerd is recommended instead of Container-Optimized OS with Docker.
(Please see https://acloudguru.com/blog/engineering/kubernetes-is-deprecating-docker-what-you-need-to-know)

So basically, switching to Containerd-based COS should not affect the basic functionality.
However when I switch to Containerd rather than Docker, I met this kind of error logs:

2021-03-11 23:15:26.838 [ERROR] i.t.a.h Failed to initialize Agent
java.io.FileNotFoundException: Source ‘/opt/testproject/agent/drivers’ does not exist
at org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1383)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1346)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1246)
at org.apache.commons.io.FileUtils.copyDirectory(FileUtils.java:1215)
at io.testproject.agent.managers.q.b(TestProjectAgent:305)
at io.testproject.agent.h.e(TestProjectAgent:334)
at io.testproject.agent.e.a.K.a(TestProjectAgent:64)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at io.testproject.agent.c.d.run(TestProjectAgent:33)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)

Actually there was no INFO logs such as “Running inside docker aaa-bbb-ccc-xxxxxx”.
So I can assume that TestProject Docker Agent is only able to run in Docker-based K8 cluster at this point.

I would like to know two things as follows:

#1. Workaround
Is there any workaround that I can run TestProject Docker Agent in my case?

#2. Roadmap
Is there any plan to follow up on this kind of Kubernetes direction?

Hi @jzwaylove

TestProject Agent v0.66.4 adds support for non Docker based CRIs.
It should now work on GKE with any container runtime.

You can also run a Docker of your own as described here:
https://hub.docker.com/r/testproject/agent

Hi @marat

Thanks for your quick feedback.

I have tested with 1.17.15-gke.800 and TestProject Agent <= v0.66.3, and it failed.
Kindly let me try v0.66.4 and will get back to you with the result.

Hi @marat and all,

As you mentioned, now I can see the log saying “Running inside docker…”, and it started well as expected.

However, I have found another problem with v0.66.4.

[1.17.15-gke.800 with Docker and TestProject Agent v0.66.4]
The agent is working fine as follows:

[INFO] Searching for installed browsers…
[INFO] Getting external browser version via localhost:4444/wd/hub
[INFO] External CHROME version is 89.0.4389.82
[INFO] Finished detecting installed browsers.

[1.17.15-gke.800 with Containerd and TestProject Agent v0.66.4]
The agent is logging errors as follows while searching for installed browsers:

[INFO] Searching for installed browsers…
[INFO] Getting external browser version via localhost:4444/wd/hub
[ERROR] Failed to get external CHROME version via: http://localhost:4444/wd/hub
[INFO] Finished detecting installed browsers.

There was no problem with calling K8 DNS URL such as chrome.jenkins.svc.cluster.local:4444.
But when I create a multi-container pod including both chrome and testproject-agent, and trying to call chrome by localhost, it FAILED.

Therefore I can assume that communication via LOCALHOST is still not available with TestProject Agent in case of multi-container pod.

Could you please let me know if the same problem occurs at your end?

Thanks and regards.

Hello @jzwaylove
Did you manage to run the agent container in the same environment not as a pod in K8s but on its own using docker-compose for example?

I would like to see if it is a problem in the configuration of K8s or something in your environemtn.

Hi @ran.tzur and @marat

I’d appreciate your prompt and helpful advice.
Please disregard my additional question and requests regarding multi-container pod.
I think there was a temporary configuration error in our K8 cluster.
The same issue is not occurring at our end.
Kindly help me to close this ticket.

Thanks and regards,
Jay Lee

Hello @jzwaylove ,
Following is an example of how to run TestProject agent on Kubernetes with standalone-chrome and standalone-firefox .

Tested on

GKE Version 1.18.12-gke.1210
EKS Version 1.19

Agent

TestProject Agent => v0.66.4

# To execute this yml file use this command in your local kubernetes environment `kubectl apply -f k8s-testproject.yaml
#***************************************
#SELENIUM STANDALONE CHROME
#***************************************
apiVersion: v1
kind: Service
metadata:
  name: chrome
  labels:
    name: selenium-standalone-chrome
spec:
  selector:
    app: selenium-standalone-chrome
  ports:
    - name: "chrome"
      protocol: TCP    
      port: 4444
      targetPort: 4444
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: chrome
  labels:
    app: selenium-standalone-chrome
    name: selenium-standalone-chrome
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selenium-standalone-chrome
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: selenium-standalone-chrome
        name: selenium-standalone-chrome
      labels:
        app: selenium-standalone-chrome
        name: selenium-standalone-chrome                              
    spec:
      restartPolicy: Always        
      containers:
        - image: selenium/standalone-chrome
          name: chrome       
          ports:
            - containerPort: 4444
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
          resources:
            requests:
              memory: "1Gi"
              cpu: "1"
            limits:
              memory: "1Gi"
              cpu: "1"
      volumes:
        - name: dshm
          emptyDir: { "medium": "Memory" }
---          
#***************************************
#SELENIUM STANDALONE FIREFOX
#*************************************** 
apiVersion: v1
kind: Service
metadata:
  name: firefox
  labels:
    name: selenium-standalone-firefox
spec:
  selector:
    app: selenium-standalone-firefox
  ports:
    - name: "firefox"
      protocol: TCP    
      port: 4444
      targetPort: 4444
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: firefox
  labels:
    app: selenium-standalone-firefox
    name: selenium-standalone-firefox
spec:
  replicas: 1
  selector:
    matchLabels:
      app: selenium-standalone-firefox
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: selenium-standalone-firefox
        name: selenium-standalone-firefox
      labels:
        app: selenium-standalone-firefox
        name: selenium-standalone-firefox                              
    spec:
      restartPolicy: Always        
      containers:
        - image: selenium/standalone-firefox
          name: firefox       
          ports:
            - containerPort: 4444
          volumeMounts:
            - name: dshm
              mountPath: /dev/shm
          resources:
            requests:
              memory: "1Gi"
              cpu: "1"
            limits:
              memory: "1Gi"
              cpu: "1"
      volumes:
        - name: dshm
          emptyDir: { "medium": "Memory" }
---            
#***************************************
#TESTPROJECT AGENT
#***************************************           
apiVersion: v1
kind: Service
metadata:
  name: testproject-agent
  labels:
    name: testproject-agent
spec:
  selector:
    app: testproject-agent
  ports:
    - name: "8585"
      port: 8585
      targetPort: 8585
    - name: "8686"
      port: 8686
      targetPort: 8686
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: testproject-agent
    name: testproject-agent
  name: testproject-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testproject-agent
  template:
    metadata:
      labels:
        app: testproject-agent
        name: testproject-agent
    spec:
      restartPolicy: Always    
      containers:
        - env:
            - name: CHROME
              value: chrome:4444
            - name: FIREFOX
              value: firefox:4444
            - name: TP_AGENT_TEMP
              value: "true"
            - name: TP_AGENT_ALIAS # OPTIONAL variable that can be used for registering the agent with a custom alias.
              value: "My First k8s Agent"             
            - name: TP_API_KEY # The API key can be created in the TestProject Web Application (as described https://docs.testproject.io/testproject-integrations/integration-with-jenkins#creating-a-testproject-api-key).
              valueFrom:
                secretKeyRef:
                  name: testproject-secrets
                  key: TP_API_KEY             
            - name: TP_SDK_PORT # Agent will listen on port 8686 for SDK connections
              value: "8686"
          image: testproject/agent:latest
          args: 
          - sleep 60 # The chrome pod initializing take time. We need to start agent after the chrome pod can accept connections.
          name: testproject-agent
          ports:
            - containerPort: 8585
            - containerPort: 8686
          resources:
            requests:
              memory: "1Gi"
              cpu: "1"
            limits:
              memory: "1Gi"
              cpu: "1"      
---
apiVersion: v1
kind: Secret
metadata:
  name: "testproject-secrets"
type: Opaque
stringData:
  # Secret environment variables
  TP_API_KEY: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

Please let me know if this works for You.
Thank You!