[{"data":1,"prerenderedAt":717},["ShallowReactive",2],{"/en-us/blog/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes/":3,"navigation-en-us":34,"banner-en-us":463,"footer-en-us":480,"Steve Azzopardi":690,"next-steps-en-us":702},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":24,"_id":27,"_type":28,"title":29,"_source":30,"_file":31,"_stem":32,"_extension":33},"/en-us/blog/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"How we reduced 502 errors by caring about PID 1 in Kubernetes","For every deploy, scale down event, or pod termination, users of GitLab's Pages service were experiencing 502 errors. This explains how we found the root cause and rolled out a fix for it.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749682305/Blog/Hero%20Images/KubeCon2022.jpg","https://about.gitlab.com/blog/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"How we reduced 502 errors by caring about PID 1 in Kubernetes\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Steve Azzopardi\"}],\n        \"datePublished\": \"2022-05-17\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Steve Azzopardi","2022-05-17","\n\n_This blog post and linked pages contain information related to upcoming products, features, and functionality. It is important to note that the information presented is for informational purposes only. Please do not rely on this information for purchasing or planning purposes. As with all projects, the items mentioned in this blog post and linked pages are subject to change or delay. The development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._\n\nOur [SRE on call](https://about.gitlab.com/handbook/engineering/infrastructure/incident-management/#engineer-on-call-eoc-responsibilities)\nwas getting paged daily that one of our\n[SLIs](https://www.youtube.com/watch?v=tEylFyxbDLE) was\nburning through our\n[SLOs](https://www.youtube.com/watch?v=tEylFyxbDLE) for the [GitLab\nPages](https://docs.gitlab.com/ee/user/project/pages/) service. It was\nintermittent and short-lived, but enough to cause user-facing impact which we\nweren't comfortable with. This turned into alert fatigue because there wasn't\nenough time for the SRE on call to investigate the issue and it wasn't\nactionable since it recovered on its own.\n\nWe decided to open up an [investigation issue](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497)\nfor these alerts. We had to find out what the issue was since we were\nshowing `502` errors to our users and we needed a\n[DRI](https://about.gitlab.com/handbook/people-group/directly-responsible-individuals/)\nthat wasn't on call to investigate.\n\n## What is even going on?\n\nAs an [SRE](https://handbook.gitlab.com/job-families/engineering/infrastructure/site-reliability-engineer/)\nat GitLab, you get to touch a lot of services that you didn't build yourself and\ninteract with system dependencies that you might have not touched before.\nThere's always detective work to do!\n\nWhen we looked at the GitLab Pages logs we found that it's always returning\n[`ErrDomainDoesNotExist`](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/e1f1effa23c520d3b8b717d831ccab7ba3dd494f/internal/routing/middleware.go#L22-26)\nerrors which result in a `502` error to our users. GitLab Pages [sends a request](https://gitlab.com/gitlab-org/gitlab-pages/-/blob/e1f1effa23c520d3b8b717d831ccab7ba3dd494f/internal/source/gitlab/client/client.go#L101-127)\nto [GitLab Workhorse](https://docs.gitlab.com/ee/development/workhorse/),\nspecifically the `/api/v4/internal/pages` route.\n\nGitLab Workhorse is a Go service in front of our Ruby on Rails monolith and\nit's deployed as a [sidecar](https://www.magalix.com/blog/the-sidecar-pattern)\ninside of the `webservice pod`, which runs Ruby on Rails using the `Puma` web\nserver.\n\nWe used the internal IP to correlate the GitLab Pages requests with GitLab Workhorse\ncontainers. We looked at multiple requests and found that all the 502 requests\nhad the following error attached to them: [`502 Bad Gateway with dial tcp 127.0.0.1:8080: connect: connection refused`](https://gitlab.com/gitlab-org/gitlab/-/blob/f64be48cc737f5d12c1c30f724af540a836dcc94/workhorse/internal/badgateway/roundtripper.go#L43).\nThis means that GitLab Workhorse couldn't connect to the Puma web server. So we\nneeded to go another layer deeper.\n\nThe Puma web server is what runs the Ruby on Rails monolith which has an\ninternal API endpoint but Puma was never getting these requests since it wasn't\nrunning. What this tells us is that Kubernetes kept our pod in the\n[service](https://kubernetes.io/docs/concepts/services-networking/service/)\neven when Puma wasn't responding, despite having [readiness probes](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/4bb638bccc6a676f9fdd5bbf800f7d2b977efd55/charts/gitlab/charts/webservice/templates/deployment.yaml#L279-287)\nconfigured.\n\nBelow is the request flow between GitLab Pages, GitLab Workhorse, and Puma/Webservice to try and make it more clear:\n\n![overview of the request flow](https://about.gitlab.com/images/blogimages/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes/overview.png){: .shadow.center}\n\n## Attempt 1: Red herring\n\nWe shifted our focus on GitLab Workhorse and Puma to try and understand how\nGitLab Workhorse was returning 502 errors in the first place. We found some\n`502 Bad Gateway with dial tcp 127.0.0.1:8080: connect: connection refused`\nerrors during container startup time. How could this be? With the readiness\nprobe, the pod shouldn't be added to the\n[Endpoint](https://kubernetes.io/docs/concepts/services-networking/service/#over-capacity-endpoints)\nuntil [all readiness probes pass](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_899321775).\nWe later found out that it's because of a [polling\nmechanisim](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_899629314)\nthat we have for [Geo](https://docs.gitlab.com/ee/administration/geo/) which\nruns in the background, using a Goroutine in GitLab Workhorse, and pings Puma for Geo information.\nWe don't have Geo enabled on GitLab.com so we [simply disabled it](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1670)\nto reduce the noise.\n\nWe removed the 502 errors, but not the ones we want, just a red herring.\n\n## Attempt 2: Close but not quite\n\nAt this time, we were still burning through our SLO from time to time, so this\nwas still an urgent thing that we needed to fix. Now that we had cleaner logs for\n`502` errors it started to become a bit clearer that this is happening on pod\ntermination:\n\n```\n2022-04-05 06:03:49.000 UTC: Readiness probe failed\n2022-04-05 06:03:51.000 UTC: Puma (127.0.0.1:8080) started shutdown.\n2022-04-05 06:04:04.526 UTC: Puma shutdown finished.\n2022-04-05 06:04:04.000 UTC - 2022-04-05 06:04:46.000 UTC: workhorse started serving 502 constantly.  42 seconds of serving 502 requests for any request that comes in apart from /api/v4/jobs/request\n```\n\nIn the timeline shown above, we see that we've kept serving requests well after\nour `Puma`/`webservice` container exited, and the first readiness probe failed.\nIf we look at the readiness probes we had on that pod we see the following:\n\n```\n$ kubectl -n gitlab get po gitlab-webservice-api-785cb54bbd-xpln2 -o jsonpath='{range .spec.containers[*]} {@.name}{\":\\n\\tliveness:\"} {@.livenessProbe} {\"\\n\\treadiness:\"} {@.readinessProbe} {\"\\n\"} {end}'\n webservice:\n        liveness: {\"failureThreshold\":3,\"httpGet\":{\"path\":\"/-/liveness\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":20,\"periodSeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":30}\n        readiness: {\"failureThreshold\":3,\"httpGet\":{\"path\":\"/-/readiness\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":60,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":2}\n  gitlab-workhorse:\n        liveness: {\"exec\":{\"command\":[\"/scripts/healthcheck\"]},\"failureThreshold\":3,\"initialDelaySeconds\":20,\"periodSeconds\":60,\"successThreshold\":1,\"timeoutSeconds\":30}\n        readiness: {\"exec\":{\"command\":[\"/scripts/healthcheck\"]},\"failureThreshold\":3,\"periodSeconds\":10,\"successThreshold\":1,\"timeoutSeconds\":2}\n```\n\nThis meant that for the `webservice` pod to be marked unhealthy and removed\nfrom the endpoints, Kubernetes had to get 3 consecutive failures with an\ninterval of 10 seconds, so in total that's 30 seconds. That seems a bit slow.\n\nOur next logical step was to reduce the `periodSeconds` for the readiness probe\nfor the `webservice` pod so we don't wait 30 seconds before removing the pod\nfrom the service when it becomes unhealthy.\n\nBefore doing so we had to understand if sending more requests to `/-/readiness`\nendpoint would have any knock-on effect with using more memory or anything\nelse. We had to [understand what the `/-/readiness` endpoint was doing](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_903812722)\nand if it was safe to increase the frequency at which we send requests. We\ndecided it was safe, and after enabling it on\n[staging](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1686#note_903877755),\nand\n[canary](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1688#note_904501848)\nwe didn't see any increase in CPU/Memory usage, as expected, and saw an\nimprovement in the removal of 502 errors, which made us more confident that\nthis was the issue. We rolled this out to\n[production](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1689)\nwith high hopes.\n\nAs usual, Production is a different story than Staging or Canary, and it showed\nthat it didn't remove all the 502 errors, just [enough to stop triggering the SLO every day](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_905993144),\nbut at least we removed the alert fatigue on the SRE on call. We were close, but not quite.\n\n## Attempt 3: All gone!\n\nAt this point, we were a bit lost and weren't sure what to look at next. We had\na bit of tunnel vision and kept focusing/blaming that we aren't removing the\nPod from the `Endpoint` quickly enough. We even looked at [Google Cloud Platform\nNEGs](https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg) to\nsee if we could have faster readiness probes and remove the pod quicker. However,\nthis wasn't ideal [because we wouldn't have solved this for our self-hosting customers](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_908359286)\nwhich seem to be facing the same [problem](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2943).\n\nWhile researching we also came across a known problem with [running `Puma` in\nKubernetes](https://github.com/puma/puma/blob/bf2548ce300c2b4f671582bc756dcec5861e815f/docs/kubernetes.md),\nand thought that might be the solution. However, we already implemented a\n[blackout window](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/c1b63f3a4867886bc1212d86985fc70e66b717c5/charts/gitlab/charts/webservice/templates/deployment.yaml#L223-224)\njust for this specific reason, so it couldn't be that either...in other words, it was another dead end.\n\nWe took a step back and looked at the [timelines one more time](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_910106152)\nand then it hit us. The Puma/webservice container is terminating within a\nfew seconds, but the GitLab Workhorse one is always taking 30 seconds. Is it because\nof the [long polling from GitLab Runner](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21698)? 30 seconds\nis a \"special\" number for Kubernetes [pod termination](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination).\nWhen Kubernetes deletes a pod it firsts sends the `TERM` signal to the\ncontainer and waits 30 seconds, if the container hasn't exited yet, it will\nsend a `KILL` signal. This indicated that maybe GitLab Workhorse was never\nshutting down and Kubernetes had to kill it.\n\nOnce more we looked at GitLab Workhorse source code and [searched for the `SIGTERM` usage](https://gitlab.com/gitlab-org/gitlab/-/blob/d66f10e169a08cedcbfe70e3ea46cbfbb20d972d/workhorse/main.go#L238-258)\nand it did seem to support [graceful termination](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62701) and\nit also had explicit logic about long polling requests, so is this just another\ndead end? Luckily when the `TERM` signal is sent, Workhorse [logs a message that\nit's shutting down](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/62701). We looked\nat our logs for this specific message and didn't see anything. Is this it? We\naren't gracefully shutting down? But how? Why does it result in 502 errors?\nWhy do the GitLab Pages keep using the same pod that is terminating?\n\nWe know that the `TERM` signal is being sent to PID 1 inside of the container,\nand that process should handle the `TERM` signal for graceful shutdown. We\nlooked at the GitLab Workhorse process tree and this is what we found:\n\n```sh\ngit@gitlab-webservice-default-5d85b6854c-sbx2z:/$ ps faux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nroot        1015  0.0  0.0 805036  4588 ?        Rsl  13:12   0:00 runc init\ngit         1005  0.3  0.0   5992  3784 pts/0    Ss   13:12   0:00 bash\ngit         1014  0.0  0.0   8592  3364 pts/0    R+   13:12   0:00  \\_ ps faux\ngit            1  0.0  0.0   2420   532 ?        Ss   12:52   0:00 /bin/sh -c /scripts/start-workhorse\ngit           16  0.0  0.0   5728  3408 ?        S    12:52   0:00 /bin/bash /scripts/start-workhorse\ngit           19  0.0  0.3 1328480 33080 ?       Sl   12:52   0:00  \\_ gitlab-workhorse -logFile stdout -logFormat json -listenAddr 0.0.0.0:8181 -documentRoot /srv/gitlab/public -secretPath /etc/gitlab/gitlab-workhorse/secret -config /srv/gitlab/config/workhorse-config.toml\n```\n\nBingo! `gitlab-workhorse` is PID 19 in this case, and a child process of a\n[script](https://gitlab.com/gitlab-org/build/CNG/-/blob/92d3e22e9ff6c5cbb685aeea99813751d5e19a9d/gitlab-workhorse/Dockerfile#L51)\nthat we invoke. Taking a close look at the\n[script](https://gitlab.com/gitlab-org/build/CNG/-/blob/92d3e22e9ff6c5cbb685aeea99813751d5e19a9d/gitlab-workhorse/scripts/start-workhors)\nwe check if it listens to `TERM` and it doesn't! So far everything indicated\nthat GitLab Workhorse was never getting the `TERM` signal which ended up in receiving\n`KILL` after 30 seconds. We updated our `scripts/start-workhorse` to use\n[`exec(1)`](https://linux.die.net/man/1/exec) so that `gitlab-workhorse`\nreplaced the PID of our bash script, that should have worked, right? When we tested\nthis locally we then saw the following process tree.\n\n```\ngit@gitlab-webservice-default-84c68fc9c9-xcsnm:/$ ps faux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\ngit          167  0.0  0.0   5992  3856 pts/0    Ss   14:27   0:00 bash\ngit          181  0.0  0.0   8592  3220 pts/0    R+   14:27   0:00  \\_ ps faux\ngit            1  0.0  0.0   2420   520 ?        Ss   14:24   0:00 /bin/sh -c /scripts/start-workhorse\ngit           17  0.0  0.3 1328228 32800 ?       Sl   14:24   0:00 gitlab-workhorse -logFile stdout -logFormat json -listenAddr 0.0.0.0:8181 -documentRoot /srv/gitlab/public -secretPath /etc/gitlab/gitlab-workhorse/secret -config /srv/gitlab/config/workhorse-config.toml\n```\n\nThis changed a bit: this shows that `gitlab-workhorse` was no longer a child\nprocess of `/scripts/start-workhorse` however `/bin/sh` was still PID 1. What is even\ninvoking `/bin/sh` that we didn't see anywhere in our\n[Dockerfile](https://gitlab.com/gitlab-org/build/CNG/-/blob/92d3e22e9ff6c5cbb685aeea99813751d5e19a9d/gitlab-workhorse/Dockerfile)?\nAfter some thumb-twiddling, we had an idea that the container runtime is invoking\n`/bin/sh`. We went back to basics and looked at the\n[`CMD`](https://docs.docker.com/engine/reference/builder/#cmd) documentation to\nsee if we were missing something, and we were. We read the following:\n\n> If you use the shell form of the CMD, then the \u003Ccommand> will execute in `/bin/sh -c`:\n>\n> ```\n> FROM ubuntu\n> CMD echo \"This is a test.\" | wc -\n> ```\n>\n> If you want to run your \u003Ccommand> without a shell then you must express the command as a JSON array and give the full path to the executable. This array form is the preferred format of CMD. Any additional parameters must be individually expressed as strings in the array:\n>\n> ```\n> FROM ubuntu\n> CMD [\"/usr/bin/wc\",\"--help\"]\n> ```\n\nThis was exactly [what we were doing](https://gitlab.com/gitlab-org/build/CNG/-/blob/92d3e22e9ff6c5cbb685aeea99813751d5e19a9d/gitlab-workhorse/Dockerfile#L51)! \nwe weren't using `CMD` in `exec form`, but in `shell form`. Changing this confirmed\nthat `gitlab-workhorse` is now PID 1, and also receives the termination signal\nafter testing it locally:\n\n```\ngit@gitlab-webservice-default-84c68fc9c9-lzwmp:/$ ps faux\nUSER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\ngit           65  1.0  0.0   5992  3704 pts/0    Ss   15:25   0:00 bash\ngit           73  0.0  0.0   8592  3256 pts/0    R+   15:25   0:00  \\_ ps faux\ngit            1  0.2  0.3 1328228 32288 ?       Ssl  15:24   0:00 gitlab-workhorse -logFile stdout -logFormat json -listenAddr 0.0.0.0:8181 -documentRoot /srv/gitlab/public -secretPath /etc/gitlab/gitlab-workhorse/secret -config /srv/gitlab/config/workhorse-config.toml\n```\n\n```\n{\"level\":\"info\",\"msg\":\"shutdown initiated\",\"shutdown_timeout_s\":61,\"signal\":\"terminated\",\"time\":\"2022-04-13T15:27:57Z\"}\n{\"level\":\"info\",\"msg\":\"keywatcher: shutting down\",\"time\":\"2022-04-13T15:27:57Z\"}\n{\"error\":null,\"level\":\"fatal\",\"msg\":\"shutting down\",\"time\":\"2022-04-13T15:27:57Z\"}\n```\n\nOk, then we just needed to update `exec` and `CMD []` and we would have been\ndone, right? Almost. GitLab Workhorse proxies all of the requests for the API, Web, and Git requests so we couldn't just do a big change and expect that everything is going to be OK. We had to progressively roll this out to make\nsure we didn't break any existing working behavior since this affects all the\nrequests we get to GitLab.com. To do this, we hid it behind a [feature\nflag](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/972) so GitLab\nWorkhorse is only PID 1 when the `GITLAB_WORKHORSE_EXEC` environment variable\nis set. This allowed us to deploy the change and only enable it on a small part\nof our fleet to see if we see any problems. We were a bit more careful here and\nrolled it out zone by zone in Production since we run on 3 zones. When we\nrolled it out in the [first\nzone](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_919259030)\nwe saw all 502 errors disappear! After fully rolling this out we see that [the\nproblem is fixed and it had no negative side\neffects](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_920585707). Hurray!\n\nWe still had one question unanswered, why were GitLab Pages still trying to use\nthe same connection even after the Pod was removed from the Service because it was\nscheduled for deletion? When we looked at Go internals we see that [Go reuses\nTCP connections](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15497#note_920642770)\nif we close the body of the request. So even though it's not part of the Service\nwe can still keep the TCP connection open and send requests – this explains why\nwe kept seeing 502 on pod being terminated and always from the same GitLab\nPages pod.\n\nNow it's all gone!\n\n## More things that we can explore\n\n1. We've made graceful termination for GitLab Workhorse as [default](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/merge_requests/1732).\n1. Audit all of our Dockerfiles that use `CMD command` and fix them. We've found 10, and [fixed all of them](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3249).\n1. [Better readiness Probe defaults for `webservice` pod](https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/2518).\n1. Add [linting](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3253) for Dockerfiles.\n1. See if any of our child processes need [zombie process reaping](https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/).\n\n## Takeaways\n\n1. We should always care about what is PID 1 in a container.\n1. Always try and use `CMD [\"executable\",\"param1\",\"param2\"]` in your Dockerfile.\n1. Pods are removed from the Service/Endpoint in async.\n1. If you are on GKE [NEGs](https://cloud.google.com/kubernetes-engine/docs/how-to/standalone-neg) might be better for readinessProbes.\n1. By default, there is a 30 second grace period between the `TERM` signal and the `KILL` signal when Pods terminate. You can update the time between the signals `terminationGracePeriodSeconds`.\n1. The Go `http.Client` will reuse the TCP connection if [the body is closed](https://cs.opensource.google/go/go/+/refs/tags/go1.18.2:src/net/http/response.go;l=59-64) which in this case made the issue worse.\n\nThank you to [@igorwwwwwwwwwwwwwwwwwwww](https://gitlab.com/igorwwwwwwwwwwwwwwwwwwww), [@gsgl](https://gitlab.com/gsgl), [@jarv](https://gitlab.com/jarv), and [@cmcfarland](https://gitlab.com/cmcfarland) for helping me debug this problem!\n\n","engineering",[23],"kubernetes",{"slug":25,"featured":6,"template":26},"how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes","BlogPost","content:en-us:blog:how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes.yml","yaml","How We Removed All 502 Errors By Caring About Pid 1 In Kubernetes","content","en-us/blog/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes.yml","en-us/blog/how-we-removed-all-502-errors-by-caring-about-pid-1-in-kubernetes","yml",{"_path":35,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"data":37,"_id":459,"_type":28,"title":460,"_source":30,"_file":461,"_stem":462,"_extension":33},"/shared/en-us/main-navigation","en-us",{"logo":38,"freeTrial":43,"sales":48,"login":53,"items":58,"search":390,"minimal":421,"duo":440,"pricingDeployment":449},{"config":39},{"href":40,"dataGaName":41,"dataGaLocation":42},"/","gitlab logo","header",{"text":44,"config":45},"Get free trial",{"href":46,"dataGaName":47,"dataGaLocation":42},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":49,"config":50},"Talk to sales",{"href":51,"dataGaName":52,"dataGaLocation":42},"/sales/","sales",{"text":54,"config":55},"Sign in",{"href":56,"dataGaName":57,"dataGaLocation":42},"https://gitlab.com/users/sign_in/","sign in",[59,103,201,206,311,371],{"text":60,"config":61,"cards":63,"footer":86},"Platform",{"dataNavLevelOne":62},"platform",[64,70,78],{"title":60,"description":65,"link":66},"The most comprehensive AI-powered DevSecOps Platform",{"text":67,"config":68},"Explore our Platform",{"href":69,"dataGaName":62,"dataGaLocation":42},"/platform/",{"title":71,"description":72,"link":73},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":74,"config":75},"Meet GitLab Duo",{"href":76,"dataGaName":77,"dataGaLocation":42},"/gitlab-duo/","gitlab duo ai",{"title":79,"description":80,"link":81},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":82,"config":83},"Learn more",{"href":84,"dataGaName":85,"dataGaLocation":42},"/why-gitlab/","why gitlab",{"title":87,"items":88},"Get started with",[89,94,99],{"text":90,"config":91},"Platform Engineering",{"href":92,"dataGaName":93,"dataGaLocation":42},"/solutions/platform-engineering/","platform engineering",{"text":95,"config":96},"Developer Experience",{"href":97,"dataGaName":98,"dataGaLocation":42},"/developer-experience/","Developer experience",{"text":100,"config":101},"MLOps",{"href":102,"dataGaName":100,"dataGaLocation":42},"/topics/devops/the-role-of-ai-in-devops/",{"text":104,"left":105,"config":106,"link":108,"lists":112,"footer":183},"Product",true,{"dataNavLevelOne":107},"solutions",{"text":109,"config":110},"View all Solutions",{"href":111,"dataGaName":107,"dataGaLocation":42},"/solutions/",[113,138,162],{"title":114,"description":115,"link":116,"items":121},"Automation","CI/CD and automation to accelerate deployment",{"config":117},{"icon":118,"href":119,"dataGaName":120,"dataGaLocation":42},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[122,126,130,134],{"text":123,"config":124},"CI/CD",{"href":125,"dataGaLocation":42,"dataGaName":123},"/solutions/continuous-integration/",{"text":127,"config":128},"AI-Assisted Development",{"href":76,"dataGaLocation":42,"dataGaName":129},"AI assisted development",{"text":131,"config":132},"Source Code Management",{"href":133,"dataGaLocation":42,"dataGaName":131},"/solutions/source-code-management/",{"text":135,"config":136},"Automated Software Delivery",{"href":119,"dataGaLocation":42,"dataGaName":137},"Automated software delivery",{"title":139,"description":140,"link":141,"items":146},"Security","Deliver code faster without compromising security",{"config":142},{"href":143,"dataGaName":144,"dataGaLocation":42,"icon":145},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[147,152,157],{"text":148,"config":149},"Application Security Testing",{"href":150,"dataGaName":151,"dataGaLocation":42},"/solutions/application-security-testing/","Application security testing",{"text":153,"config":154},"Software Supply Chain Security",{"href":155,"dataGaLocation":42,"dataGaName":156},"/solutions/supply-chain/","Software supply chain security",{"text":158,"config":159},"Software Compliance",{"href":160,"dataGaName":161,"dataGaLocation":42},"/solutions/software-compliance/","software compliance",{"title":163,"link":164,"items":169},"Measurement",{"config":165},{"icon":166,"href":167,"dataGaName":168,"dataGaLocation":42},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[170,174,178],{"text":171,"config":172},"Visibility & Measurement",{"href":167,"dataGaLocation":42,"dataGaName":173},"Visibility and Measurement",{"text":175,"config":176},"Value Stream Management",{"href":177,"dataGaLocation":42,"dataGaName":175},"/solutions/value-stream-management/",{"text":179,"config":180},"Analytics & Insights",{"href":181,"dataGaLocation":42,"dataGaName":182},"/solutions/analytics-and-insights/","Analytics and insights",{"title":184,"items":185},"GitLab for",[186,191,196],{"text":187,"config":188},"Enterprise",{"href":189,"dataGaLocation":42,"dataGaName":190},"/enterprise/","enterprise",{"text":192,"config":193},"Small Business",{"href":194,"dataGaLocation":42,"dataGaName":195},"/small-business/","small business",{"text":197,"config":198},"Public Sector",{"href":199,"dataGaLocation":42,"dataGaName":200},"/solutions/public-sector/","public sector",{"text":202,"config":203},"Pricing",{"href":204,"dataGaName":205,"dataGaLocation":42,"dataNavLevelOne":205},"/pricing/","pricing",{"text":207,"config":208,"link":210,"lists":214,"feature":298},"Resources",{"dataNavLevelOne":209},"resources",{"text":211,"config":212},"View all resources",{"href":213,"dataGaName":209,"dataGaLocation":42},"/resources/",[215,248,270],{"title":216,"items":217},"Getting started",[218,223,228,233,238,243],{"text":219,"config":220},"Install",{"href":221,"dataGaName":222,"dataGaLocation":42},"/install/","install",{"text":224,"config":225},"Quick start guides",{"href":226,"dataGaName":227,"dataGaLocation":42},"/get-started/","quick setup checklists",{"text":229,"config":230},"Learn",{"href":231,"dataGaLocation":42,"dataGaName":232},"https://university.gitlab.com/","learn",{"text":234,"config":235},"Product documentation",{"href":236,"dataGaName":237,"dataGaLocation":42},"https://docs.gitlab.com/","product documentation",{"text":239,"config":240},"Best practice videos",{"href":241,"dataGaName":242,"dataGaLocation":42},"/getting-started-videos/","best practice videos",{"text":244,"config":245},"Integrations",{"href":246,"dataGaName":247,"dataGaLocation":42},"/integrations/","integrations",{"title":249,"items":250},"Discover",[251,256,260,265],{"text":252,"config":253},"Customer success stories",{"href":254,"dataGaName":255,"dataGaLocation":42},"/customers/","customer success stories",{"text":257,"config":258},"Blog",{"href":259,"dataGaName":5,"dataGaLocation":42},"/blog/",{"text":261,"config":262},"Remote",{"href":263,"dataGaName":264,"dataGaLocation":42},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":266,"config":267},"TeamOps",{"href":268,"dataGaName":269,"dataGaLocation":42},"/teamops/","teamops",{"title":271,"items":272},"Connect",[273,278,283,288,293],{"text":274,"config":275},"GitLab Services",{"href":276,"dataGaName":277,"dataGaLocation":42},"/services/","services",{"text":279,"config":280},"Community",{"href":281,"dataGaName":282,"dataGaLocation":42},"/community/","community",{"text":284,"config":285},"Forum",{"href":286,"dataGaName":287,"dataGaLocation":42},"https://forum.gitlab.com/","forum",{"text":289,"config":290},"Events",{"href":291,"dataGaName":292,"dataGaLocation":42},"/events/","events",{"text":294,"config":295},"Partners",{"href":296,"dataGaName":297,"dataGaLocation":42},"/partners/","partners",{"backgroundColor":299,"textColor":300,"text":301,"image":302,"link":306},"#2f2a6b","#fff","Insights for the future of software development",{"altText":303,"config":304},"the source promo card",{"src":305},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":307,"config":308},"Read the latest",{"href":309,"dataGaName":310,"dataGaLocation":42},"/the-source/","the source",{"text":312,"config":313,"lists":315},"Company",{"dataNavLevelOne":314},"company",[316],{"items":317},[318,323,329,331,336,341,346,351,356,361,366],{"text":319,"config":320},"About",{"href":321,"dataGaName":322,"dataGaLocation":42},"/company/","about",{"text":324,"config":325,"footerGa":328},"Jobs",{"href":326,"dataGaName":327,"dataGaLocation":42},"/jobs/","jobs",{"dataGaName":327},{"text":289,"config":330},{"href":291,"dataGaName":292,"dataGaLocation":42},{"text":332,"config":333},"Leadership",{"href":334,"dataGaName":335,"dataGaLocation":42},"/company/team/e-group/","leadership",{"text":337,"config":338},"Team",{"href":339,"dataGaName":340,"dataGaLocation":42},"/company/team/","team",{"text":342,"config":343},"Handbook",{"href":344,"dataGaName":345,"dataGaLocation":42},"https://handbook.gitlab.com/","handbook",{"text":347,"config":348},"Investor relations",{"href":349,"dataGaName":350,"dataGaLocation":42},"https://ir.gitlab.com/","investor relations",{"text":352,"config":353},"Trust Center",{"href":354,"dataGaName":355,"dataGaLocation":42},"/security/","trust center",{"text":357,"config":358},"AI Transparency Center",{"href":359,"dataGaName":360,"dataGaLocation":42},"/ai-transparency-center/","ai transparency center",{"text":362,"config":363},"Newsletter",{"href":364,"dataGaName":365,"dataGaLocation":42},"/company/contact/","newsletter",{"text":367,"config":368},"Press",{"href":369,"dataGaName":370,"dataGaLocation":42},"/press/","press",{"text":372,"config":373,"lists":374},"Contact us",{"dataNavLevelOne":314},[375],{"items":376},[377,380,385],{"text":49,"config":378},{"href":51,"dataGaName":379,"dataGaLocation":42},"talk to sales",{"text":381,"config":382},"Get help",{"href":383,"dataGaName":384,"dataGaLocation":42},"/support/","get help",{"text":386,"config":387},"Customer portal",{"href":388,"dataGaName":389,"dataGaLocation":42},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":391,"login":392,"suggestions":399},"Close",{"text":393,"link":394},"To search repositories and projects, login to",{"text":395,"config":396},"gitlab.com",{"href":56,"dataGaName":397,"dataGaLocation":398},"search login","search",{"text":400,"default":401},"Suggestions",[402,404,408,410,414,418],{"text":71,"config":403},{"href":76,"dataGaName":71,"dataGaLocation":398},{"text":405,"config":406},"Code Suggestions (AI)",{"href":407,"dataGaName":405,"dataGaLocation":398},"/solutions/code-suggestions/",{"text":123,"config":409},{"href":125,"dataGaName":123,"dataGaLocation":398},{"text":411,"config":412},"GitLab on AWS",{"href":413,"dataGaName":411,"dataGaLocation":398},"/partners/technology-partners/aws/",{"text":415,"config":416},"GitLab on Google Cloud",{"href":417,"dataGaName":415,"dataGaLocation":398},"/partners/technology-partners/google-cloud-platform/",{"text":419,"config":420},"Why GitLab?",{"href":84,"dataGaName":419,"dataGaLocation":398},{"freeTrial":422,"mobileIcon":427,"desktopIcon":432,"secondaryButton":435},{"text":423,"config":424},"Start free trial",{"href":425,"dataGaName":47,"dataGaLocation":426},"https://gitlab.com/-/trials/new/","nav",{"altText":428,"config":429},"Gitlab Icon",{"src":430,"dataGaName":431,"dataGaLocation":426},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":428,"config":433},{"src":434,"dataGaName":431,"dataGaLocation":426},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":436,"config":437},"Get Started",{"href":438,"dataGaName":439,"dataGaLocation":426},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":441,"mobileIcon":445,"desktopIcon":447},{"text":442,"config":443},"Learn more about GitLab Duo",{"href":76,"dataGaName":444,"dataGaLocation":426},"gitlab duo",{"altText":428,"config":446},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":448},{"src":434,"dataGaName":431,"dataGaLocation":426},{"freeTrial":450,"mobileIcon":455,"desktopIcon":457},{"text":451,"config":452},"Back to pricing",{"href":204,"dataGaName":453,"dataGaLocation":426,"icon":454},"back to pricing","GoBack",{"altText":428,"config":456},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":458},{"src":434,"dataGaName":431,"dataGaLocation":426},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":464,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"title":465,"button":466,"image":471,"config":475,"_id":477,"_type":28,"_source":30,"_file":478,"_stem":479,"_extension":33},"/shared/en-us/banner","is now in public beta!",{"text":467,"config":468},"Try the Beta",{"href":469,"dataGaName":470,"dataGaLocation":42},"/gitlab-duo/agent-platform/","duo banner",{"altText":472,"config":473},"GitLab Duo Agent Platform",{"src":474},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":476},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":481,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"data":482,"_id":686,"_type":28,"title":687,"_source":30,"_file":688,"_stem":689,"_extension":33},"/shared/en-us/main-footer",{"text":483,"source":484,"edit":490,"contribute":495,"config":500,"items":505,"minimal":678},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":485,"config":486},"View page source",{"href":487,"dataGaName":488,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":491,"config":492},"Edit this page",{"href":493,"dataGaName":494,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":496,"config":497},"Please contribute",{"href":498,"dataGaName":499,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":501,"facebook":502,"youtube":503,"linkedin":504},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[506,529,585,614,648],{"title":60,"links":507,"subMenu":512},[508],{"text":509,"config":510},"DevSecOps platform",{"href":69,"dataGaName":511,"dataGaLocation":489},"devsecops platform",[513],{"title":202,"links":514},[515,519,524],{"text":516,"config":517},"View plans",{"href":204,"dataGaName":518,"dataGaLocation":489},"view plans",{"text":520,"config":521},"Why Premium?",{"href":522,"dataGaName":523,"dataGaLocation":489},"/pricing/premium/","why premium",{"text":525,"config":526},"Why Ultimate?",{"href":527,"dataGaName":528,"dataGaLocation":489},"/pricing/ultimate/","why ultimate",{"title":530,"links":531},"Solutions",[532,537,539,541,546,551,555,558,562,567,569,572,575,580],{"text":533,"config":534},"Digital transformation",{"href":535,"dataGaName":536,"dataGaLocation":489},"/topics/digital-transformation/","digital transformation",{"text":148,"config":538},{"href":150,"dataGaName":148,"dataGaLocation":489},{"text":137,"config":540},{"href":119,"dataGaName":120,"dataGaLocation":489},{"text":542,"config":543},"Agile development",{"href":544,"dataGaName":545,"dataGaLocation":489},"/solutions/agile-delivery/","agile delivery",{"text":547,"config":548},"Cloud transformation",{"href":549,"dataGaName":550,"dataGaLocation":489},"/topics/cloud-native/","cloud transformation",{"text":552,"config":553},"SCM",{"href":133,"dataGaName":554,"dataGaLocation":489},"source code management",{"text":123,"config":556},{"href":125,"dataGaName":557,"dataGaLocation":489},"continuous integration & delivery",{"text":559,"config":560},"Value stream management",{"href":177,"dataGaName":561,"dataGaLocation":489},"value stream management",{"text":563,"config":564},"GitOps",{"href":565,"dataGaName":566,"dataGaLocation":489},"/solutions/gitops/","gitops",{"text":187,"config":568},{"href":189,"dataGaName":190,"dataGaLocation":489},{"text":570,"config":571},"Small business",{"href":194,"dataGaName":195,"dataGaLocation":489},{"text":573,"config":574},"Public sector",{"href":199,"dataGaName":200,"dataGaLocation":489},{"text":576,"config":577},"Education",{"href":578,"dataGaName":579,"dataGaLocation":489},"/solutions/education/","education",{"text":581,"config":582},"Financial services",{"href":583,"dataGaName":584,"dataGaLocation":489},"/solutions/finance/","financial services",{"title":207,"links":586},[587,589,591,593,596,598,600,602,604,606,608,610,612],{"text":219,"config":588},{"href":221,"dataGaName":222,"dataGaLocation":489},{"text":224,"config":590},{"href":226,"dataGaName":227,"dataGaLocation":489},{"text":229,"config":592},{"href":231,"dataGaName":232,"dataGaLocation":489},{"text":234,"config":594},{"href":236,"dataGaName":595,"dataGaLocation":489},"docs",{"text":257,"config":597},{"href":259,"dataGaName":5,"dataGaLocation":489},{"text":252,"config":599},{"href":254,"dataGaName":255,"dataGaLocation":489},{"text":261,"config":601},{"href":263,"dataGaName":264,"dataGaLocation":489},{"text":274,"config":603},{"href":276,"dataGaName":277,"dataGaLocation":489},{"text":266,"config":605},{"href":268,"dataGaName":269,"dataGaLocation":489},{"text":279,"config":607},{"href":281,"dataGaName":282,"dataGaLocation":489},{"text":284,"config":609},{"href":286,"dataGaName":287,"dataGaLocation":489},{"text":289,"config":611},{"href":291,"dataGaName":292,"dataGaLocation":489},{"text":294,"config":613},{"href":296,"dataGaName":297,"dataGaLocation":489},{"title":312,"links":615},[616,618,620,622,624,626,628,632,637,639,641,643],{"text":319,"config":617},{"href":321,"dataGaName":314,"dataGaLocation":489},{"text":324,"config":619},{"href":326,"dataGaName":327,"dataGaLocation":489},{"text":332,"config":621},{"href":334,"dataGaName":335,"dataGaLocation":489},{"text":337,"config":623},{"href":339,"dataGaName":340,"dataGaLocation":489},{"text":342,"config":625},{"href":344,"dataGaName":345,"dataGaLocation":489},{"text":347,"config":627},{"href":349,"dataGaName":350,"dataGaLocation":489},{"text":629,"config":630},"Sustainability",{"href":631,"dataGaName":629,"dataGaLocation":489},"/sustainability/",{"text":633,"config":634},"Diversity, inclusion and belonging (DIB)",{"href":635,"dataGaName":636,"dataGaLocation":489},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":352,"config":638},{"href":354,"dataGaName":355,"dataGaLocation":489},{"text":362,"config":640},{"href":364,"dataGaName":365,"dataGaLocation":489},{"text":367,"config":642},{"href":369,"dataGaName":370,"dataGaLocation":489},{"text":644,"config":645},"Modern Slavery Transparency Statement",{"href":646,"dataGaName":647,"dataGaLocation":489},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":649,"links":650},"Contact Us",[651,654,656,658,663,668,673],{"text":652,"config":653},"Contact an expert",{"href":51,"dataGaName":52,"dataGaLocation":489},{"text":381,"config":655},{"href":383,"dataGaName":384,"dataGaLocation":489},{"text":386,"config":657},{"href":388,"dataGaName":389,"dataGaLocation":489},{"text":659,"config":660},"Status",{"href":661,"dataGaName":662,"dataGaLocation":489},"https://status.gitlab.com/","status",{"text":664,"config":665},"Terms of use",{"href":666,"dataGaName":667,"dataGaLocation":489},"/terms/","terms of use",{"text":669,"config":670},"Privacy statement",{"href":671,"dataGaName":672,"dataGaLocation":489},"/privacy/","privacy statement",{"text":674,"config":675},"Cookie preferences",{"dataGaName":676,"dataGaLocation":489,"id":677,"isOneTrustButton":105},"cookie preferences","ot-sdk-btn",{"items":679},[680,682,684],{"text":664,"config":681},{"href":666,"dataGaName":667,"dataGaLocation":489},{"text":669,"config":683},{"href":671,"dataGaName":672,"dataGaLocation":489},{"text":674,"config":685},{"dataGaName":676,"dataGaLocation":489,"id":677,"isOneTrustButton":105},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[691],{"_path":692,"_dir":693,"_draft":6,"_partial":6,"_locale":7,"content":694,"config":697,"_id":699,"_type":28,"title":18,"_source":30,"_file":700,"_stem":701,"_extension":33},"/en-us/blog/authors/steve-azzopardi","authors",{"name":18,"config":695},{"headshot":7,"ctfId":696},"steveazz",{"template":698},"BlogAuthor","content:en-us:blog:authors:steve-azzopardi.yml","en-us/blog/authors/steve-azzopardi.yml","en-us/blog/authors/steve-azzopardi",{"_path":703,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"header":704,"eyebrow":705,"blurb":706,"button":707,"secondaryButton":711,"_id":713,"_type":28,"title":714,"_source":30,"_file":715,"_stem":716,"_extension":33},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":44,"config":708},{"href":709,"dataGaName":47,"dataGaLocation":710},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":49,"config":712},{"href":51,"dataGaName":52,"dataGaLocation":710},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1758326265934]