[{"data":1,"prerenderedAt":720},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":3,"navigation-en-us":36,"banner-en-us":465,"footer-en-us":482,"Igor Wiedler":692,"next-steps-en-us":705},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE's HAProxy Config Change: An Unexpected Journey","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...",[19],"Igor Wiedler","2021-01-14","This blog post was originally published on the GitLab Unfiltered\nblog. It was reviewed and republished on\n2021-02-12.\n\n{: .note .alert-info .text-center}\n\n\n## TL;DR\n\n\n- HAProxy has a `server-state-file` directive that persists some of its\nstate across restarts.\n\n- This state file contains the port of each backend server.\n\n- If an `haproxy.cfg` change modifies the port, the new port will be\noverwritten with the previous one from the state file.\n\n- A workaround is to change the backend server name, so that it is\nconsidered to be a separate server that does not match what is in the state\nfile.\n\n- This has implications for the rollout procedure we use on HAProxy.\n\n\n## Background\n\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\n\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\n\nThe rollout to staging involves changing the request flow from TCP\nproxying...\n\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n\n... to using the [PROXY\nprotocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [\nweb-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n\n`fe-pages-01-lb-gstg` (note the port change):\n\n```diff\n\n-    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n\n## The brokenness\n\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests\nto\n\nthat node start failing.\n\n\nBy retrying a few times via `curl` on the command line, we see this error:\n\n```\n\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n\n*   Trying 35.229.69.78...\n\n* TCP_NODELAY set\n\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n\n* ALPN, offering h2\n\n* ALPN, offering http/1.1\n\n* successfully set certificate verify locations:\n\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n* Closing connection 0\n\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to\njarv.staging.gitlab.io:443\n\n```\n\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\n\nconnection. It turns out that `LibreSSL` does not give us much insight into\nthe\n\nunderlying issue here.\n\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n\n```\n\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n\n```\n\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and\npull\n\nthe dump down for further analysis. That `pcap` file can be opened in\nWireshark, and this allows the data to be\n\nexplored and filtered interactively. Here, the first really surprising thing\nhappens:\n\n\n**We do not see any traffic on port 2443.**\n\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to\nlook at what underlies the LibreSSL error, and what we find\n\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a\nTCP SYN/ACK, establishing the connection. Followed by the client\n\nsending a TLS \"hello\". After which the server closes the connection with a\nFIN.\n\n\nIn other words, the server is closing the connection on the client.\n\n\n## The early hypotheses\n\n\nSo here come the usual suspects:\n\n\n* Did we modify the correct place in the config file?\n\n* Did we catch all places we need to update in the config?\n\n* Did the HAProxy process parse th econfig successfully?\n\n* Did HAProxy actually reload?\n\n* Is there a difference between reload and restart?\n\n* Did we modify the correct config file?\n\n* Are there old lingering HAProxy processes on the box?\n\n* Are we actually sending traffic to this node?\n\n* Are backend health checks failing?\n\n* Is there anything in the HAProxy logs?\n\n\nNone of these gave any insights whatsoever.\n\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with\na\n\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise,\nthis\n\nworked correctly. I tested with different HAProxy versions. It worked\nlocally, but not on\n\n`fe-pages-01`.\n\n\nAt this point I'm stumped. The local config is not identical to gstg, but\nquite\n\nsimilar. What could possibly be the difference?\n\n\n## Digging deeper\n\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help\nwith the investigation.\n\n\nWe started off by repeating the experiment. We saw the same results:\n\n\n* Server closes connection after client sends TLS hello\n\n* No traffic from fe-pages to web-pages on port 2443\n\n* Traffic from fe-pages to web-pages on port 1443\n\n\nThe first lead was to look at the packets going to port 1443. What do they\n\ncontain? We see this:\n\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the\nstream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){:\n.shadow.center}\n\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the\nstream\n\n{: .note.text-center}\n\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the\nclient sent. And before that there is some really weird preamble:\n\n\n```\n\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n\n```\n\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the\nword\n\n\"QUIT.\" Nothing.\n\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\n\nreveals some hits, but none that explain this.\n\n\nSo this is a mystery. We leave it for now, and probe in a different\ndirection.\n\n\n## Honing in\n\n\nHow come we are sending traffic to port 1443, when that port is not\nmentioned in\n\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\n\nI suggested running `strace` on HAProxy startup, so that we can see which\nfiles\n\nare being `open`ed. This is a bit tricky to do though, because the process\nis\n\nsystemd-managed.\n\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc),\nwe\n\ncan actually listen on open events system-wide using the wonderful\n\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py).\nSo we run `opensnoop` and restart `haproxy`, and this is what we see,\nhighlighting the relevant bit:\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo\n/usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n\n...\n\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n\n...\n\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n\n...\n\n24.118333000  16702  haproxy             4   0\n/etc/haproxy/cloudflare_ips_v4.lst\n\n...\n\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n\n```\n\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly\nsuspicious.\n\nWhat could it possibly be? Let's see what this file contains.\n\n```\n\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat\n/etc/haproxy/state/global\n\n\n1\n\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state\nsrv_uweight srv_iweight srv_time_since_last_change srv_check_status\nsrv_check_result srv_check_health srv_check_state srv_agent_state\nbk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0\n0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0\n0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n\n```\n\n\nIt appears we are storing some metadata for each backend server, including\nits old port number!\n\n\nNow, looking again in `haproxy.cfg`, we see:\n\n```\n\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\n\nSo we are using the\n\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\n\ndirective. This will persist server state across HAProxy restarts. That is\n\nuseful to keep metadata consistent, such as whether a server was marked as\n\nMAINT.\n\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\n\nThe suspected behavior is:\n\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`,\n`2443`, `send-proxy-v2`\n\n* HAProxy reload is initiated\n\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the\nold port of each backend server)\n\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new\nconfig: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the\nbackend server `web-pages-01-sv-gstg`, and overrides all values, including\nthe port!\n\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS\nport.\n\n\n## The workaround\n\n\nTo validate the theory and develop a potential workaround, we modify\n\n`haproxy.cfg` to use a different backend server name.\n\n\nThe new diff is:\n\n```diff\n\n-    server web-pages-01-sv-gstg        \nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n-    server web-pages-02-sv-gstg        \nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080\n\n+    server web-pages-01-sv-gstg-proxyv2\nweb-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n+    server web-pages-02-sv-gstg-proxyv2\nweb-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s\nfastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n\n```\n\n\nWith this config change in place, we reload HAProxy and indeed, it is now\n\nserving traffic correctly. See [the merge request fixing\nit](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n\n## A follow-up on those `QUIT` bytes\n\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol?\nRemember, searching [the\n\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\n\nstring did not find any matches. However, Matt actually read the spec, and\nfound this section on version 2 of\n\nthe protocol:\n\n```\n\nThe binary header format starts with a constant 12 bytes block containing\nthe\n\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly\nless mnemonic than the header from text-based version 1 of the protocol:\n\n```\n\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\n\nWell, I suppose that explains it.\n\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[24,25],"production","inside GitLab",{"slug":27,"featured":6,"template":28},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":461,"_type":30,"title":462,"_source":32,"_file":463,"_stem":464,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":392,"minimal":423,"duo":442,"pricingDeployment":451},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,203,208,313,373],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":185},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,164],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,154,159],{"text":150,"config":151},"Application Security Testing",{"href":152,"dataGaName":153,"dataGaLocation":44},"/solutions/application-security-testing/","Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":44,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":44,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":44,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":44,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":44,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":44,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":44,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":44,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":44},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":44},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":44,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":44},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":44},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":44},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":44},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":44},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":44},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":44},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":44},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":44},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":44},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":44},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":44},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":44},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":51,"config":380},{"href":53,"dataGaName":381,"dataGaLocation":44},"talk to sales",{"text":383,"config":384},"Get help",{"href":385,"dataGaName":386,"dataGaLocation":44},"/support/","get help",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":58,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":73,"config":405},{"href":78,"dataGaName":73,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":125,"config":411},{"href":127,"dataGaName":125,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":86,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":49,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":78,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},{"freeTrial":452,"mobileIcon":457,"desktopIcon":459},{"text":453,"config":454},"Back to pricing",{"href":206,"dataGaName":455,"dataGaLocation":428,"icon":456},"back to pricing","GoBack",{"altText":430,"config":458},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":460},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":466,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":467,"button":468,"image":473,"config":477,"_id":479,"_type":30,"_source":32,"_file":480,"_stem":481,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":469,"config":470},"Try the Beta",{"href":471,"dataGaName":472,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":474,"config":475},"GitLab Duo Agent Platform",{"src":476},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":478},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":483,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":484,"_id":688,"_type":30,"title":689,"_source":32,"_file":690,"_stem":691,"_extension":35},"/shared/en-us/main-footer",{"text":485,"source":486,"edit":492,"contribute":497,"config":502,"items":507,"minimal":680},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":487,"config":488},"View page source",{"href":489,"dataGaName":490,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":493,"config":494},"Edit this page",{"href":495,"dataGaName":496,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":498,"config":499},"Please contribute",{"href":500,"dataGaName":501,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":503,"facebook":504,"youtube":505,"linkedin":506},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[508,531,587,616,650],{"title":62,"links":509,"subMenu":514},[510],{"text":511,"config":512},"DevSecOps platform",{"href":71,"dataGaName":513,"dataGaLocation":491},"devsecops platform",[515],{"title":204,"links":516},[517,521,526],{"text":518,"config":519},"View plans",{"href":206,"dataGaName":520,"dataGaLocation":491},"view plans",{"text":522,"config":523},"Why Premium?",{"href":524,"dataGaName":525,"dataGaLocation":491},"/pricing/premium/","why premium",{"text":527,"config":528},"Why Ultimate?",{"href":529,"dataGaName":530,"dataGaLocation":491},"/pricing/ultimate/","why ultimate",{"title":532,"links":533},"Solutions",[534,539,541,543,548,553,557,560,564,569,571,574,577,582],{"text":535,"config":536},"Digital transformation",{"href":537,"dataGaName":538,"dataGaLocation":491},"/topics/digital-transformation/","digital transformation",{"text":150,"config":540},{"href":152,"dataGaName":150,"dataGaLocation":491},{"text":139,"config":542},{"href":121,"dataGaName":122,"dataGaLocation":491},{"text":544,"config":545},"Agile development",{"href":546,"dataGaName":547,"dataGaLocation":491},"/solutions/agile-delivery/","agile delivery",{"text":549,"config":550},"Cloud transformation",{"href":551,"dataGaName":552,"dataGaLocation":491},"/topics/cloud-native/","cloud transformation",{"text":554,"config":555},"SCM",{"href":135,"dataGaName":556,"dataGaLocation":491},"source code management",{"text":125,"config":558},{"href":127,"dataGaName":559,"dataGaLocation":491},"continuous integration & delivery",{"text":561,"config":562},"Value stream management",{"href":179,"dataGaName":563,"dataGaLocation":491},"value stream management",{"text":565,"config":566},"GitOps",{"href":567,"dataGaName":568,"dataGaLocation":491},"/solutions/gitops/","gitops",{"text":189,"config":570},{"href":191,"dataGaName":192,"dataGaLocation":491},{"text":572,"config":573},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":491},{"text":575,"config":576},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":491},{"text":578,"config":579},"Education",{"href":580,"dataGaName":581,"dataGaLocation":491},"/solutions/education/","education",{"text":583,"config":584},"Financial services",{"href":585,"dataGaName":586,"dataGaLocation":491},"/solutions/finance/","financial services",{"title":209,"links":588},[589,591,593,595,598,600,602,604,606,608,610,612,614],{"text":221,"config":590},{"href":223,"dataGaName":224,"dataGaLocation":491},{"text":226,"config":592},{"href":228,"dataGaName":229,"dataGaLocation":491},{"text":231,"config":594},{"href":233,"dataGaName":234,"dataGaLocation":491},{"text":236,"config":596},{"href":238,"dataGaName":597,"dataGaLocation":491},"docs",{"text":259,"config":599},{"href":261,"dataGaName":5,"dataGaLocation":491},{"text":254,"config":601},{"href":256,"dataGaName":257,"dataGaLocation":491},{"text":263,"config":603},{"href":265,"dataGaName":266,"dataGaLocation":491},{"text":276,"config":605},{"href":278,"dataGaName":279,"dataGaLocation":491},{"text":268,"config":607},{"href":270,"dataGaName":271,"dataGaLocation":491},{"text":281,"config":609},{"href":283,"dataGaName":284,"dataGaLocation":491},{"text":286,"config":611},{"href":288,"dataGaName":289,"dataGaLocation":491},{"text":291,"config":613},{"href":293,"dataGaName":294,"dataGaLocation":491},{"text":296,"config":615},{"href":298,"dataGaName":299,"dataGaLocation":491},{"title":314,"links":617},[618,620,622,624,626,628,630,634,639,641,643,645],{"text":321,"config":619},{"href":323,"dataGaName":316,"dataGaLocation":491},{"text":326,"config":621},{"href":328,"dataGaName":329,"dataGaLocation":491},{"text":334,"config":623},{"href":336,"dataGaName":337,"dataGaLocation":491},{"text":339,"config":625},{"href":341,"dataGaName":342,"dataGaLocation":491},{"text":344,"config":627},{"href":346,"dataGaName":347,"dataGaLocation":491},{"text":349,"config":629},{"href":351,"dataGaName":352,"dataGaLocation":491},{"text":631,"config":632},"Sustainability",{"href":633,"dataGaName":631,"dataGaLocation":491},"/sustainability/",{"text":635,"config":636},"Diversity, inclusion and belonging (DIB)",{"href":637,"dataGaName":638,"dataGaLocation":491},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":640},{"href":356,"dataGaName":357,"dataGaLocation":491},{"text":364,"config":642},{"href":366,"dataGaName":367,"dataGaLocation":491},{"text":369,"config":644},{"href":371,"dataGaName":372,"dataGaLocation":491},{"text":646,"config":647},"Modern Slavery Transparency Statement",{"href":648,"dataGaName":649,"dataGaLocation":491},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":651,"links":652},"Contact Us",[653,656,658,660,665,670,675],{"text":654,"config":655},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":491},{"text":383,"config":657},{"href":385,"dataGaName":386,"dataGaLocation":491},{"text":388,"config":659},{"href":390,"dataGaName":391,"dataGaLocation":491},{"text":661,"config":662},"Status",{"href":663,"dataGaName":664,"dataGaLocation":491},"https://status.gitlab.com/","status",{"text":666,"config":667},"Terms of use",{"href":668,"dataGaName":669,"dataGaLocation":491},"/terms/","terms of use",{"text":671,"config":672},"Privacy statement",{"href":673,"dataGaName":674,"dataGaLocation":491},"/privacy/","privacy statement",{"text":676,"config":677},"Cookie preferences",{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":681},[682,684,686],{"text":666,"config":683},{"href":668,"dataGaName":669,"dataGaLocation":491},{"text":671,"config":685},{"href":673,"dataGaName":674,"dataGaLocation":491},{"text":676,"config":687},{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[693],{"_path":694,"_dir":695,"_draft":6,"_partial":6,"_locale":7,"content":696,"config":700,"_id":702,"_type":30,"title":19,"_source":32,"_file":703,"_stem":704,"_extension":35},"/en-us/blog/authors/igor-wiedler","authors",{"name":19,"config":697},{"headshot":698,"ctfId":699},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":701},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":706,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":707,"eyebrow":708,"blurb":709,"button":710,"secondaryButton":714,"_id":716,"_type":30,"title":717,"_source":32,"_file":718,"_stem":719,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":711},{"href":712,"dataGaName":49,"dataGaLocation":713},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":715},{"href":53,"dataGaName":54,"dataGaLocation":713},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1758326265365]