[{"data":1,"prerenderedAt":720},["ShallowReactive",2],{"/en-us/blog/git-fetch-performance-2021-part-2/":3,"navigation-en-us":36,"banner-en-us":465,"footer-en-us":482,"Jacob Vosmaer":692,"next-steps-en-us":705},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/git-fetch-performance-2021-part-2","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"Git fetch performance improvements in 2021, Part 2 ","Looking back at the server-side performance improvements we made in 2021 for Git fetch.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749663383/Blog/Hero%20Images/tanuki-bg-full.png","https://about.gitlab.com/blog/git-fetch-performance-2021-part-2","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"Git fetch performance improvements in 2021, Part 2 \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Jacob Vosmaer\"}],\n        \"datePublished\": \"2022-02-07\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Jacob Vosmaer","2022-02-07","\nIn [Part 1](/blog/git-fetch-performance/) of this two-part series, we looked at how much server-side Git fetch performance, especially for CI, has improved in GitLab in 2021. Now, we will discuss how we achieved this.\n\n## Recap of Part 1\n-   In December 2019, we set up custom CI fetch caching automation for\n   `gitlab-org/gitlab`, which we internally called \"the CI pre-clone\n   script\".\n-   In December 2020, we encountered some production incidents on GitLab.com,\n   which highlighted that the CI pre-clone script had become critical\n   infrastructure but, at the same time, it had not yet matured beyond\n   a custom one-off solution.\n-   Over the course of 2021, we built an alternative caching solution\n   for CI Git fetch traffic called the pack-objects cache. In Part 1,\n   we discussed a benchmark simulating CI fetch traffic which shows\n   that the pack-objects cache combined with other efficiency\n   improvements reduced GitLab server CPU consumption 9x compared to\n   the baseline of December 2020.\n\n## The pack-objects cache\n\nAs discussed in Part 1, what we realized through the\nproduction incidents in December 2020 was that the CI pre-clone script\nfor `gitlab-org/gitlab` had become a critical piece of infrastructure.\nAt the same time, it benefited only one Git repository on GitLab.com,\nand it was not very robust. It would be much better to have an\nintegrated solution that benefits all repositories. We achieved this\ngoal by building the [pack-objects cache](https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache).\n\nThe name \"pack-objects cache\" refers to `git pack-objects`, which is\nthe Git [subcommand](https://git-scm.com/docs/git-pack-objects) that\nimplements the [packfile](https://git-scm.com/book/en/v2/Git-Internals-Packfiles) compression algorithm. As this [Git commit message from Jeff King](https://gitlab.com/gitlab-org/gitlab-git/-/commit/20b20a22f8f7c1420e259c97ef790cb93091f475) explains, `git pack-objects` is a good candidate for a CI fetch cache.\n\n> You may want to insert a caching layer around\n> pack-objects; it is the most CPU- and memory-intensive\n> part of serving a fetch, and its output is a pure\n> function of its input, making it an ideal place to\n> consolidate identical requests.\n\nThe pack-objects cache is GitLab's take on this \"caching layer\". It\ndeduplicates identical Git fetch requests that arrive within a short\ntime window.\n\nAt a high level, when serving a fetch, we buffer the output of `git\npack-objects` into a temporary file. If an identical request comes in,\nwe serve it from the buffer file instead of creating a new `git\npack-objects` process. After 5 minutes, we delete the buffer file. If\nyou want to know more about how exactly the cache is implemented, you\ncan look at the implementation\n([1](https://gitlab.com/gitlab-org/gitaly/-/blob/v14.6.3/internal/gitaly/service/hook/pack_objects.go),\n[2](https://gitlab.com/gitlab-org/gitaly/-/tree/v14.6.3/internal/streamcache)).\n\n![Architecture diagram](https://about.gitlab.com/images/blogimages/git-fetch-2021/pack-objects-cache-architecture.jpg)\n\nBecause the amount of space used by the cache files is bounded roughly\nby the eviction window (5 minutes) multiplied by the maximum network bandwidth\nof the Gitaly server, we don't have to worry about the cache using a\nlot of storage. In fact, on GitLab.com, we store the cache files on the\nsame disks that hold the repository data. We leave a safety margin of\nfree space on these disks at all times anyway, and the cache fits in\nthat safety margin comfortably.\n\nSimilarly, we also don't notice the increase disk input/output\noperations per second (IOPS) used by the cache on GitLab.com. There\nare two reasons for this. First of all, whenever we _read_ data from\nthe cache, it is usually still in the Linux page cache, so it gets\nserved from RAM. The cache barely does any disk read I/O operations.\nSecond, although the cache does do _write_ operations, these fit\ncomfortably within the maximum sustained IOPS rate supported by the\nGoogle Compute Engine persistent disks we use.\n\nThis leads us to a disadvantage of the pack-objects cache, which is\nthat it really does write a lot of data to disk. On GitLab.com, we saw\nthe disk write throughput jump up by an order of magnitude. You can\nsee this in the graph below, which shows disk writes for a single\nGitaly server with a busy, large repository on it: (the GitLab [company\nwebsite](https://gitlab.com/gitlab-com/www-gitlab-com)). You can\nclearly see the number of bytes written to disk per second jump up when we\nturned the cache on.\n\n![increased disk writes with cache enabled](https://about.gitlab.com/images/blogimages/git-fetch-2021/cache-disk-writes.jpg)\n\nThis increase in disk writes is not a problem for our infrastructure because we have the\nspare capacity, but we were not sure we could assume the same for all\nother GitLab installations in the world. Because of this, we decided\nto leave the pack-objects cache off by default.\n\nThis was a difficult decision because we think almost all GitLab\ninstallations would benefit from having this cache enabled. One of the\nreasons we are writing this blog post is to raise awareness that this\nfeature is available, so that self-managed GitLab administrators can\nopt in to using it.\n\nAgain, on the positive side, the cache did not introduce a new\npoint of failure on GitLab.com. If the `gitaly` service is running,\nand if the repository storage disk is available, then the cache is\navailable. There are no external dependencies. And if `gitaly` is not\nrunning, or the repository storage disk is unavailable, then the whole\nGitaly server is unavailable anyway.\n\nAnd finally, cache capacity grows naturally with the number of Gitaly\nservers. Because the cache is completely local to each Gitaly server,\nwe do not have to worry about whether the cache keeps working as we\ngrow GitLab.com.\n\nThe pack-objects cache was introduced in GitLab 13.11. In GitLab 14.5,\nwe made it a lot more efficient by optimizing its transport using Unix\nsockets\n([1](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3758),\n[2](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3759)). If\nyou want to [try out the pack-objects cache](https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache) on\nyour self-managed GitLab instance, we recommend that you upgrade to\nGitLab 14.5 or newer first.\n\n## Improved RPC transport for Git HTTP\n\nAfter we built the pack-objects cache, we were able to generate a much\nhigher volume of Git fetch responses on a single Gitaly server.\nHowever, we then found out that the RPC transport between the HTTP\nfront-end (GitLab Workhorse) and the Gitaly server became a\nbottleneck. We tried disabling the CI pre-clone script of\n`gitlab-org/gitlab` in April 2021 but we quickly had to turn it back\non because the increased volume of Git fetch data transfer was slowing\ndown the rest of Gitaly.\n\nThe fetch traffic was acting as a noisy neighbor to all the other\ntraffic on `gitlab-org/gitlab`. For each GitLab.com Gitaly server, we\nhave a request latency\n[SLI](https://sre.google/sre-book/service-level-objectives/). This is\na metric that observes request latencies for a selection of RPCs that\nwe expect to be fast, and it tracks how many requests for these RPCs\nare \"fast enough\". If the percentage of fast-enough requests drops\nbelow a certain threshold, we know we have a problem.\n\nWhen we disabled the pre-clone script, the network traffic to the\nGitaly server hosting `gitlab-org/gitlab` went up, as expected. What\nwent wrong was that the percentage of fast-enough requests started to\ndrop. This was not because the server had to serve up more data: The\nRPCs that serve the Git fetch data do not count towards the latency\nSLI.\n\nBelow you see two graphs from the day we tried disabling the CI\npre-clone script. First, see how the network traffic off of the Gitaly\nserver increased once we disabled the CI pre-clone script. This is\nbecause instead of pulling most of the data from object storage, and\nonly some of the data from Gitaly, the CI runners now started pulling\nall of the Git data they needed from Gitaly.\n\n![network peaks](https://about.gitlab.com/images/blogimages/git-fetch-2021/no-script-network-annotated.png)\n\nNow consider our Gitaly request latency SLI for this particular\nserver. For historical reasons, we call this \"Apdex\" in our dashboards.\nRecall that this SLI tracks the percentage of fast-enough requests from\na selection of Gitaly RPCs. The ideal number would be 100%. In the\ntime window where the CI pre-clone script was disabled, this graph\nspent more time below 99%, and it even dipped below 96% several times.\n\n![latency drops](https://about.gitlab.com/images/blogimages/git-fetch-2021/no-script-latency-annotated.png)\n\nEven though we could not explain what was going on, the latency SLI dips\nwere clear evidence that disabling the CI pre-clone script slowed down\nunrelated requests to this Gitaly server, to a point which is\nunacceptable. This was a setback for our plan to replace the CI pre-clone script.\n\nBecause we did not want to just give up, we set aside some time to try\nand understand what the bottleneck was, and if it could be\ncircumvented. The bad news is that we did not come up with a\nsatisfactory answer about what the bottleneck is. But the good news is\nthat we were able to circumvent it.\n\nBy building a simplified [prototype alternate RPC\ntransport](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1046),\nwe were able to find out that with the pack-objects cache, the\nhardware we run on and Git itself were able to serve up much more\ntraffic than we were able to get out of GitLab. We [never got to the\nbottom](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1024)\nof what was causing all the overhead but a likely suspect is the fact\nthat gRPC-Go allocates memory for each message it sends, and with Git\nfetch traffic we send a lot of messages. Gitaly was spending a lot of\ntime doing garbage collection.\n\nWe then had to decide how to improve the situation. Because we were\nuncertain if we could fix the apparent bottleneck in gRPC, and because\nwe were certain that we could go faster by not sending the Git fetch data\nthrough gRPC in the first place, we chose to do the latter. We created\nmodified versions of the RPCs that carry the bulk of the Git fetch\ndata. On the surface, the new versions are still gRPC methods. But\nduring a call, each will establish a side channel, and use that for\nthe bulk data transfer.\n\n![side channel diagram](https://about.gitlab.com/images/blogimages/git-fetch-2021/sidechannel.png)\n\nThis way we avoided making major changes to the structure of Gitaly:\nit is still a gRPC server application. Logging, metrics,\nauthentication, and other middleware work as normal on the optimized\nRPCs. But most of the data transfer happens on either Unix sockets (for localhost RPC calls) or [Yamux streams](https://github.com/hashicorp/yamux/) (for the regular RPC calls).\n\nBecause we have 6x more Git HTTP traffic than Git SSH traffic on\nGitLab.com, we decided to initially only optimize the transport for\nGit HTTP traffic. We are still working on [doing the same for Git\nSSH](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/652) because, even though Git HTTP efficiency is more important for\nGitLab.com than that of Git SSH, we know that for some self-managed\nGitLab instances it is the other way around.\n\nThe new server-side RPC transport for Git HTTP was released in GitLab\n14.5. There is no configuration required for this improved transport.\nRegardless of whether you use the pack-objects cache on your GitLab\ninstance, Gitaly, Workhorse, and Praefect all use less CPU to handle\nGit HTTP fetch requests now.\n\nThe payoff for this work came in October 2021 when we disabled the CI\npre-clone script for `gitlab-org/gitlab`, which did not cause any\nnoisy neighbor problems this time. We have had no issues since then\nserving the Git fetch traffic for that project.\n\n## Improvements to Git itself\n\nAside from the pack-objects cache and the new RPC transport between\nWorkhorse and Gitaly, we also saw some improvements because of changes\nin Git itself. We discovered a few inefficiencies which we\nreported to the Git mailing list and helped get fixed.\n\nOur main repository `gitlab-org/gitlab` has hundreds of thousands of [Git\nreferences](https://git-scm.com/book/en/v2/Git-Internals-Git-References). Looking at CPU profiles, we [noticed](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/400) that a lot of Git\nfetch time was spent on the server iterating over these references.\nThese references were not even being sent back to the client; Git was\njust scanning through all of them on the server twice for each CI job.\n\nIn both cases, the problem could be fixed by doing a scan over a\nsubset instead of a scan across all references. These two problems got fixed\n([1](https://gitlab.com/gitlab-org/gitlab-git/-/commit/b3970c702cb0acc0551d88a5f34ad4ad2e2a6d39), [2](https://gitlab.com/gitlab-org/gitlab-git/-/commit/be18153b975844f8792b03e337f1a4c86fe87531)) in Git 2.31.0, released in March 2021.\n\nLater on, we found a different problem, also in the reference-related\nworkload of Git fetch. As part of the fetch protocol, the server sends\na list of references to the client so that the client can update its\nlocal branches etc. It turned out that for each reference, Git was\ndoing 1 or 2 `write` system calls on the server. This led to [a lot of\noverhead](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1257), and this was made worse by our old RPC transport which could\nend up sending 1 RPC message per advertised Git reference.\n\nThis problem got fixed in Git itself by changing the functions that\nwrite the references to [use buffered\nIO](https://gitlab.com/gitlab-org/gitlab-git/-/commit/70afef5cdf29b5159f18df1b93722055f78740f8).\nThis change landed in Git 2.34.0, released in November 2021. Ahead of\nthat, it got shipped in GitLab 14.4 as a custom Git patch.\n\nFinally, we discovered that increasing the copy buffer size used by\n`git upload-pack` to relay `git pack-objects` output made both `git\nupload-pack` and [every link in the chain after\nit](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4224) more\nefficient. This got fixed in Git by [increasing the buffer\nsize](https://gitlab.com/gitlab-org/gitlab-git/-/commit/55a9651d26a6b88c68445e7d6c9f511d1207cbd8).\nThis change is part of Git 2.35.0 and is included in GitLab 14.7, both\nof which were released in January 2022.\n\n## Summary\n\nIn Part 1, we showed that GitLab server performance when service CI Git fetch traffic has improved a lot in 2021. In this post, we explained that the improvements are due to:\n\n- The pack-objects cache\n- A more efficient Git data transport between server-side GitLab components\n- Efficiency improvements in Git itself\n\n## Thanks\n\nMany people have contributed to the work described in this blog post.\nI would like to specifically thank Quang-Minh Nguyen and Sean McGivern\nfrom the Scalability team, and Patrick Steinhardt and Sami Hiltunen\nfrom the Gitaly team.\n\n## Related content\n\n- Improvements to the client-side performance of `git fetch` (although GitLab is a server application, it sometimes acts as a Git client): [mirror fetches](https://gitlab.com/gitlab-org/git/-/issues/95), [fetches into repositories with many references](https://gitlab.com/gitlab-org/git/-/issues/94)\n- Improvements to server-side Git push performance: [consistency check improvements](https://gitlab.com/gitlab-org/git/-/issues/92)\n","engineering",[23,24,25],"git","production","performance",{"slug":27,"featured":6,"template":28},"git-fetch-performance-2021-part-2","BlogPost","content:en-us:blog:git-fetch-performance-2021-part-2.yml","yaml","Git Fetch Performance 2021 Part 2","content","en-us/blog/git-fetch-performance-2021-part-2.yml","en-us/blog/git-fetch-performance-2021-part-2","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":461,"_type":30,"title":462,"_source":32,"_file":463,"_stem":464,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":392,"minimal":423,"duo":442,"pricingDeployment":451},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,203,208,313,373],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":185},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,164],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,154,159],{"text":150,"config":151},"Application Security Testing",{"href":152,"dataGaName":153,"dataGaLocation":44},"/solutions/application-security-testing/","Application security testing",{"text":155,"config":156},"Software Supply Chain Security",{"href":157,"dataGaLocation":44,"dataGaName":158},"/solutions/supply-chain/","Software supply chain security",{"text":160,"config":161},"Software Compliance",{"href":162,"dataGaName":163,"dataGaLocation":44},"/solutions/software-compliance/","software compliance",{"title":165,"link":166,"items":171},"Measurement",{"config":167},{"icon":168,"href":169,"dataGaName":170,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[172,176,180],{"text":173,"config":174},"Visibility & Measurement",{"href":169,"dataGaLocation":44,"dataGaName":175},"Visibility and Measurement",{"text":177,"config":178},"Value Stream Management",{"href":179,"dataGaLocation":44,"dataGaName":177},"/solutions/value-stream-management/",{"text":181,"config":182},"Analytics & Insights",{"href":183,"dataGaLocation":44,"dataGaName":184},"/solutions/analytics-and-insights/","Analytics and insights",{"title":186,"items":187},"GitLab for",[188,193,198],{"text":189,"config":190},"Enterprise",{"href":191,"dataGaLocation":44,"dataGaName":192},"/enterprise/","enterprise",{"text":194,"config":195},"Small Business",{"href":196,"dataGaLocation":44,"dataGaName":197},"/small-business/","small business",{"text":199,"config":200},"Public Sector",{"href":201,"dataGaLocation":44,"dataGaName":202},"/solutions/public-sector/","public sector",{"text":204,"config":205},"Pricing",{"href":206,"dataGaName":207,"dataGaLocation":44,"dataNavLevelOne":207},"/pricing/","pricing",{"text":209,"config":210,"link":212,"lists":216,"feature":300},"Resources",{"dataNavLevelOne":211},"resources",{"text":213,"config":214},"View all resources",{"href":215,"dataGaName":211,"dataGaLocation":44},"/resources/",[217,250,272],{"title":218,"items":219},"Getting started",[220,225,230,235,240,245],{"text":221,"config":222},"Install",{"href":223,"dataGaName":224,"dataGaLocation":44},"/install/","install",{"text":226,"config":227},"Quick start guides",{"href":228,"dataGaName":229,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":231,"config":232},"Learn",{"href":233,"dataGaLocation":44,"dataGaName":234},"https://university.gitlab.com/","learn",{"text":236,"config":237},"Product documentation",{"href":238,"dataGaName":239,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":241,"config":242},"Best practice videos",{"href":243,"dataGaName":244,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":246,"config":247},"Integrations",{"href":248,"dataGaName":249,"dataGaLocation":44},"/integrations/","integrations",{"title":251,"items":252},"Discover",[253,258,262,267],{"text":254,"config":255},"Customer success stories",{"href":256,"dataGaName":257,"dataGaLocation":44},"/customers/","customer success stories",{"text":259,"config":260},"Blog",{"href":261,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":263,"config":264},"Remote",{"href":265,"dataGaName":266,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":268,"config":269},"TeamOps",{"href":270,"dataGaName":271,"dataGaLocation":44},"/teamops/","teamops",{"title":273,"items":274},"Connect",[275,280,285,290,295],{"text":276,"config":277},"GitLab Services",{"href":278,"dataGaName":279,"dataGaLocation":44},"/services/","services",{"text":281,"config":282},"Community",{"href":283,"dataGaName":284,"dataGaLocation":44},"/community/","community",{"text":286,"config":287},"Forum",{"href":288,"dataGaName":289,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":291,"config":292},"Events",{"href":293,"dataGaName":294,"dataGaLocation":44},"/events/","events",{"text":296,"config":297},"Partners",{"href":298,"dataGaName":299,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":301,"textColor":302,"text":303,"image":304,"link":308},"#2f2a6b","#fff","Insights for the future of software development",{"altText":305,"config":306},"the source promo card",{"src":307},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":309,"config":310},"Read the latest",{"href":311,"dataGaName":312,"dataGaLocation":44},"/the-source/","the source",{"text":314,"config":315,"lists":317},"Company",{"dataNavLevelOne":316},"company",[318],{"items":319},[320,325,331,333,338,343,348,353,358,363,368],{"text":321,"config":322},"About",{"href":323,"dataGaName":324,"dataGaLocation":44},"/company/","about",{"text":326,"config":327,"footerGa":330},"Jobs",{"href":328,"dataGaName":329,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":329},{"text":291,"config":332},{"href":293,"dataGaName":294,"dataGaLocation":44},{"text":334,"config":335},"Leadership",{"href":336,"dataGaName":337,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":339,"config":340},"Team",{"href":341,"dataGaName":342,"dataGaLocation":44},"/company/team/","team",{"text":344,"config":345},"Handbook",{"href":346,"dataGaName":347,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":349,"config":350},"Investor relations",{"href":351,"dataGaName":352,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":354,"config":355},"Trust Center",{"href":356,"dataGaName":357,"dataGaLocation":44},"/security/","trust center",{"text":359,"config":360},"AI Transparency Center",{"href":361,"dataGaName":362,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":364,"config":365},"Newsletter",{"href":366,"dataGaName":367,"dataGaLocation":44},"/company/contact/","newsletter",{"text":369,"config":370},"Press",{"href":371,"dataGaName":372,"dataGaLocation":44},"/press/","press",{"text":374,"config":375,"lists":376},"Contact us",{"dataNavLevelOne":316},[377],{"items":378},[379,382,387],{"text":51,"config":380},{"href":53,"dataGaName":381,"dataGaLocation":44},"talk to sales",{"text":383,"config":384},"Get help",{"href":385,"dataGaName":386,"dataGaLocation":44},"/support/","get help",{"text":388,"config":389},"Customer portal",{"href":390,"dataGaName":391,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":393,"login":394,"suggestions":401},"Close",{"text":395,"link":396},"To search repositories and projects, login to",{"text":397,"config":398},"gitlab.com",{"href":58,"dataGaName":399,"dataGaLocation":400},"search login","search",{"text":402,"default":403},"Suggestions",[404,406,410,412,416,420],{"text":73,"config":405},{"href":78,"dataGaName":73,"dataGaLocation":400},{"text":407,"config":408},"Code Suggestions (AI)",{"href":409,"dataGaName":407,"dataGaLocation":400},"/solutions/code-suggestions/",{"text":125,"config":411},{"href":127,"dataGaName":125,"dataGaLocation":400},{"text":413,"config":414},"GitLab on AWS",{"href":415,"dataGaName":413,"dataGaLocation":400},"/partners/technology-partners/aws/",{"text":417,"config":418},"GitLab on Google Cloud",{"href":419,"dataGaName":417,"dataGaLocation":400},"/partners/technology-partners/google-cloud-platform/",{"text":421,"config":422},"Why GitLab?",{"href":86,"dataGaName":421,"dataGaLocation":400},{"freeTrial":424,"mobileIcon":429,"desktopIcon":434,"secondaryButton":437},{"text":425,"config":426},"Start free trial",{"href":427,"dataGaName":49,"dataGaLocation":428},"https://gitlab.com/-/trials/new/","nav",{"altText":430,"config":431},"Gitlab Icon",{"src":432,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":430,"config":435},{"src":436,"dataGaName":433,"dataGaLocation":428},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":438,"config":439},"Get Started",{"href":440,"dataGaName":441,"dataGaLocation":428},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":443,"mobileIcon":447,"desktopIcon":449},{"text":444,"config":445},"Learn more about GitLab Duo",{"href":78,"dataGaName":446,"dataGaLocation":428},"gitlab duo",{"altText":430,"config":448},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":450},{"src":436,"dataGaName":433,"dataGaLocation":428},{"freeTrial":452,"mobileIcon":457,"desktopIcon":459},{"text":453,"config":454},"Back to pricing",{"href":206,"dataGaName":455,"dataGaLocation":428,"icon":456},"back to pricing","GoBack",{"altText":430,"config":458},{"src":432,"dataGaName":433,"dataGaLocation":428},{"altText":430,"config":460},{"src":436,"dataGaName":433,"dataGaLocation":428},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":466,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":467,"button":468,"image":473,"config":477,"_id":479,"_type":30,"_source":32,"_file":480,"_stem":481,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":469,"config":470},"Try the Beta",{"href":471,"dataGaName":472,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"altText":474,"config":475},"GitLab Duo Agent Platform",{"src":476},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":478},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":483,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":484,"_id":688,"_type":30,"title":689,"_source":32,"_file":690,"_stem":691,"_extension":35},"/shared/en-us/main-footer",{"text":485,"source":486,"edit":492,"contribute":497,"config":502,"items":507,"minimal":680},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":487,"config":488},"View page source",{"href":489,"dataGaName":490,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":493,"config":494},"Edit this page",{"href":495,"dataGaName":496,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":498,"config":499},"Please contribute",{"href":500,"dataGaName":501,"dataGaLocation":491},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":503,"facebook":504,"youtube":505,"linkedin":506},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[508,531,587,616,650],{"title":62,"links":509,"subMenu":514},[510],{"text":511,"config":512},"DevSecOps platform",{"href":71,"dataGaName":513,"dataGaLocation":491},"devsecops platform",[515],{"title":204,"links":516},[517,521,526],{"text":518,"config":519},"View plans",{"href":206,"dataGaName":520,"dataGaLocation":491},"view plans",{"text":522,"config":523},"Why Premium?",{"href":524,"dataGaName":525,"dataGaLocation":491},"/pricing/premium/","why premium",{"text":527,"config":528},"Why Ultimate?",{"href":529,"dataGaName":530,"dataGaLocation":491},"/pricing/ultimate/","why ultimate",{"title":532,"links":533},"Solutions",[534,539,541,543,548,553,557,560,564,569,571,574,577,582],{"text":535,"config":536},"Digital transformation",{"href":537,"dataGaName":538,"dataGaLocation":491},"/topics/digital-transformation/","digital transformation",{"text":150,"config":540},{"href":152,"dataGaName":150,"dataGaLocation":491},{"text":139,"config":542},{"href":121,"dataGaName":122,"dataGaLocation":491},{"text":544,"config":545},"Agile development",{"href":546,"dataGaName":547,"dataGaLocation":491},"/solutions/agile-delivery/","agile delivery",{"text":549,"config":550},"Cloud transformation",{"href":551,"dataGaName":552,"dataGaLocation":491},"/topics/cloud-native/","cloud transformation",{"text":554,"config":555},"SCM",{"href":135,"dataGaName":556,"dataGaLocation":491},"source code management",{"text":125,"config":558},{"href":127,"dataGaName":559,"dataGaLocation":491},"continuous integration & delivery",{"text":561,"config":562},"Value stream management",{"href":179,"dataGaName":563,"dataGaLocation":491},"value stream management",{"text":565,"config":566},"GitOps",{"href":567,"dataGaName":568,"dataGaLocation":491},"/solutions/gitops/","gitops",{"text":189,"config":570},{"href":191,"dataGaName":192,"dataGaLocation":491},{"text":572,"config":573},"Small business",{"href":196,"dataGaName":197,"dataGaLocation":491},{"text":575,"config":576},"Public sector",{"href":201,"dataGaName":202,"dataGaLocation":491},{"text":578,"config":579},"Education",{"href":580,"dataGaName":581,"dataGaLocation":491},"/solutions/education/","education",{"text":583,"config":584},"Financial services",{"href":585,"dataGaName":586,"dataGaLocation":491},"/solutions/finance/","financial services",{"title":209,"links":588},[589,591,593,595,598,600,602,604,606,608,610,612,614],{"text":221,"config":590},{"href":223,"dataGaName":224,"dataGaLocation":491},{"text":226,"config":592},{"href":228,"dataGaName":229,"dataGaLocation":491},{"text":231,"config":594},{"href":233,"dataGaName":234,"dataGaLocation":491},{"text":236,"config":596},{"href":238,"dataGaName":597,"dataGaLocation":491},"docs",{"text":259,"config":599},{"href":261,"dataGaName":5,"dataGaLocation":491},{"text":254,"config":601},{"href":256,"dataGaName":257,"dataGaLocation":491},{"text":263,"config":603},{"href":265,"dataGaName":266,"dataGaLocation":491},{"text":276,"config":605},{"href":278,"dataGaName":279,"dataGaLocation":491},{"text":268,"config":607},{"href":270,"dataGaName":271,"dataGaLocation":491},{"text":281,"config":609},{"href":283,"dataGaName":284,"dataGaLocation":491},{"text":286,"config":611},{"href":288,"dataGaName":289,"dataGaLocation":491},{"text":291,"config":613},{"href":293,"dataGaName":294,"dataGaLocation":491},{"text":296,"config":615},{"href":298,"dataGaName":299,"dataGaLocation":491},{"title":314,"links":617},[618,620,622,624,626,628,630,634,639,641,643,645],{"text":321,"config":619},{"href":323,"dataGaName":316,"dataGaLocation":491},{"text":326,"config":621},{"href":328,"dataGaName":329,"dataGaLocation":491},{"text":334,"config":623},{"href":336,"dataGaName":337,"dataGaLocation":491},{"text":339,"config":625},{"href":341,"dataGaName":342,"dataGaLocation":491},{"text":344,"config":627},{"href":346,"dataGaName":347,"dataGaLocation":491},{"text":349,"config":629},{"href":351,"dataGaName":352,"dataGaLocation":491},{"text":631,"config":632},"Sustainability",{"href":633,"dataGaName":631,"dataGaLocation":491},"/sustainability/",{"text":635,"config":636},"Diversity, inclusion and belonging (DIB)",{"href":637,"dataGaName":638,"dataGaLocation":491},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":354,"config":640},{"href":356,"dataGaName":357,"dataGaLocation":491},{"text":364,"config":642},{"href":366,"dataGaName":367,"dataGaLocation":491},{"text":369,"config":644},{"href":371,"dataGaName":372,"dataGaLocation":491},{"text":646,"config":647},"Modern Slavery Transparency Statement",{"href":648,"dataGaName":649,"dataGaLocation":491},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":651,"links":652},"Contact Us",[653,656,658,660,665,670,675],{"text":654,"config":655},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":491},{"text":383,"config":657},{"href":385,"dataGaName":386,"dataGaLocation":491},{"text":388,"config":659},{"href":390,"dataGaName":391,"dataGaLocation":491},{"text":661,"config":662},"Status",{"href":663,"dataGaName":664,"dataGaLocation":491},"https://status.gitlab.com/","status",{"text":666,"config":667},"Terms of use",{"href":668,"dataGaName":669,"dataGaLocation":491},"/terms/","terms of use",{"text":671,"config":672},"Privacy statement",{"href":673,"dataGaName":674,"dataGaLocation":491},"/privacy/","privacy statement",{"text":676,"config":677},"Cookie preferences",{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":681},[682,684,686],{"text":666,"config":683},{"href":668,"dataGaName":669,"dataGaLocation":491},{"text":671,"config":685},{"href":673,"dataGaName":674,"dataGaLocation":491},{"text":676,"config":687},{"dataGaName":678,"dataGaLocation":491,"id":679,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[693],{"_path":694,"_dir":695,"_draft":6,"_partial":6,"_locale":7,"content":696,"config":700,"_id":702,"_type":30,"title":18,"_source":32,"_file":703,"_stem":704,"_extension":35},"/en-us/blog/authors/jacob-vosmaer","authors",{"name":18,"config":697},{"headshot":698,"ctfId":699},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749659488/Blog/Author%20Headshots/gitlab-logo-extra-whitespace.png","Jacob-Vosmaer",{"template":701},"BlogAuthor","content:en-us:blog:authors:jacob-vosmaer.yml","en-us/blog/authors/jacob-vosmaer.yml","en-us/blog/authors/jacob-vosmaer",{"_path":706,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":707,"eyebrow":708,"blurb":709,"button":710,"secondaryButton":714,"_id":716,"_type":30,"title":717,"_source":32,"_file":718,"_stem":719,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":711},{"href":712,"dataGaName":49,"dataGaLocation":713},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":715},{"href":53,"dataGaName":54,"dataGaLocation":713},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1758326226558]