[{"data":1,"prerenderedAt":717},["ShallowReactive",2],{"/en-us/blog/anomaly-detection-using-prometheus/":3,"navigation-en-us":34,"banner-en-us":463,"footer-en-us":480,"Sara Kassabian":690,"next-steps-en-us":702},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":24,"_id":27,"_type":28,"title":29,"_source":30,"_file":31,"_stem":32,"_extension":33},"/en-us/blog/anomaly-detection-using-prometheus","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"How to use Prometheus for anomaly detection in GitLab","Explore how Prometheus query language can be used to help you diagnose incidents, detect performance regressions, tackle abuse, and more.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749667819/Blog/Hero%20Images/anomaly-detection-cover.png","https://about.gitlab.com/blog/anomaly-detection-using-prometheus","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"How to use Prometheus for anomaly detection in GitLab\",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Sara Kassabian\"}],\n        \"datePublished\": \"2019-07-23\",\n      }",{"title":9,"description":10,"authors":17,"heroImage":11,"date":19,"body":20,"category":21,"tags":22},[18],"Sara Kassabian","2019-07-23","One of the more basic functions of the Prometheus query language is\nreal-time aggregation of [time series\ndata](https://prometheus.io/docs/prometheus/latest/querying/basics/).\n[Andrew Newdigate](/company/team/#suprememoocow), a distinguished engineer\non the GitLab infrastructure team, hypothesized that Prometheus query\nlanguage can also be used to detect anomalies in time series data.\n\n\n[Andrew broke down the different ways Prometheus can be\nused](https://vimeo.com/341141334) for the attendees of [Monitorama\n2019](https://monitorama.com/index.html). This blog post explains how\nanomaly detection works with Prometheus and includes the code snippets\nyou’ll need to try it out for yourself on your own system.\n\n\n## Why is anomaly detection useful?\n\n\nThere are four key reasons why anomaly detection is important to GitLab:\n\n\n1. **Diagnosing incidents**: We can figure out which services are performing\noutside their normal bounds quickly and reduce the average time it takes to\n[detect an incident\n(MTTD)](/handbook/engineering/infrastructure/incident-management/), bringing\nabout a faster resolution.\n\n2. **Detecting application performance regressions**: For example, if an n +\n1 regression is introduced and leads to one service calling another at a\nvery high rate, we can quickly track the issue down and resolve it.\n\n3. **Identify and resolve abuse**: GitLab offers free computing ([GitLab\nCI/CD](/topics/ci-cd/)) and hosting (GitLab Pages), and there are a small\nsubset of users who might take advantage.\n\n4. **Security**: Anomaly detection is essential to spotting unusual trends\nin GitLab time series data.\n\n\nFor these reasons and many others, Andrew investigated whether it was\npossible to perform anomaly detection on GitLab time series data by simply\nusing Prometheus queries and rules.\n\n\n## What level of aggregation is the correct one?\n\n\nFirst, time series data must be aggregated correctly. Andrew used a standard\ncounter of `http_requests_total` as the data source for this demonstration,\nalthough many other metrics can be applied using the same techniques.\n\n\n```\n\nhttp_requests_total{\n job=\"apiserver\",\n method=\"GET\",\n controller=\"ProjectsController\",\n status_code=\"200\",\n environment=\"prod\"\n}\n\n```\n\n{: .language-ruby}\n\n\nThis example metric has **some extra dimensions**: `method`, `controller`,\n`status_code`, `environment`, as well as the dimensions that Prometheus\nadds, such as `instance` and `job`.\n\n\nNext, you must choose the correct level of aggregation for the data you are\nusing. This is a bit of a Goldilocks problem – too much, too little, or just\nright – but it is essential for finding anomalies. By **aggregating the data\ntoo much**, it can be reduced to too few dimensions, creating two potential\nproblems:\n\n\n1. You can miss genuine anomalies because the aggregation hides problems\nthat are occurring within subsets of your data.\n\n2. If you do detect an anomaly, it's difficult to attribute it to a\nparticular part of your system without more investigation into the anomaly.\n\n\nBut by **aggregating the data too little**, you might end up with a series\nof data with very small sample sizes which can lead to false positives and\ncould mean flagging genuine data as outliers.\n\n\nJust right: Our experience has shown the **right level of aggregation is the\nservice level**, so we include the job label and the environment label, but\ndrop other dimensions. The data aggregation used through the rest of the\ntalk includes: job `http requests`, rate five minutes, which is basically a\nrate across job and environment on a five-minute window.\n\n\n```\n\n- record: job:http_requests:rate5m\n\nexpr: sum without(instance, method, controller, status_code)\n\n(rate(http_requests_total[5m]))\n\n# --> job:http_requests:rate5m{job=\"apiserver\", environment=\"prod\"}  21321\n\n# --> job:http_requests:rate5m{job=\"gitserver\", environment=\"prod\"}  2212\n\n# --> job:http_requests:rate5m{job=\"webserver\", environment=\"prod\"}  53091\n\n```\n\n{: .language-ruby}\n\n\n## Using z-score for anomaly detection\n\n\nSome of the primary principles of statistics can be applied to detecting\nanomalies with Prometheus.\n\n\nIf we know the average value and [standard deviation\n(σ)](https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/standard-deviation/)\nof a Prometheus series, we can use any sample in the series to calculate the\nz-score. The z-score is measured in the number of standard deviations from\nthe mean. So a z-score of 0 would mean the z-score is identical to the mean\nin a data set with a normal distribution, while a z-score of 1 is 1.0 σ from\nthe mean, etc.\n\n\nAssuming the underlying data has a normal distribution, 99.7% of the samples\nshould have a z-score between zero to three. The further the z-score is from\nzero, the less likely it is to exist. We apply this property to detecting\nanomalies in the Prometheus series.\n\n\n1. Calculate the average and standard deviation for the metric using data\nwith a large sample size. For this example, we use one week’s worth of data.\nIf we assume we're evaluating the recording rule once a minute, over a\none-week period we'll have just over 10,000 samples.\n\n\n```\n\n# Long-term average value for the series\n\n- record: job:http_requests:rate5m:avg_over_time_1w\n\nexpr: avg_over_time(job:http_requests:rate5m[1w])\n\n\n# Long-term standard deviation for the series\n\n- record: job:http_requests:rate5m:stddev_over_time_1w\n\nexpr: stddev_over_time(job:http_requests:rate5m[1w])\n\n```\n\n{: .language-ruby}\n\n\n2.  We can calculate the z-score for the Prometheus query once we have the\naverage and standard deviation for the aggregation.\n\n\n```\n\n# Z-Score for aggregation\n\n(\n\njob:http_requests:rate5m -\n\njob:http_requests:rate5m:avg_over_time_1w\n\n) /  job:http_requests:rate5m:stddev_over_time_1w\n\n```\n\n{: .language-ruby}\n\n\nBased on the statistical principles of normal distributions, **we can assume\nthat any value that falls outside of the range of roughly +3 to -3 is an\nanomaly**. We can build an alert around this principle. For example, we can\nget an alert when our aggregation is out of this range for more than five\nminutes.\n\n\n![Graph showing RPS on GitLab.com over 48\nhours](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image1.png){:\n.shadow.medium.center}\n\n\nGitLab.com Pages service RPS over 48 hours, with ±3 z-score region in green\n\n{: .note.text-center}\n\n\nZ-scores are a bit awkward to interpret on a graph because they don’t have a\nunit of measurement. But anomalies on this chart are easy to detect.\nAnything that appears outside of the green area (which denotes z-scores that\nfall within a range of +3 or -3) is an anomaly.\n\n\n### What if you don’t have a normal distribution?\n\n\n**But wait**: We make a big leap by assuming that our underlying aggregation\nhas a normal distribution. If we calculate the z-score with data that isn’t\nnormally distributed, our results will be incorrect.\n\n\nThere are numerous statistical techniques for testing your data for a normal\ndistribution, but the best option is to test that your underlying data has a\nz-score of about **+4 to -4**.\n\n\n```\n\n(\n max_over_time(job:http_requests:rate5m[1w]) - avg_over_time(job:http_requests:rate5m[1w])\n) / stddev_over_time(job:http_requests:rate5m[1w])\n\n# --> {job=\"apiserver\", environment=\"prod\"}  4.01\n\n# --> {job=\"gitserver\", environment=\"prod\"}  3.96\n\n# --> {job=\"webserver\", environment=\"prod\"}  2.96\n\n\n(\n min_over_time(job:http_requests:rate5m[1w]) - avg_over_time(job:http_requests:rate5m[1w])\n) / stddev_over_time(job:http_requests:rate5m[1w])\n\n# --> {job=\"apiserver\", environment=\"prod\"}  -3.8\n\n# --> {job=\"gitserver\", environment=\"prod\"}  -4.1\n\n# --> {job=\"webserver\", environment=\"prod\"}  -3.2\n\n```\n\n{: .language-ruby}\n\n\nTwo Prometheus queries testing the minimum and maximum z-scores.\n\n{: .note.text-center}\n\n\nIf your results return with a range of +20 to -20, the tail is too long and\nyour results will be skewed. Remember too that this needs to be run on an\naggregated, not unaggregated series. Metrics that probably don’t have normal\ndistributions include things like error rates, latencies, queue lengths\netc., but many of these metrics will tend to work better with fixed\nthresholds for alerting anyway.\n\n\n## Detecting anomalies using seasonality\n\n\nWhile calculating z-scores works well with normal distributions of time\nseries data, there is a second method that can yield _even more accurate_\nanomaly detection results. **Seasonality** is a characteristic of a time\nseries metric in which the metric experiences regular and predictable\nchanges that recur every cycle.\n\n\n![Graph showing Gitaly RPS, Mon-Sun over four\nweeks](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image2.png){:\n.shadow.medium.center}\n\n\nGitaly requests per second (RPS), Monday-Sunday, over four consecutive weeks\n\n{: .note.text-center}\n\n\nThis graph illustrates the RPS (requests per second) rates for Gitaly over\nseven days, Monday through Sunday, over four consecutive weeks. The\nseven-day range is referred to as the “offset,” meaning the pattern that\nwill be measured.\n\n\nEach week on the graph is in a different color. The seasonality in the data\nis indicated by the consistency in trends indicated on the graph – every\nMonday morning, we see the same rise in RPS rates, and on Friday evenings,\nwe see the RPS rates drop off, week after week.\n\n\nBy leveraging the seasonality in our time series data we can create more\naccurate predictions which will lead to better anomaly detection.\n\n\n### How do we leverage seasonality?\n\n\nCalculating seasonality with Prometheus required that we iterate on a few\ndifferent statistical principles.\n\n\nIn the first iteration, we calculate by adding the growth trend we’ve seen\nover a one-week period to the value from the previous week. Calculate the\ngrowth trend by subtracting the rolling one-week average for last week from\nthe rolling one-week average for now.\n\n\n```\n\n- record: job:http_requests:rate5m_prediction\n  expr: >\n    job:http_requests:rate5m offset 1w                     # Value from last period\n    + job:http_requests:rate5m:avg_over_time_1w            # One-week growth trend\n    - job:http_requests:rate5m:avg_over_time_1w offset 1w\n```\n\n\nThe first iteration is a bit narrow; we’re using a five-minute window from\nthis week and the previous week to derive our predictions.\n\n\nIn the second iteration, we expand our scope by taking the average of a\nfour-hour period for the previous week and comparing it to the current week.\nSo, if we’re trying to predict the value of a metric at 8am on a Monday\nmorning, instead of using the same five-minute window from one week prior,\nwe use the average value for the metric from 6am until 10am for the previous\nmorning.\n\n\n```\n\n- record: job:http_requests:rate5m_prediction\n  expr: >\n    avg_over_time(job:http_requests:rate5m[4h] offset 166h) # Rounded value from last period\n    + job:http_requests:rate5m:avg_over_time_1w             # Add 1w growth trend\n    - job:http_requests:rate5m:avg_over_time_1w offset 1w\n```\n\n{: .language-yaml}\n\n\nWe use the 166 hours in the query instead of one week because we want to use\na four-hour period based on the current time of day, so we need the offset\nto be two hours short of a full week.\n\n\n![Comparing the real Gitaly RPS with our\nprediction](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image3.png){:\n.shadow.medium.center}\n\n\nGitaly service RPS (yellow) vs prediction (blue), over two weeks.\n\n{: .note.text-center}\n\n\nA comparison of the actual Gitaly RPS (yellow) with our prediction (blue)\nindicate that our calculations were fairly accurate. However, this method\nhas a flaw.\n\n\nGitLab usage was lower than the typical Wednesday because May 1 was\nInternational Labor Day, a holiday celebrated in many different countries.\nBecause our growth rate is informed by the previous week’s usage, our\npredictions for the next week, on Wednesday, May 8, were for a lower RPS\nthan it would have been had it not been a holiday on Wednesday, May 1.\n\n\nThis can be fixed by making three predictions for three consecutive weeks\nbefore Wednesday, May 1; for the previous Wednesday, the Wednesday before\nthat, and the Wednesday before that. The query stays the same, but the\noffset is adjusted.\n\n\n```\n\n- record: job:http_requests:rate5m_prediction\n  expr: >\n   quantile(0.5,\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 166h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 1w\n       , \"offset\", \"1w\", \"\", \"\")\n     or\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 334h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 2w\n       , \"offset\", \"2w\", \"\", \"\")\n     or\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 502h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 3w\n       , \"offset\", \"3w\", \"\", \"\")\n   )\n   without (offset)\n```\n\n{: .language-yaml}\n\n\n![A graph showing three predictions for three Wednesdays vs. actual Gitaly\nRPS](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image4.png){:\n.shadow.medium.center}\n\n\nThree predictions for three Wednesdays vs actual Gitaly RPS, Wednesday, May\n8 (one week following International Labor Day)\n\n{: .note.text-center}\n\n\nOn the graph we’ve plotted Wednesday, May 8 and three predictions for the\nthree consecutive weeks before May 8. We can see that two of the predictions\nare good, but the May 1 prediction is still far off base.\n\n\nAlso, we don’t want three predictions, we want **one prediction**. Taking\nthe average is not an option, because it will be diluted by our skewed May 1\nRPS data. Instead, we want to calculate the median. Prometheus does not have\na median query, but we can use a quantile aggregation in lieu of the median.\n\n\nThe one problem with this approach is that we're trying to include three\nseries in an aggregation, and those three series are actually all the same\nseries over three weeks. In other words, they all have the same labels, so\nconnecting them is tricky. To avoid confusion, we create a label called\n`offset` and use the label-replace function to add an offset to each of the\nthree weeks. Next, in the quantile aggregation, we strip that off, and that\ngives us the middle value out of the three.\n\n\n```\n\n- record: job:http_requests:rate5m_prediction\n  expr: >\n   quantile(0.5,\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 166h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 1w\n       , \"offset\", \"1w\", \"\", \"\")\n     or\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 334h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 2w\n       , \"offset\", \"2w\", \"\", \"\")\n     or\n     label_replace(\n       avg_over_time(job:http_requests:rate5m[4h] offset 502h)\n       + job:http_requests:rate5m:avg_over_time_1w - job:http_requests:rate5m:avg_over_time_1w offset 3w\n       , \"offset\", \"3w\", \"\", \"\")\n   )\n   without (offset)\n```\n\n{: .language-yaml}\n\n\nNow, our prediction deriving the median value from the series of three\naggregations is much more accurate.\n\n\n![Graph showing median predications vs. actual Gitaly RPS on Weds May\n8](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image5.png){:\n.shadow.medium.center}\n\n\nMedian predictions vs actual Gitaly RPS, Wednesday, May 8 (one week\nfollowing International Labor Day)\n\n{: .note.text-center}\n\n\n### How do we know our prediction is truly accurate?\n\n\nTo test the accuracy of our prediction, we can return to the z-score. We can\nuse the z-score to measure the sample's distance from its prediction in\nstandard deviations. The more standard deviations away from our prediction\nwe are, the greater the likelihood is that a particular value is an outlier.\n\n\n![Predicted normal range\n+1.5σ/-1.5σ](https://about.gitlab.com/images/blogimages/prometheus_anomaly/image6.png){:\n.shadow.medium.center}\n\n\nPredicted normal range ± 1.5σ for Gitaly Service\n\n{: .note.text-center}\n\n\nWe can update our Grafana chart to use the seasonal prediction rather than\nthe weekly rolling average value. The range of normality for a certain time\nof day is shaded in green. Anything that falls outside of the shaded green\narea is considered an outlier. In this case, the outlier was on Sunday\nafternoon when our cloud provider encountered some network issues.\n\n\nUsing boundaries of ±2σ on either side of our prediction is a pretty good\nmeasurement for determining an outlier with seasonal predictions.\n\n\n## How to set up alerting using Prometheus\n\n\nIf you want to set up alerts for anomaly events, you can apply a pretty\nstraightforward rule to Prometheus that checks if the z-score of the metric\nis between a standard deviation of **+2 or -2**.\n\n\n```\n\n- alert: RequestRateOutsideNormalRange\n  expr: >\n   abs(\n     (\n       job:http_requests:rate5m - job:http_requests:rate5m_prediction\n     ) / job:http_requests:rate5m:stddev_over_time_1w\n   ) > 2\n  for: 10m\n  labels:\n    severity: warning\n  annotations:\n    summary: Requests for job {{ $labels.job }} are outside of expected operating parameters\n```\n\n{: .language-yaml}\n\n\nAt GitLab, we use a custom routing rule that pings Slack when any anomalies\nare detected, but doesn’t page our on-call support staff.\n\n\n## The takeaway\n\n\n1. Prometheus can be used for some types of anomaly detection\n\n2. The right level of data aggregation is the key to anomaly detection\n\n3. Z-scoring is an effective method, if your data has a normal distribution\n\n4. Seasonal metrics can provide great results for anomaly detection\n\n\nWatch Andrew’s full presentation from [Monitorama\n2019](https://monitorama.com/index.html). If you have questions for Andrew,\nreach him on Slack at #talk-andrew-newdigate. You can also read more about\n[why you need Prometheus](/blog/why-all-organizations-need-prometheus/).\n\n\n\u003C!-- blank line -->\n\n\u003Cfigure class=\"video_container\">\n  \u003Ciframe src=\"https://player.vimeo.com/video/341141334?portrait=0\" frameborder=\"0\" allowfullscreen=\"true\"> \u003C/iframe>\n\u003C/figure>\n","engineering",[23],"inside GitLab",{"slug":25,"featured":6,"template":26},"anomaly-detection-using-prometheus","BlogPost","content:en-us:blog:anomaly-detection-using-prometheus.yml","yaml","Anomaly Detection Using Prometheus","content","en-us/blog/anomaly-detection-using-prometheus.yml","en-us/blog/anomaly-detection-using-prometheus","yml",{"_path":35,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"data":37,"_id":459,"_type":28,"title":460,"_source":30,"_file":461,"_stem":462,"_extension":33},"/shared/en-us/main-navigation","en-us",{"logo":38,"freeTrial":43,"sales":48,"login":53,"items":58,"search":390,"minimal":421,"duo":440,"pricingDeployment":449},{"config":39},{"href":40,"dataGaName":41,"dataGaLocation":42},"/","gitlab logo","header",{"text":44,"config":45},"Get free trial",{"href":46,"dataGaName":47,"dataGaLocation":42},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":49,"config":50},"Talk to sales",{"href":51,"dataGaName":52,"dataGaLocation":42},"/sales/","sales",{"text":54,"config":55},"Sign in",{"href":56,"dataGaName":57,"dataGaLocation":42},"https://gitlab.com/users/sign_in/","sign in",[59,103,201,206,311,371],{"text":60,"config":61,"cards":63,"footer":86},"Platform",{"dataNavLevelOne":62},"platform",[64,70,78],{"title":60,"description":65,"link":66},"The most comprehensive AI-powered DevSecOps Platform",{"text":67,"config":68},"Explore our Platform",{"href":69,"dataGaName":62,"dataGaLocation":42},"/platform/",{"title":71,"description":72,"link":73},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":74,"config":75},"Meet GitLab Duo",{"href":76,"dataGaName":77,"dataGaLocation":42},"/gitlab-duo/","gitlab duo ai",{"title":79,"description":80,"link":81},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":82,"config":83},"Learn more",{"href":84,"dataGaName":85,"dataGaLocation":42},"/why-gitlab/","why gitlab",{"title":87,"items":88},"Get started with",[89,94,99],{"text":90,"config":91},"Platform Engineering",{"href":92,"dataGaName":93,"dataGaLocation":42},"/solutions/platform-engineering/","platform engineering",{"text":95,"config":96},"Developer Experience",{"href":97,"dataGaName":98,"dataGaLocation":42},"/developer-experience/","Developer experience",{"text":100,"config":101},"MLOps",{"href":102,"dataGaName":100,"dataGaLocation":42},"/topics/devops/the-role-of-ai-in-devops/",{"text":104,"left":105,"config":106,"link":108,"lists":112,"footer":183},"Product",true,{"dataNavLevelOne":107},"solutions",{"text":109,"config":110},"View all Solutions",{"href":111,"dataGaName":107,"dataGaLocation":42},"/solutions/",[113,138,162],{"title":114,"description":115,"link":116,"items":121},"Automation","CI/CD and automation to accelerate deployment",{"config":117},{"icon":118,"href":119,"dataGaName":120,"dataGaLocation":42},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[122,126,130,134],{"text":123,"config":124},"CI/CD",{"href":125,"dataGaLocation":42,"dataGaName":123},"/solutions/continuous-integration/",{"text":127,"config":128},"AI-Assisted Development",{"href":76,"dataGaLocation":42,"dataGaName":129},"AI assisted development",{"text":131,"config":132},"Source Code Management",{"href":133,"dataGaLocation":42,"dataGaName":131},"/solutions/source-code-management/",{"text":135,"config":136},"Automated Software Delivery",{"href":119,"dataGaLocation":42,"dataGaName":137},"Automated software delivery",{"title":139,"description":140,"link":141,"items":146},"Security","Deliver code faster without compromising security",{"config":142},{"href":143,"dataGaName":144,"dataGaLocation":42,"icon":145},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[147,152,157],{"text":148,"config":149},"Application Security Testing",{"href":150,"dataGaName":151,"dataGaLocation":42},"/solutions/application-security-testing/","Application security testing",{"text":153,"config":154},"Software Supply Chain Security",{"href":155,"dataGaLocation":42,"dataGaName":156},"/solutions/supply-chain/","Software supply chain security",{"text":158,"config":159},"Software Compliance",{"href":160,"dataGaName":161,"dataGaLocation":42},"/solutions/software-compliance/","software compliance",{"title":163,"link":164,"items":169},"Measurement",{"config":165},{"icon":166,"href":167,"dataGaName":168,"dataGaLocation":42},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[170,174,178],{"text":171,"config":172},"Visibility & Measurement",{"href":167,"dataGaLocation":42,"dataGaName":173},"Visibility and Measurement",{"text":175,"config":176},"Value Stream Management",{"href":177,"dataGaLocation":42,"dataGaName":175},"/solutions/value-stream-management/",{"text":179,"config":180},"Analytics & Insights",{"href":181,"dataGaLocation":42,"dataGaName":182},"/solutions/analytics-and-insights/","Analytics and insights",{"title":184,"items":185},"GitLab for",[186,191,196],{"text":187,"config":188},"Enterprise",{"href":189,"dataGaLocation":42,"dataGaName":190},"/enterprise/","enterprise",{"text":192,"config":193},"Small Business",{"href":194,"dataGaLocation":42,"dataGaName":195},"/small-business/","small business",{"text":197,"config":198},"Public Sector",{"href":199,"dataGaLocation":42,"dataGaName":200},"/solutions/public-sector/","public sector",{"text":202,"config":203},"Pricing",{"href":204,"dataGaName":205,"dataGaLocation":42,"dataNavLevelOne":205},"/pricing/","pricing",{"text":207,"config":208,"link":210,"lists":214,"feature":298},"Resources",{"dataNavLevelOne":209},"resources",{"text":211,"config":212},"View all resources",{"href":213,"dataGaName":209,"dataGaLocation":42},"/resources/",[215,248,270],{"title":216,"items":217},"Getting started",[218,223,228,233,238,243],{"text":219,"config":220},"Install",{"href":221,"dataGaName":222,"dataGaLocation":42},"/install/","install",{"text":224,"config":225},"Quick start guides",{"href":226,"dataGaName":227,"dataGaLocation":42},"/get-started/","quick setup checklists",{"text":229,"config":230},"Learn",{"href":231,"dataGaLocation":42,"dataGaName":232},"https://university.gitlab.com/","learn",{"text":234,"config":235},"Product documentation",{"href":236,"dataGaName":237,"dataGaLocation":42},"https://docs.gitlab.com/","product documentation",{"text":239,"config":240},"Best practice videos",{"href":241,"dataGaName":242,"dataGaLocation":42},"/getting-started-videos/","best practice videos",{"text":244,"config":245},"Integrations",{"href":246,"dataGaName":247,"dataGaLocation":42},"/integrations/","integrations",{"title":249,"items":250},"Discover",[251,256,260,265],{"text":252,"config":253},"Customer success stories",{"href":254,"dataGaName":255,"dataGaLocation":42},"/customers/","customer success stories",{"text":257,"config":258},"Blog",{"href":259,"dataGaName":5,"dataGaLocation":42},"/blog/",{"text":261,"config":262},"Remote",{"href":263,"dataGaName":264,"dataGaLocation":42},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":266,"config":267},"TeamOps",{"href":268,"dataGaName":269,"dataGaLocation":42},"/teamops/","teamops",{"title":271,"items":272},"Connect",[273,278,283,288,293],{"text":274,"config":275},"GitLab Services",{"href":276,"dataGaName":277,"dataGaLocation":42},"/services/","services",{"text":279,"config":280},"Community",{"href":281,"dataGaName":282,"dataGaLocation":42},"/community/","community",{"text":284,"config":285},"Forum",{"href":286,"dataGaName":287,"dataGaLocation":42},"https://forum.gitlab.com/","forum",{"text":289,"config":290},"Events",{"href":291,"dataGaName":292,"dataGaLocation":42},"/events/","events",{"text":294,"config":295},"Partners",{"href":296,"dataGaName":297,"dataGaLocation":42},"/partners/","partners",{"backgroundColor":299,"textColor":300,"text":301,"image":302,"link":306},"#2f2a6b","#fff","Insights for the future of software development",{"altText":303,"config":304},"the source promo card",{"src":305},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758208064/dzl0dbift9xdizyelkk4.svg",{"text":307,"config":308},"Read the latest",{"href":309,"dataGaName":310,"dataGaLocation":42},"/the-source/","the source",{"text":312,"config":313,"lists":315},"Company",{"dataNavLevelOne":314},"company",[316],{"items":317},[318,323,329,331,336,341,346,351,356,361,366],{"text":319,"config":320},"About",{"href":321,"dataGaName":322,"dataGaLocation":42},"/company/","about",{"text":324,"config":325,"footerGa":328},"Jobs",{"href":326,"dataGaName":327,"dataGaLocation":42},"/jobs/","jobs",{"dataGaName":327},{"text":289,"config":330},{"href":291,"dataGaName":292,"dataGaLocation":42},{"text":332,"config":333},"Leadership",{"href":334,"dataGaName":335,"dataGaLocation":42},"/company/team/e-group/","leadership",{"text":337,"config":338},"Team",{"href":339,"dataGaName":340,"dataGaLocation":42},"/company/team/","team",{"text":342,"config":343},"Handbook",{"href":344,"dataGaName":345,"dataGaLocation":42},"https://handbook.gitlab.com/","handbook",{"text":347,"config":348},"Investor relations",{"href":349,"dataGaName":350,"dataGaLocation":42},"https://ir.gitlab.com/","investor relations",{"text":352,"config":353},"Trust Center",{"href":354,"dataGaName":355,"dataGaLocation":42},"/security/","trust center",{"text":357,"config":358},"AI Transparency Center",{"href":359,"dataGaName":360,"dataGaLocation":42},"/ai-transparency-center/","ai transparency center",{"text":362,"config":363},"Newsletter",{"href":364,"dataGaName":365,"dataGaLocation":42},"/company/contact/","newsletter",{"text":367,"config":368},"Press",{"href":369,"dataGaName":370,"dataGaLocation":42},"/press/","press",{"text":372,"config":373,"lists":374},"Contact us",{"dataNavLevelOne":314},[375],{"items":376},[377,380,385],{"text":49,"config":378},{"href":51,"dataGaName":379,"dataGaLocation":42},"talk to sales",{"text":381,"config":382},"Get help",{"href":383,"dataGaName":384,"dataGaLocation":42},"/support/","get help",{"text":386,"config":387},"Customer portal",{"href":388,"dataGaName":389,"dataGaLocation":42},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":391,"login":392,"suggestions":399},"Close",{"text":393,"link":394},"To search repositories and projects, login to",{"text":395,"config":396},"gitlab.com",{"href":56,"dataGaName":397,"dataGaLocation":398},"search login","search",{"text":400,"default":401},"Suggestions",[402,404,408,410,414,418],{"text":71,"config":403},{"href":76,"dataGaName":71,"dataGaLocation":398},{"text":405,"config":406},"Code Suggestions (AI)",{"href":407,"dataGaName":405,"dataGaLocation":398},"/solutions/code-suggestions/",{"text":123,"config":409},{"href":125,"dataGaName":123,"dataGaLocation":398},{"text":411,"config":412},"GitLab on AWS",{"href":413,"dataGaName":411,"dataGaLocation":398},"/partners/technology-partners/aws/",{"text":415,"config":416},"GitLab on Google Cloud",{"href":417,"dataGaName":415,"dataGaLocation":398},"/partners/technology-partners/google-cloud-platform/",{"text":419,"config":420},"Why GitLab?",{"href":84,"dataGaName":419,"dataGaLocation":398},{"freeTrial":422,"mobileIcon":427,"desktopIcon":432,"secondaryButton":435},{"text":423,"config":424},"Start free trial",{"href":425,"dataGaName":47,"dataGaLocation":426},"https://gitlab.com/-/trials/new/","nav",{"altText":428,"config":429},"Gitlab Icon",{"src":430,"dataGaName":431,"dataGaLocation":426},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203874/jypbw1jx72aexsoohd7x.svg","gitlab icon",{"altText":428,"config":433},{"src":434,"dataGaName":431,"dataGaLocation":426},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1758203875/gs4c8p8opsgvflgkswz9.svg",{"text":436,"config":437},"Get Started",{"href":438,"dataGaName":439,"dataGaLocation":426},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":441,"mobileIcon":445,"desktopIcon":447},{"text":442,"config":443},"Learn more about GitLab Duo",{"href":76,"dataGaName":444,"dataGaLocation":426},"gitlab duo",{"altText":428,"config":446},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":448},{"src":434,"dataGaName":431,"dataGaLocation":426},{"freeTrial":450,"mobileIcon":455,"desktopIcon":457},{"text":451,"config":452},"Back to pricing",{"href":204,"dataGaName":453,"dataGaLocation":426,"icon":454},"back to pricing","GoBack",{"altText":428,"config":456},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":458},{"src":434,"dataGaName":431,"dataGaLocation":426},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":464,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"title":465,"button":466,"image":471,"config":475,"_id":477,"_type":28,"_source":30,"_file":478,"_stem":479,"_extension":33},"/shared/en-us/banner","is now in public beta!",{"text":467,"config":468},"Try the Beta",{"href":469,"dataGaName":470,"dataGaLocation":42},"/gitlab-duo/agent-platform/","duo banner",{"altText":472,"config":473},"GitLab Duo Agent Platform",{"src":474},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":476},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":481,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"data":482,"_id":686,"_type":28,"title":687,"_source":30,"_file":688,"_stem":689,"_extension":33},"/shared/en-us/main-footer",{"text":483,"source":484,"edit":490,"contribute":495,"config":500,"items":505,"minimal":678},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":485,"config":486},"View page source",{"href":487,"dataGaName":488,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":491,"config":492},"Edit this page",{"href":493,"dataGaName":494,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":496,"config":497},"Please contribute",{"href":498,"dataGaName":499,"dataGaLocation":489},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":501,"facebook":502,"youtube":503,"linkedin":504},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[506,529,585,614,648],{"title":60,"links":507,"subMenu":512},[508],{"text":509,"config":510},"DevSecOps platform",{"href":69,"dataGaName":511,"dataGaLocation":489},"devsecops platform",[513],{"title":202,"links":514},[515,519,524],{"text":516,"config":517},"View plans",{"href":204,"dataGaName":518,"dataGaLocation":489},"view plans",{"text":520,"config":521},"Why Premium?",{"href":522,"dataGaName":523,"dataGaLocation":489},"/pricing/premium/","why premium",{"text":525,"config":526},"Why Ultimate?",{"href":527,"dataGaName":528,"dataGaLocation":489},"/pricing/ultimate/","why ultimate",{"title":530,"links":531},"Solutions",[532,537,539,541,546,551,555,558,562,567,569,572,575,580],{"text":533,"config":534},"Digital transformation",{"href":535,"dataGaName":536,"dataGaLocation":489},"/topics/digital-transformation/","digital transformation",{"text":148,"config":538},{"href":150,"dataGaName":148,"dataGaLocation":489},{"text":137,"config":540},{"href":119,"dataGaName":120,"dataGaLocation":489},{"text":542,"config":543},"Agile development",{"href":544,"dataGaName":545,"dataGaLocation":489},"/solutions/agile-delivery/","agile delivery",{"text":547,"config":548},"Cloud transformation",{"href":549,"dataGaName":550,"dataGaLocation":489},"/topics/cloud-native/","cloud transformation",{"text":552,"config":553},"SCM",{"href":133,"dataGaName":554,"dataGaLocation":489},"source code management",{"text":123,"config":556},{"href":125,"dataGaName":557,"dataGaLocation":489},"continuous integration & delivery",{"text":559,"config":560},"Value stream management",{"href":177,"dataGaName":561,"dataGaLocation":489},"value stream management",{"text":563,"config":564},"GitOps",{"href":565,"dataGaName":566,"dataGaLocation":489},"/solutions/gitops/","gitops",{"text":187,"config":568},{"href":189,"dataGaName":190,"dataGaLocation":489},{"text":570,"config":571},"Small business",{"href":194,"dataGaName":195,"dataGaLocation":489},{"text":573,"config":574},"Public sector",{"href":199,"dataGaName":200,"dataGaLocation":489},{"text":576,"config":577},"Education",{"href":578,"dataGaName":579,"dataGaLocation":489},"/solutions/education/","education",{"text":581,"config":582},"Financial services",{"href":583,"dataGaName":584,"dataGaLocation":489},"/solutions/finance/","financial services",{"title":207,"links":586},[587,589,591,593,596,598,600,602,604,606,608,610,612],{"text":219,"config":588},{"href":221,"dataGaName":222,"dataGaLocation":489},{"text":224,"config":590},{"href":226,"dataGaName":227,"dataGaLocation":489},{"text":229,"config":592},{"href":231,"dataGaName":232,"dataGaLocation":489},{"text":234,"config":594},{"href":236,"dataGaName":595,"dataGaLocation":489},"docs",{"text":257,"config":597},{"href":259,"dataGaName":5,"dataGaLocation":489},{"text":252,"config":599},{"href":254,"dataGaName":255,"dataGaLocation":489},{"text":261,"config":601},{"href":263,"dataGaName":264,"dataGaLocation":489},{"text":274,"config":603},{"href":276,"dataGaName":277,"dataGaLocation":489},{"text":266,"config":605},{"href":268,"dataGaName":269,"dataGaLocation":489},{"text":279,"config":607},{"href":281,"dataGaName":282,"dataGaLocation":489},{"text":284,"config":609},{"href":286,"dataGaName":287,"dataGaLocation":489},{"text":289,"config":611},{"href":291,"dataGaName":292,"dataGaLocation":489},{"text":294,"config":613},{"href":296,"dataGaName":297,"dataGaLocation":489},{"title":312,"links":615},[616,618,620,622,624,626,628,632,637,639,641,643],{"text":319,"config":617},{"href":321,"dataGaName":314,"dataGaLocation":489},{"text":324,"config":619},{"href":326,"dataGaName":327,"dataGaLocation":489},{"text":332,"config":621},{"href":334,"dataGaName":335,"dataGaLocation":489},{"text":337,"config":623},{"href":339,"dataGaName":340,"dataGaLocation":489},{"text":342,"config":625},{"href":344,"dataGaName":345,"dataGaLocation":489},{"text":347,"config":627},{"href":349,"dataGaName":350,"dataGaLocation":489},{"text":629,"config":630},"Sustainability",{"href":631,"dataGaName":629,"dataGaLocation":489},"/sustainability/",{"text":633,"config":634},"Diversity, inclusion and belonging (DIB)",{"href":635,"dataGaName":636,"dataGaLocation":489},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":352,"config":638},{"href":354,"dataGaName":355,"dataGaLocation":489},{"text":362,"config":640},{"href":364,"dataGaName":365,"dataGaLocation":489},{"text":367,"config":642},{"href":369,"dataGaName":370,"dataGaLocation":489},{"text":644,"config":645},"Modern Slavery Transparency Statement",{"href":646,"dataGaName":647,"dataGaLocation":489},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":649,"links":650},"Contact Us",[651,654,656,658,663,668,673],{"text":652,"config":653},"Contact an expert",{"href":51,"dataGaName":52,"dataGaLocation":489},{"text":381,"config":655},{"href":383,"dataGaName":384,"dataGaLocation":489},{"text":386,"config":657},{"href":388,"dataGaName":389,"dataGaLocation":489},{"text":659,"config":660},"Status",{"href":661,"dataGaName":662,"dataGaLocation":489},"https://status.gitlab.com/","status",{"text":664,"config":665},"Terms of use",{"href":666,"dataGaName":667,"dataGaLocation":489},"/terms/","terms of use",{"text":669,"config":670},"Privacy statement",{"href":671,"dataGaName":672,"dataGaLocation":489},"/privacy/","privacy statement",{"text":674,"config":675},"Cookie preferences",{"dataGaName":676,"dataGaLocation":489,"id":677,"isOneTrustButton":105},"cookie preferences","ot-sdk-btn",{"items":679},[680,682,684],{"text":664,"config":681},{"href":666,"dataGaName":667,"dataGaLocation":489},{"text":669,"config":683},{"href":671,"dataGaName":672,"dataGaLocation":489},{"text":674,"config":685},{"dataGaName":676,"dataGaLocation":489,"id":677,"isOneTrustButton":105},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[691],{"_path":692,"_dir":693,"_draft":6,"_partial":6,"_locale":7,"content":694,"config":697,"_id":699,"_type":28,"title":18,"_source":30,"_file":700,"_stem":701,"_extension":33},"/en-us/blog/authors/sara-kassabian","authors",{"name":18,"config":695},{"headshot":7,"ctfId":696},"skassabian",{"template":698},"BlogAuthor","content:en-us:blog:authors:sara-kassabian.yml","en-us/blog/authors/sara-kassabian.yml","en-us/blog/authors/sara-kassabian",{"_path":703,"_dir":36,"_draft":6,"_partial":6,"_locale":7,"header":704,"eyebrow":705,"blurb":706,"button":707,"secondaryButton":711,"_id":713,"_type":28,"title":714,"_source":30,"_file":715,"_stem":716,"_extension":33},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":44,"config":708},{"href":709,"dataGaName":47,"dataGaLocation":710},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":49,"config":712},{"href":51,"dataGaName":52,"dataGaLocation":710},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1758326246603]