mirror of
https://github.com/awatertrevi/infisical.git
synced 2026-09-22 13:39:35 +00:00
Merge pull request #3677 from Infisical/secret-scanning-v2-pt-1
feature(secret-scanning-v2): secret scanning architecture and github data source
This commit is contained in:
Scott Wilson
@@ -1,175 +0,0 @@
|
||||
---
|
||||
title: 'Secret Scanning'
|
||||
description: "Scan and prevent secret leaks in your code repositories"
|
||||
---
|
||||
|
||||
The Infisical Secret Scanner allows you to keep an overview and stay alert of exposed secrets across your entire GitHub organization and repositories.
|
||||
|
||||
To further enhance security, we recommend you also use our [CLI Secret Scanner](/cli/scanning-overview#automatically-scan-changes-before-you-commit) to scan for exposed secrets prior to pushing your changes.
|
||||
|
||||
|
||||
<Accordion title="Self-hosting">
|
||||
|
||||
To setup secret scanning on your own instance of Infisical, you can follow the steps below.
|
||||
|
||||
<Steps>
|
||||
<Step title="Create a GitHub App">
|
||||
Create a new GitHub app in your GitHub organization or personal [Developer Settings](https://github.com/settings/apps).
|
||||
|
||||
|
||||
|
||||
### Configure the GitHub App
|
||||
To configure the GitHub app to work with Infisical, you'll need to modify the following settings:
|
||||
- **Homepage URL**: Required to be set. Set it to the URL of your Infisical instance. (e.g. `https://app.infisical.com`)
|
||||
- **Setup URL**: Set this to `https://<your-infisical-instance.com>/organization/secret-scanning`
|
||||
- **Webhook URL**: Set this to `https://<your-infisical-instance.com>/api/v1/secret-scanning/webhook`
|
||||
- **Webhook Secret**: Set this to a random string. This is used to verify the webhook request from Infisical. Use `openssl rand -base64 32` in your terminal to generate a random secret.
|
||||
|
||||
<Note>
|
||||
Remember to save the webhook secret as you will need it in the next step.
|
||||
</Note>
|
||||
|
||||
|
||||
|
||||
### Configure the GitHub App Permissions
|
||||
The GitHub app needs the following permissions:
|
||||
|
||||
Repository permissions:
|
||||
- `Checks`: Read and Write
|
||||
- `Contents`: Read-only
|
||||
- `Issues`: Read and Write
|
||||
- `Pull Requests`: Read and Write
|
||||
- `Metadata`: Read-only (enabled by default)
|
||||
|
||||
|
||||
|
||||
Subscribed events:
|
||||
- `Check run`
|
||||
- `Pull request`
|
||||
- `Push`
|
||||
|
||||
|
||||
|
||||
|
||||
### Create the GitHub App
|
||||
Now you can create the GitHub app by clicking on the "Create GitHub App" button.
|
||||
|
||||
<Note>
|
||||
If you want other Github users to be able to install the app, you need to tick the "Any account" option under "Where can this GitHub App be installed?"
|
||||
</Note>
|
||||
|
||||
|
||||
</Step>
|
||||
|
||||
<Step title="Retrieve the GitHub App ID">
|
||||
After clicking the "Create GitHub App" button, you will be redirected to the GitHub settings page. Here you can copy the "App ID" and save it for later when you need to configure your environment variables for your Infisical instance.
|
||||
|
||||
|
||||
</Step>
|
||||
|
||||
<Step title="Retrieve your GitHub App slug">
|
||||
The GitHub App slug is the name of the app you created in a slug friendly format. You can find the slug in the URL of the app you created.
|
||||
|
||||
|
||||
</Step>
|
||||
|
||||
<Step title="Create a new GitHub App private key">
|
||||
Create a new app private key by clicking on the "Generate a private key" button under the "Private keys" section.
|
||||
|
||||
Once you click the "Generate a private key" button, the private key will be downloaded to your computer. Save this file for later as you will need the private key when configuring Infisical.
|
||||
|
||||
|
||||
|
||||
<Note>
|
||||
Remember to save the private key as you will need it in the next step.
|
||||
</Note>
|
||||
|
||||
</Step>
|
||||
|
||||
|
||||
<Step title="Configure your Infisical instance">
|
||||
Now you can configure your Infisical instance by setting the following environment variables:
|
||||
|
||||
- `SECRET_SCANNING_GIT_APP_ID`: The App ID of your GitHub App.
|
||||
- `SECRET_SCANNING_GIT_APP_SLUG`: The slug of your GitHub App.
|
||||
- `SECRET_SCANNING_PRIVATE_KEY`: The private key of your GitHub App that you created in a previous step.
|
||||
- `SECRET_SCANNING_WEBHOOK_SECRET`: The webhook secret of your GitHub App that you created in a previous step.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
After restarting your Infisical instance, you should be able to use the secret scanning feature within your organization. Follow the steps below to add the GitHub App to your Infisical organization.
|
||||
</Accordion>
|
||||
|
||||
## Install the Infisical Radar GitHub App
|
||||
|
||||
To install the GitHub App, press the "Integrate With GitHub" button in the top right corner of your Infisical Secret Scanning dashboard.
|
||||
|
||||
|
||||
|
||||
Next, you'll be prompted to select which organization you'd like to install the app into. Select the organization you'd like to install the app into by clicking the organization in the menu.
|
||||
|
||||
|
||||
|
||||
Select the repositories you'd like to scan for secrets and press the "Install" button.
|
||||
|
||||
|
||||
|
||||
## Code Scanning
|
||||
|
||||
|
||||
|
||||
Secret scans are built on event-driven architecture. This means that every time a push is made to one of your selected repositories, Infisical will scan the modified files for any exposed secrets.
|
||||
|
||||
If one or more exposed secrets are detected, it will be displayed in your Infisical dashboard. An exposed secret is known as a **"Risk"**. Each risk has the following data associated with it:
|
||||
- **Date**: When the risk was first detected.
|
||||
- **Secret Type**: Which type of secret was detected.
|
||||
- **Info**: Information about the secret, such as the repository, file name, and the committer who made the change.
|
||||
|
||||
Once an exposed secret is detected, all organization admins will be sent an e-mail notification containing details about the exposed secret.
|
||||
|
||||
<Tip>
|
||||
Each risk also contains a "View Exposed Secret" button, which will take you directly to the GitHub commit and to the line where the secret was exposed.
|
||||
</Tip>
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Responding to Exposed Secrets
|
||||
|
||||
After an exposed secret is detected, it will be marked as `Needs Attention`. When there are risks marked as needs attention, it's important to address them as soon as possible.
|
||||
|
||||
You can mark the risk as `Resolved` by changing the status to one of the following states:
|
||||
- **This Is a False Positive**: The secret was not exposed, but was detected by the scanner.
|
||||
- **I Have Rotated The Secret**: The secret was exposed, but it has now been removed.
|
||||
- **No Rotation Needed**: You are choosing to ignore this risk. You may choose to do this if the risk is non-sensitive or otherwise not a security risk.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
## Ignoring Known Secrets
|
||||
If you're intentionally committing a test secret that the secret scanner might flag, you can instruct Infisical to overlook that secret with the methods listed below.
|
||||
|
||||
### infisical-scan:ignore
|
||||
|
||||
To ignore a secret contained in line of code, simply add `infisical-scan:ignore ` at the end of the line as comment in the given programming.
|
||||
|
||||
```js example.js
|
||||
function helloWorld() {
|
||||
console.log("8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ"); // infisical-scan:ignore
|
||||
}
|
||||
```
|
||||
|
||||
### .infisicalignore
|
||||
An alternative method to exclude specific findings involves creating a .infisicalignore file at your repository's root.
|
||||
You can then add the fingerprints of the findings you wish to exclude. The [Infisical scan](/cli/scanning-overview) report provides a unique Fingerprint for each secret found.
|
||||
By incorporating these Fingerprints into the .infisicalignore file, Infisical will skip the corresponding secret findings in subsequent scans.
|
||||
|
||||
```.ignore .infisicalignore
|
||||
bea0ff6e05a4de73a5db625d4ae181a015b50855:frontend/components/utilities/attemptLogin.js:stripe-access-token:147
|
||||
bea0ff6e05a4de73a5db625d4ae181a015b50855:backend/src/json/integrations.json:generic-api-key:5
|
||||
1961b92340e5d2613acae528b886c842427ce5d0:frontend/components/utilities/attemptLogin.js:stripe-access-token:148
|
||||
```
|
||||
96
docs/documentation/platform/secret-scanning/github.mdx
Normal file
96
docs/documentation/platform/secret-scanning/github.mdx
Normal file
@@ -0,0 +1,96 @@
|
||||
---
|
||||
title: "GitHub Secret Scanning"
|
||||
sidebarTitle: "GitHub"
|
||||
description: "Learn how to configure secret scanning for GitHub."
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Create a [GitHub Radar Connection](/integrations/app-connections/github-radar)
|
||||
|
||||
## Create a GitHub Data Source in Infisical
|
||||
|
||||
<Tabs>
|
||||
<Tab title="Infisical UI">
|
||||
1. Navigate to your Secret Scanning Project's Dashboard and click the **Add Data Source** button.
|
||||
|
||||
|
||||
2. Select the **GitHub** option.
|
||||
|
||||
|
||||
3. Select the **GitHub Radar Connection** to use and configure which repositories you would like to scan. Then click **Next**.
|
||||
|
||||
|
||||
- **GitHub Radar Connection** - the connection that has access to the repositories you want to scan.
|
||||
- **Scan Repositories** - select which repositories you would like to scan.
|
||||
- **All Repositories** - Infisical will scan all repositories associated with your connection.
|
||||
- **Select Repositories** - Infisical will scan the selected repositories.
|
||||
- **Auto-Scan Enabled** - whether Infisical should automatically perform a scan when a push is made to configured repositories.
|
||||
|
||||
4. Give your data source a name and description (optional). Then click **Next**.
|
||||
|
||||
|
||||
- **Name** - the name of the data source. Must be slug-friendly.
|
||||
- **Description** (optional) - a description of this data source.
|
||||
|
||||
5. Review your data source, then click **Create Data Source**.
|
||||
|
||||
|
||||
6. Your **GitHub Data Source** is now available and will begin a full scan if **Auto-Scan** is enabled.
|
||||
|
||||
|
||||
7. You can view repositories and scan results by clicking on your data source.
|
||||
|
||||
|
||||
8. In addition, you can review any findings from the **Findings Page**.
|
||||
|
||||
</Tab>
|
||||
<Tab title="API">
|
||||
To create a GitHub Data Source, make an API request to the [Create GitHub Data Source](/api-reference/endpoints/secret-scanning/data-sources/github/create) API endpoint.
|
||||
|
||||
### Sample request
|
||||
|
||||
```bash Request
|
||||
curl --request POST \
|
||||
--url https://us.infisical.com/api/v2/secret-scanning/data-sources/github \
|
||||
--header 'Content-Type: application/json' \
|
||||
--data '{
|
||||
"name": "my-github-source",
|
||||
"projectId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
|
||||
"description": "my github data source",
|
||||
"connectionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
|
||||
"isAutoScanEnabled": true,
|
||||
"config": {
|
||||
"includeRepos": ["*"]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Sample response
|
||||
|
||||
```bash Response
|
||||
{
|
||||
"dataSource": {
|
||||
"id": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
|
||||
"externalId": "1234567890",
|
||||
"name": "my-github-source",
|
||||
"description": "my github data source",
|
||||
"isAutoScanEnabled": true,
|
||||
"projectId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
|
||||
"createdAt": "2023-11-07T05:31:56Z",
|
||||
"updatedAt": "2023-11-07T05:31:56Z",
|
||||
"type": "github",
|
||||
"connectionId": "3c90c3cc-0d44-4b50-8888-8dd25736052a",
|
||||
"connection": {
|
||||
"app": "github-radar",
|
||||
"name": "my-radar-app",
|
||||
"id": "3c90c3cc-0d44-4b50-8888-8dd25736052a"
|
||||
},
|
||||
"config": {
|
||||
"includeRepos": ["*"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
</Tab>
|
||||
</Tabs>
|
||||
230
docs/documentation/platform/secret-scanning/overview.mdx
Normal file
230
docs/documentation/platform/secret-scanning/overview.mdx
Normal file
@@ -0,0 +1,230 @@
|
||||
---
|
||||
title: "Secret Scanning"
|
||||
sidebarTitle: "Overview"
|
||||
description: "Scan and prevent secret leaks in your code repositories"
|
||||
---
|
||||
|
||||
## Introduction
|
||||
|
||||
Monitor and detect exposed secrets across your data sources, including code repositories, with Infisical Secret Scanning.
|
||||
|
||||
For additional security, we recommend using our [CLI Secret Scanner](/cli/scanning-overview#automatically-scan-changes-before-you-commit) to check for exposed secrets before pushing your code changes.
|
||||
|
||||
<Note>
|
||||
Secret Scanning is a paid feature.
|
||||
If you're using Infisical Cloud, then it is available under the **Enterprise Tier**. If you're self-hosting Infisical,
|
||||
then you should contact team@infisical.com to purchase an enterprise license to use it.
|
||||
</Note>
|
||||
|
||||
## How Secret Scanning Works
|
||||
|
||||
Secret Scanning consists of several components that enable you to quickly respond to secret leaks:
|
||||
|
||||
- **Scanner Engine**: The core component that analyzes your code and detects potential secrets using pattern matching and entropy analysis
|
||||
- **Real-time Monitoring**: Provides continuous surveillance of your repositories for immediate detection of exposed secrets
|
||||
- **Alert System**: Notifies organization admins via email when secrets are detected
|
||||
- **Risk Management**: Allows tracking and managing detected secrets with different status options
|
||||
- **Data Sources**: Integrates with various data sources and version control systems
|
||||
- **Customizable Rules**: Supports ignore patterns and custom configurations to reduce false positives
|
||||
|
||||
These components work together to provide comprehensive secret detection and incident response capabilities.
|
||||
|
||||
### Data Sources
|
||||
|
||||
Data sources are configured integrations with external platforms, such as a GitHub organization or a GitLab group, that establish secure connections for scanning purposes using [App Connections](/integrations/app-connections/overview).
|
||||
|
||||
A data source acts as a secure intermediary between the external system and the scanner engine. It manages a collection of scannable resources (such as repositories) and handles the authentication and communication required for scanning operations.
|
||||
|
||||
|
||||
|
||||
### Resources
|
||||
|
||||
Resources are the atomic, scannable units, such as a repository, that can be monitored for secret exposure. Resources are added automatically when a data source is scanned and updated when scanning events are triggered, such as when a user pushes changes to GitHub.
|
||||
|
||||
Each resource maintains its own scanning history and status, allowing for granular monitoring and management of secret scanning across your organization.
|
||||
|
||||
|
||||
|
||||
### Scans
|
||||
|
||||
Scans can be initiated in two ways:
|
||||
|
||||
1. **Full Scan** - Manually triggered scan that comprehensively checks either all resources associated with a data source or a single selected resource.
|
||||
|
||||
2. **Diff Scan** - Automatically executed when **Auto-Scan** is enabled on a data source. This scan type specifically focuses on updates to existing resources.
|
||||
|
||||
All scan activities can be monitored in real-time through the Infisical UI, which displays:
|
||||
- Current scan status
|
||||
- Timestamp of the scan
|
||||
- Resource(s) being scanned
|
||||
- Detection results (whether any secrets were found)
|
||||
|
||||
|
||||
|
||||
### Findings
|
||||
|
||||
Findings are automatically generated when secret leaks are detected during scanning operations. Each finding contains comprehensive information including:
|
||||
- The specific scanning rule that identified the leak
|
||||
- File location and line number where the secret was found
|
||||
- Resource-specific details (e.g., commit hash and author for Git repositories)
|
||||
|
||||
Findings are initially marked as **Unresolved** and can be updated to one of the following statuses with additional remarks:
|
||||
- **Resolved** - The issue has been addressed
|
||||
- **False Positive** - The detection was incorrect
|
||||
- **Ignore** - The finding can be safely disregarded
|
||||
|
||||
These status options help teams effectively track and manage the lifecycle of detected secret leaks.
|
||||
|
||||
|
||||
|
||||
### Configuration
|
||||
|
||||
You can configure custom scanning rules and exceptions by updating your project's scanning configuration via the UI or API.
|
||||
|
||||
The configuration options allow you to:
|
||||
- Define custom scanning patterns and rules
|
||||
- Set up ignore patterns to reduce false positives
|
||||
- Specify file path exclusions
|
||||
- Configure entropy thresholds for secret detection
|
||||
- Add allowlists for known safe patterns
|
||||
|
||||
For detailed configuration options, expand the example configuration below.
|
||||
|
||||
<Accordion title="Example Configuration">
|
||||
```toml
|
||||
# Title for the configuration file
|
||||
title = "Some title"
|
||||
|
||||
|
||||
# This configuration is the foundation that can be expanded. If there are any overlapping rules
|
||||
# between this base and the expanded configuration, the rules in this base will take priority.
|
||||
# Another aspect of extending configurations is the ability to link multiple files, up to a depth of 2.
|
||||
# "Allowlist" arrays get appended and may have repeated elements.
|
||||
# "useDefault" and "path" cannot be used simultaneously. Please choose one.
|
||||
[extend]
|
||||
# useDefault will extend the base configuration with the default config:
|
||||
# https://raw.githubusercontent.com/Infisical/infisical/main/cli/config/infisical-scan.toml
|
||||
useDefault = true
|
||||
# or you can supply a path to a configuration. Path is relative to where infisical cli
|
||||
# was invoked, not the location of the base config.
|
||||
path = "common_config.toml"
|
||||
|
||||
# An array of tables that contain information that define instructions
|
||||
# on how to detect secrets
|
||||
[[rules]]
|
||||
|
||||
# Unique identifier for this rule
|
||||
id = "some-identifier-for-rule"
|
||||
|
||||
# Short human readable description of the rule.
|
||||
description = "awesome rule 1"
|
||||
|
||||
# Golang regular expression used to detect secrets. Note Golang's regex engine
|
||||
# does not support lookaheads.
|
||||
regex = '''one-go-style-regex-for-this-rule'''
|
||||
|
||||
# Golang regular expression used to match paths. This can be used as a standalone rule or it can be used
|
||||
# in conjunction with a valid `regex` entry.
|
||||
path = '''a-file-path-regex'''
|
||||
|
||||
# Array of strings used for metadata and reporting purposes.
|
||||
tags = ["tag","another tag"]
|
||||
|
||||
# A regex match may have many groups, this allows you to specify the group that should be used as (which group the secret is contained in)
|
||||
# its entropy checked if `entropy` is set.
|
||||
secretGroup = 3
|
||||
|
||||
# Float representing the minimum shannon entropy a regex group must have to be considered a secret.
|
||||
# Shannon entropy measures how random a data is. Since secrets are usually composed of many random characters, they typically have high entropy
|
||||
entropy = 3.5
|
||||
|
||||
# Keywords are used for pre-regex check filtering.
|
||||
# If rule has keywords but the text fragment being scanned doesn't have at least one of it's keywords, it will be skipped for processing further.
|
||||
# Ideally these values should either be part of the identifier or unique strings specific to the rule's regex
|
||||
# (introduced in v8.6.0)
|
||||
keywords = [
|
||||
"auth",
|
||||
"password",
|
||||
"token",
|
||||
]
|
||||
|
||||
# You can include an allowlist table for a single rule to reduce false positives or ignore commits
|
||||
# with known/rotated secrets
|
||||
[rules.allowlist]
|
||||
description = "ignore commit A"
|
||||
commits = [ "commit-A", "commit-B"]
|
||||
paths = [
|
||||
'''go\.mod''',
|
||||
'''go\.sum'''
|
||||
]
|
||||
# note: (rule) regexTarget defaults to check the _Secret_ in the finding.
|
||||
# if regexTarget is not specified then _Secret_ will be used.
|
||||
# Acceptable values for regexTarget are "match" and "line"
|
||||
regexTarget = "match"
|
||||
regexes = [
|
||||
'''process''',
|
||||
'''getenv''',
|
||||
]
|
||||
# note: stopwords targets the extracted secret, not the entire regex match
|
||||
# if the extracted secret is found in the stopwords list, the finding will be skipped (i.e not included in report)
|
||||
stopwords = [
|
||||
'''client''',
|
||||
'''endpoint''',
|
||||
]
|
||||
|
||||
|
||||
# This is a global allowlist which has a higher order of precedence than rule-specific allowlists.
|
||||
# If a commit listed in the `commits` field below is encountered then that commit will be skipped and no
|
||||
# secrets will be detected for said commit. The same logic applies for regexes and paths.
|
||||
[allowlist]
|
||||
description = "global allow list"
|
||||
commits = [ "commit-A", "commit-B", "commit-C"]
|
||||
paths = [
|
||||
'''gitleaks\.toml''',
|
||||
'''(.*?)(jpg|gif|doc)'''
|
||||
]
|
||||
|
||||
# note: (global) regexTarget defaults to check the _Secret_ in the finding.
|
||||
# if regexTarget is not specified then _Secret_ will be used.
|
||||
# Acceptable values for regexTarget are "match" and "line"
|
||||
regexTarget = "match"
|
||||
|
||||
regexes = [
|
||||
'''219-09-9999''',
|
||||
'''078-05-1120''',
|
||||
'''(9[0-9]{2}|666)-\d{2}-\d{4}''',
|
||||
]
|
||||
# note: stopwords targets the extracted secret, not the entire regex match
|
||||
# if the extracted secret is found in the stopwords list, the finding will be skipped (i.e not included in report)
|
||||
stopwords = [
|
||||
'''client''',
|
||||
'''endpoint''',
|
||||
]
|
||||
```
|
||||
</Accordion>
|
||||
|
||||
|
||||
|
||||
## Ignoring Known Secrets
|
||||
If you're intentionally committing a test secret that the secret scanner might flag, you can instruct Infisical to overlook that secret with the methods listed below.
|
||||
|
||||
### infisical-scan:ignore
|
||||
|
||||
To ignore a secret contained in line of code, simply add `infisical-scan:ignore ` at the end of the line as comment in the given programming.
|
||||
|
||||
```js example.js
|
||||
function helloWorld() {
|
||||
console.log("8dyfuiRyq=vVc3RRr_edRk-fK__JItpZ"); // infisical-scan:ignore
|
||||
}
|
||||
```
|
||||
|
||||
### .infisicalignore
|
||||
An alternative method to exclude specific findings involves creating a .infisicalignore file at your repository's root.
|
||||
You can then add the fingerprints of the findings you wish to exclude. The [Infisical scan](/cli/scanning-overview) report provides a unique Fingerprint for each secret found.
|
||||
By incorporating these Fingerprints into the .infisicalignore file, Infisical will skip the corresponding secret findings in subsequent scans.
|
||||
|
||||
```.ignore .infisicalignore
|
||||
bea0ff6e05a4de73a5db625d4ae181a015b50855:frontend/components/utilities/attemptLogin.js:stripe-access-token:147
|
||||
bea0ff6e05a4de73a5db625d4ae181a015b50855:backend/src/json/integrations.json:generic-api-key:5
|
||||
1961b92340e5d2613acae528b886c842427ce5d0:frontend/components/utilities/attemptLogin.js:stripe-access-token:148
|
||||
```
|
||||
Reference in New Issue
Block a user