> For the complete documentation index, see [llms.txt](https://docs.cogram.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cogram.com/organization-administration/continuous-data-export-aws-setup.md). # Set up an S3 bucket for Continuous Data Export Step-by-step for your AWS administrator: a private S3 bucket used only by Cogram, and an access key that can access only its prefix in that bucket. This page is for the person who manages your organization's AWS account. When you finish, you have a private S3 bucket used only by Cogram, and an access key that can access only the export's prefix in that bucket. Cogram then writes new and changed records to the bucket every day. What happens to records removed in Cogram depends on the export mode your Cogram admin chooses: * **Archive** (recommended): Cogram never deletes. Removed records keep their files, and their `state.json` says `delete: true`, for your own cleanup. * **Mirror**: Cogram deletes what was removed in Cogram, so the bucket equals your organization's data. The two modes need different permissions (step 3). For what the export does and how to run it, see [Continuous Data Export (S3 Push)](/organization-administration/continuous-data-export.md). Continuous Data Export is in beta and is enabled per organization. If you would like to use it, write to before you set up the bucket. **Prerequisites** * Access to an AWS account where you can create S3 buckets and IAM users. * A Cogram owner or admin to enter the key in Cogram. This can be you. * The export mode the Cogram admin chooses: **Archive** or **Mirror**. **What Cogram can and cannot do with the key** * It can upload and overwrite files in the bucket, and list the bucket. In Mirror mode it can also delete files. * It cannot open or download a file, change the bucket's settings or policies, or access any other bucket or AWS service. ## 1. Create the bucket Use a new bucket only for Cogram. Cogram keeps the bucket in line from its own record, and in Mirror mode it deletes the files it wrote when their records leave Cogram, so other tools and people must not write to it. 1. In the AWS console, open **S3** and select **Create bucket**. 2. Enter a **Bucket name**, for example `acme-cogram-export`. 3. Select the **AWS Region**. Cogram runs in **US East (N. Virginia)**, `us-east-1`. A bucket in `us-east-1` avoids AWS data transfer charges between regions. If your data residency rules require another region, choose it; the export works the same way. 4. Keep **Object Ownership** at **ACLs disabled**. 5. Keep **Block all public access** turned on. 6. **Bucket Versioning** is your choice: with versioning, AWS keeps the earlier versions of files that Cogram overwrites or deletes, and Cogram works the same either way. 7. Keep the default encryption, **Amazon S3 managed keys (SSE-S3)**. To use an AWS KMS key instead, see [Troubleshooting](#troubleshooting). 8. Select **Create bucket**. **Recommended: a lifecycle rule for incomplete uploads.** On the bucket's **Management** tab, select **Create lifecycle rule**, and choose **Delete incomplete multipart uploads**, for example after 7 days. Cogram uploads large files in parts. If an upload is cut off and Cogram cannot clean it up, the parts would otherwise stay and be billed. In Mirror mode, do not add a rule that expires or deletes current files: Cogram decides what the bucket holds. In Archive mode, your own process decides what to delete and when. ## 2. Create an IAM user for Cogram 1. Open **IAM**, select **Users**, then **Create user**. 2. Enter a **User name**, for example `cogram-continuous-data-export`. 3. Leave **Provide user access to the AWS Management Console** cleared. Cogram only needs an access key. 4. Select **Next**, skip the permissions step, and select **Create user**. ## 3. Give the user access to the bucket 1. Open the user, then select **Add permissions** → **Create inline policy**. 2. Select **JSON** and paste the policy of the export mode below. Replace `` with your bucket name, and `` with the key prefix that you enter in Cogram, with its trailing slash, for example `cogram/`. In Cogram, **Copy IAM policy** copies the same policy with the bucket and the prefix filled in. **Archive** (recommended): no delete permission. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:AbortMultipartUpload"], "Resource": "arn:aws:s3:::/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": "*" } } } ] } ``` **Mirror**: adds `s3:DeleteObject`. ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject", "s3:AbortMultipartUpload", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": "*" } } } ] } ``` If you leave the prefix empty (the bucket root), use `arn:aws:s3:::/*` as the first `Resource` and remove the `Condition`. 3. Select **Next**, name the policy, for example `cogram-continuous-data-export`, and select **Create policy**. What each permission is for: | Permission | What Cogram does with it | | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `s3:PutObject` | Uploads new and changed files, Cogram's own files under `_cogram/`, and the connection test file. | | `s3:AbortMultipartUpload` | Cleans up a large file it could not finish uploading, so no incomplete upload is left behind. | | `s3:DeleteObject` | Mirror mode only. Deletes the files of records removed in Cogram, and the old location of moved records. The connection test writes and deletes a temporary `.tmp` file to prove it. `_cogram/connection-test.json` stays. | | `s3:ListBucket` | Lists the whole prefix when an admin selects **Verify bucket**, and lists the connection test file. In Archive mode, a run also lists single keys of a record that comes back, to check which of its files the bucket still holds. | The policy has no `s3:GetObject`: Cogram never opens or downloads a file from your bucket. The policy covers only the prefix. If you later save another prefix in Cogram, for example to change the export mode, update the policy first. In Archive mode, your own process deletes what Cogram archived, with its own permissions. See [Clean up an Archive-mode bucket](/organization-administration/continuous-data-export-cleanup.md). ## 4. Create an access key 1. On the user's page, open **Security credentials** and select **Create access key**. 2. Select **Application running outside AWS**, then **Next**, then **Create access key**. 3. Copy the **Access key** and the **Secret access key**. AWS shows the secret only once. Hand both values to the Cogram admin through a channel your organization approves for secrets, or enter them in Cogram yourself. ## 5. Enter the key in Cogram 1. Open [Organization Settings → General → Continuous Data Export](https://app.cogram.com/dashboard/settings/admin/overview#continuous-data-export). 2. Select **Amazon S3**, the bucket's **Region**, and enter the **Bucket name**, the optional **Key prefix**, the **Access key ID** and the **Secret access key**. 3. Select the **Export mode**, then **Test connection**. Cogram writes `_cogram/connection-test.json` under your prefix and lists it. Every test writes this file again. In Mirror mode it also writes and deletes a temporary file. **Connection works** confirms that the key can upload and list, and in Mirror mode delete. The test does not use `s3:AbortMultipartUpload`, which Cogram needs only when a large upload fails. 4. Turn on **Export every day** and select **Save**. ## Expected result * After a successful test, the bucket holds `_cogram/connection-test.json`. * Within a few minutes of saving, the first run starts. * When it completes, the bucket holds `projects/`, `organization/` and `_cogram/` under your prefix, and `workflows/` when your organization has workflows. The **Runs** table in Cogram shows the run as **Completed**. * To look at the files, use your own AWS identity, for example the S3 console or `aws s3 ls s3:///`. The key that you gave Cogram can list the prefix, but it cannot open or download a file. ## Costs * **Storage and requests.** AWS bills your account for what the bucket stores and for the requests Cogram makes: * One upload per file on the first run, then one upload per new or changed file. * One small state file for each record, written again when the record changes. * In Mirror mode, one delete request for each group of up to 1,000 removed files. * **Verify bucket** lists the whole prefix, one request per 1,000 files. * **Data transfer.** Data that arrives in your bucket is not charged to you. A bucket in `us-east-1`, the region Cogram runs in, also keeps the transfer free of charge on Cogram's side. ## Rotate the key 1. In IAM, create a second access key for the same user. 2. In Cogram, enter the new **Access key ID** and **Secret access key**, select **Test connection**, then **Save**. 3. In IAM, deactivate the old key, then delete it. A new key for the same bucket and prefix does not start a new backfill: the export continues where it was. So does a change of region alone. A change of bucket or prefix starts a new backfill. The next run then copies everything to the new location, and Cogram does not touch the old one again. ## Troubleshooting **Symptom**: **Test connection** fails with "Access denied". **Likely cause**: The policy names another bucket, or it misses one of the permissions. The message names the permission to check: the upload needs `s3:PutObject`, the listing `s3:ListBucket`, the delete (Mirror mode only) `s3:DeleteObject`. **Fix**: Compare the policy with the one in step 3 above, and check that its prefix matches the **Key prefix** in Cogram, with its trailing slash. A test that fails after the upload can leave its test file under `_cogram/`. You can delete it. **Symptom**: Runs fail with "Access denied" although the test worked. **Likely cause**: A bucket policy or an organization-wide policy (a service control policy) denies a request that the user policy allows, for example uploads without a specific encryption setting. **Fix**: Review the bucket policy and your organization's policies for statements that deny `s3:PutObject`, `s3:DeleteObject` or `s3:ListBucket`. **Symptom**: The test or runs fail with "Access denied" and the bucket's default encryption uses an AWS KMS key (SSE-KMS). **Likely cause**: The IAM user may not use the KMS key, so S3 cannot encrypt what Cogram writes. **Fix**: Allow the user `kms:GenerateDataKey` on that key, and `kms:Decrypt` too, which S3 needs for large files uploaded in parts. Add them to the user's policy, and check that the key policy allows the user, or allows IAM policies to grant access to the key. Or switch the bucket's default encryption to SSE-S3. **Symptom**: The test fails with "All access to this bucket has been disabled". **Likely cause**: AWS suspended access to the bucket. **Fix**: Contact AWS Support for the account. ## Next steps * [Continuous Data Export (S3 Push)](/organization-administration/continuous-data-export.md): the schedule, the runs table, Verify bucket and the bucket layout. * [Clean up an Archive-mode bucket](/organization-administration/continuous-data-export-cleanup.md): delete what an Archive-mode export archived. * [Data export package layout](/organization-administration/data-export-package-layout.md): what each folder holds. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.cogram.com/organization-administration/continuous-data-export-aws-setup.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.