> For the complete documentation index, see [llms.txt](https://docs.cogram.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cogram.com/organization-administration/continuous-data-export-cleanup.md). # Clean up an Archive-mode bucket For your IT team: delete what an Archive-mode export says may go, by hand or on a schedule with AWS Lambda, with Cogram's reference cleanup script. In Archive mode, Cogram never deletes a file from your bucket. When a record leaves Cogram, or its retention ends, its files stay and its state file says that they may go. This page explains how your own process deletes them, with the reference script that Cogram provides. When you finish, a scheduled job deletes the files of removed and expired records every day, and keeps everything else. {% hint style="info" %} **Minimum setup: about 30 minutes** You need only three things: 1. The policy of step 1, on the execution role of a Lambda function. 2. One Lambda function that runs the script (step 4). 3. One daily schedule in EventBridge Scheduler, after Cogram's daily run at 03:00 UTC (step 4). Everything else on this page is optional: the dry run by hand (step 2), a grace period, more workers, and the CloudWatch alerts. The function starts in a dry run. Read its logs after the first runs, then set `COGRAM_CLEANUP_DRY_RUN` to `false`. {% endhint %} **Prerequisites** * Continuous Data Export runs in **Archive** mode. See [Continuous Data Export (S3 Push)](/organization-administration/continuous-data-export.md). * An AWS identity of your own that can list, read and delete under the export's prefix (step 1). The key that you gave Cogram cannot delete in Archive mode. * To run the script by hand: Python 3.11 or later, the `boto3` package, and the AWS CLI to save the script's key as a profile (or edit `~/.aws/credentials` by hand). To schedule it: an AWS account where you can create a Lambda function. **When to use this** * Your records policy says that a record removed in Cogram must also leave your archive, after a delay that you choose. * Your Data retention setting removes meeting minutes and transcripts after a period, and the archive must follow it. * A deleted part of a record, such as an attachment, must leave the archive too. ## How the cleanup works Every record in the bucket has a state file: `state.json` in the record's folder, or `.state.json` next to a record that is one JSON file. See [The state file of every record](/organization-administration/continuous-data-export.md#the-state-file-of-every-record) for every field. A cleanup needs to know only where Cogram writes state files (the table below), not the rest of the bucket layout: 1. List every key under your prefix whose name is `state.json` or ends with `.state.json`, in a place where Cogram writes state files (see the table below). Skip the `_cogram/` folder, which holds Cogram's own files, and files with such a name in other places, such as an `attachments/` folder. 2. Read each state file. If `delete` is `true`, or `expires_at` has passed, delete every key in `files`, then the state file itself. 3. Delete every key in `delete_files`. The state files are JSON files in the bucket. They are not S3 object metadata (`x-amz-meta-*`): read them with a GET request. Cogram writes state files only in these places, under your prefix. Read a file named like a state file only there: anywhere else, for example under an `attachments/` folder, it is a file of a record. | State file | Record | | ------------------------------------------------------------------------------------------------------ | ------------------------------------- | | `organization/state.json` | The organization's own files | | `organization/agents/.state.json`, `organization/prompts/.state.json` | An agent, a prompt | | `organization/knowledge//state.json` | A knowledge entry of the organization | | `projects//state.json` | A project | | `projects////state.json` | A record of the project (see below) | | `projects//chat//messages/.state.json` | A chat message | | `unassigned_project///state.json`, `unassigned_project/chat//messages/.state.json` | A record with no project | | `workflows//state.json` | A workflow | | `workflows//runs/.state.json` | A run of a workflow | `` is one of `meetings`, `emails`, `documents`, `drawings`, `drawing_sets`, `reports`, `observations`, `transmittals`, `submittals`, `rfis`, `change_orders`, `tasks`, `correspondence`, `chat` and `knowledge`. The script holds this list. Cogram added `change_orders` after the first release of the script, so if you downloaded it before then, download it again below. An older copy skips the change order folders and never deletes their files. The state file of a live record never says `delete: true`. When a record that left comes back, for example an email refiled back to its project or a restored document, Cogram writes its files again under the same keys and rewrites its state file to `active` and `delete: false`. The files go up first, so between the two writes the old state file still says `delete: true`. So: * Read each state file again right before you delete its keys. * Do not delete the files of a record that says `delete: true` when one of its files was written after its state file. Leave the record for a later run. Run the cleanup when no Cogram run is in progress: not during the daily run at 03:00 UTC, a **Run now**, or the continuation of a large run. A record that comes back during a cleanup can lose files that Cogram has just written. If that happens, select **Verify bucket** in Cogram: it uploads the missing files of live records again. ## Get the script {% file src="/files/89OEpUam6ayKTJgAnfHv" %} cogram\_cleanup.py: the reference cleanup script (Python, one file) {% endfile %} The script does the three steps above. It is one file, with no package and no build step. It runs as a command, or as an AWS Lambda function. Its header explains every setting, the permissions it needs and how to deploy it. What it does beyond the three steps: * **Dry run by default.** It deletes nothing unless you pass `--execute`, or set `COGRAM_CLEANUP_DRY_RUN=false`. A dry run logs every key that it would delete. * **A grace period** (`--grace-days`): it waits that many days after a record is marked `delete: true`, or after its `expires_at`. `delete_files` do not wait. * **A record that comes back is kept.** In the window before Cogram rewrites its state file, it skips a record that says `delete: true` when one of its files is newer than its state file (`kept_newer_files`). * **The state file goes last.** If a delete fails, the state file stays, and the next run tries again. * **Safety checks.** It reads state files only where Cogram writes them, and deletes only keys that their state file may list: `.state.json` lists only `.json` next to it, and `state.json` only keys in its own folder. A state file that breaks a check stays as it is, with a warning (`kept_refused`). * **A summary line** for each run, and an error exit (or a failed Lambda invocation) when something went wrong. The script is a reference. Review it, run it in a dry run first, and run it under your own AWS account and policies. ## 1. Give the script its own permissions Create a policy for the identity that runs the script: the Lambda's execution role, or a user of your team. Replace ``, and `` with the export's prefix and its trailing slash, for example `cogram/`: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:GetObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::/*" }, { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": "*" } } } ] } ``` * With an empty prefix (the bucket root), use `arn:aws:s3:::/*` and remove the `Condition`. * If the bucket uses an AWS KMS key (SSE-KMS), also allow `kms:Decrypt` on that key. | Permission | What the script does with it | | ----------------- | -------------------------------------------------------------------------------------------------------------- | | `s3:ListBucket` | Lists the prefix to find the state files. Lists a single key to check that it exists, and when it was written. | | `s3:GetObject` | Reads the state files. | | `s3:DeleteObject` | Deletes the files and the state files, up to 1,000 keys in one request. | To run the script by hand, give it a user of its own, not the user of the key that Cogram has: 1. In IAM, create a user, for example `cogram-archive-cleanup`, as in step 2 of [Set up an S3 bucket](/organization-administration/continuous-data-export-aws-setup.md). 2. Add the policy above to the user as an inline policy, as in step 3 of that page. 3. Create an access key for the user, as in step 4 of that page. 4. Save the key as a named profile on the machine that runs the script, for example `aws configure --profile cogram-archive-cleanup`. The commands in steps 2 and 3 below use this name as ``. ## 2. Run a dry run by hand 1. Download `cogram_cleanup.py`. 2. Install `boto3` if you do not have it: `pip install boto3`. 3. Run the script with the profile of step 1. Replace `` with the bucket's region, for example `us-east-1`: ```bash AWS_PROFILE= python cogram_cleanup.py \ --bucket --prefix --region ``` 4. Read the log. Each line `would delete ` names a key that a real run deletes. The last line is the summary. In a dry run, its counts, such as `records_deleted`, are what a real run with the same settings deletes if the bucket does not change in between. A Cogram run, or the passing of an `expires_at`, can make more records due. A dry run deletes nothing. To try a grace period, add `--grace-days` to the dry run. The script keeps a record until the grace period has passed since the record was marked `delete: true`, or since its `expires_at`. Until then, the dry run does not list the record's `files` or its state file. Keys in `delete_files` do not wait: the dry run lists them at once, as `would delete removed files of record `. ## 3. Run it for real Run the same command with `--execute` added. Keep the other flags of the dry run, so the real run deletes what the dry run listed: ```bash AWS_PROFILE= python cogram_cleanup.py \ --bucket --prefix --region \ --execute ``` With a grace period, add it to both runs, for example `--grace-days 7`. A record that you removed in Cogram today is then deleted by the first run 7 days later or more, not by this run. ## 4. Schedule it with AWS Lambda 1. Create a Lambda function with a Python 3.12 or later runtime. Upload `cogram_cleanup.py` as its code. 2. Set the handler to `cogram_cleanup.lambda_handler`. 3. Add the environment variables `COGRAM_CLEANUP_BUCKET` and `COGRAM_CLEANUP_PREFIX`, and optionally `COGRAM_CLEANUP_GRACE_DAYS`. Leave `COGRAM_CLEANUP_DRY_RUN` unset for the first runs. 4. Give the execution role the policy of step 1, and `AWSLambdaBasicExecutionRole` for its logs. 5. Set the timeout to 15 minutes and the memory to 1,024 MB. 6. In Amazon EventBridge Scheduler, create a schedule that invokes the function once a day after Cogram's daily run at 03:00 UTC, for example `cron(0 6 * * ? *)`. 7. In the function's **Asynchronous invocation** settings, set **Retry attempts** to 0. The next scheduled run repeats a run that failed. 8. Read the function's logs in CloudWatch after the first runs. When the dry runs delete what you expect, set `COGRAM_CLEANUP_DRY_RUN` to `false`. | Setting | Environment variable | Flag | Default | | ----------------------------- | --------------------------- | ------------------------ | --------------- | | The bucket (required) | `COGRAM_CLEANUP_BUCKET` | `--bucket` | | | The export's prefix | `COGRAM_CLEANUP_PREFIX` | `--prefix` | the bucket root | | Dry run | `COGRAM_CLEANUP_DRY_RUN` | `--dry-run`, `--execute` | `true` | | Grace period, in days | `COGRAM_CLEANUP_GRACE_DAYS` | `--grace-days` | `0` | | Region of the bucket | `COGRAM_CLEANUP_REGION` | `--region` | the AWS default | | State files handled at a time | `COGRAM_CLEANUP_WORKERS` | `--workers` | `16` | A flag, or a key of the `cogram_cleanup` object of the Lambda event, such as `{"cogram_cleanup": {"dry_run": false}}`, wins over the environment variable. The script ignores the rest of the event. `dry_run` takes `true`, `false`, `1` or `0` only. The script lists the whole prefix and reads every state file on each run. A run that comes within one minute of the Lambda timeout stops, logs `complete=false` and fails. Nothing is lost: the next run starts again. If your runs do not complete, raise `COGRAM_CLEANUP_WORKERS`, for example to 48, or run the script on a machine with no time limit. **Optional: alerts in CloudWatch.** The script logs `refused to delete` for each state file it refuses (`kept_refused`), and `the record is coming back` for each record it keeps because its files are newer (`kept_newer_files`). To count them, add metric filters to the function's log group, then an alarm on each metric: ```bash aws logs put-metric-filter --log-group-name /aws/lambda/ \ --filter-name cogram-cleanup-kept-refused --filter-pattern '"refused to delete"' \ --metric-transformations metricName=KeptRefused,metricNamespace=CogramCleanup,metricValue=1 aws logs put-metric-filter --log-group-name /aws/lambda/ \ --filter-name cogram-cleanup-kept-newer-files --filter-pattern '"the record is coming back"' \ --metric-transformations metricName=KeptNewerFiles,metricNamespace=CogramCleanup,metricValue=1 ``` The function's own `Errors` metric counts failed runs. ## Expected result * The log ends with a summary line, for example: ``` summary: dry_run=false complete=true state_files_scanned=1204 records_deleted=3 child_keys_deleted=17 records_kept=1201 kept_newer_files=0 kept_refused=0 skipped=0 errors=0 ``` * The files and the state files of removed and expired records are gone from the bucket. Every other record keeps all its files. * In Cogram, **Verify bucket** counts the records that you cleaned up as **cleaned up**, and does not upload them again. ## Things to know * **How often.** Once a day is enough. Cogram writes the bucket in its daily run, when an admin selects **Run now**, and in the continuation of a large run, which can go on for hours. A removed record keeps its state file until your process deletes it, so a cleanup can run at any time. * **A grace period of at least one day avoids a second pass for expired meetings.** Cogram's retention task expires a meeting within an hour of its `expires_at`, and the next daily run writes the meeting's `meeting.json` again with what Cogram still shows. If the script deleted the meeting before that, the meeting comes back once as that file and its state file, and the next cleanup deletes them again. * **Removed files stay listed.** After you delete the keys in `delete_files` of a live record, its state file still lists them. The script checks that each one still exists before it deletes it, so a bucket with versioning gets no new delete marker on every run. A key in both `files` and `delete_files` is kept. * **Versioning.** On a bucket with versioning, a delete adds a delete marker, and the earlier versions stay. To remove them too, add a lifecycle rule that expires noncurrent versions. * **Emails and their retention date.** An email's `expires_at` appears once your organization turns on email expiry, and the script can remove the email at that date. While expiry is off, `expires_at` is null, and the script removes an email only after it leaves Cogram (`delete: true`). See [Email retention](/email/email-retention.md). * **A record that comes back.** If a cleaned-up record is restored in Cogram, or its retention gets longer while it still exists in Cogram, Cogram uploads the whole record again. * **Delete only what the state files say.** Do not delete other files under the prefix. Cogram does not upload a file of an active record again until it changes in Cogram, or until you select **Verify bucket**. ## Troubleshooting **Symptom**: The run fails with `could not list`, or the summary says `errors=1` or more and the log has `kept :` lines with `AccessDenied`. **Likely cause**: The identity that runs the script misses a permission of step 1, or the policy names another prefix. **Fix**: Compare the policy with step 1. The prefix in the policy must match `--prefix`, with its trailing slash. **Symptom**: The summary says `kept_refused=1` or more, and the log has `kept : refused to delete ...`. **Likely cause**: A state file lists a key that it may not list, for example a key outside its folder. The script deletes nothing of that record, and the run does not fail. **Fix**: Write to with the log lines. **Symptom**: The summary says `kept_newer_files=1` or more. **Likely cause**: A record that said `delete: true` is coming back in Cogram. Cogram wrote its files again, and has not yet rewritten its state file. **Fix**: None is needed. Cogram rewrites the state file to `active` and `delete: false` for a record that is live again, and until then the script keeps the record on every run. If the record leaves again, its state file changes, and a later cleanup deletes it. If the count stays above zero for several days, write to with the log lines. **Symptom**: The log has `skipped : Cogram writes no state file there`, or `skipped : not a Cogram state file`. **Likely cause**: A file of a record is named `state.json` or ends with `.state.json`, for example an email attachment. **Fix**: None is needed. The script skips it, and deletes it with its record. **Symptom**: The log has `skipped : Cogram writes no state file there`, and `` is under a `change_orders/` folder. **Likely cause**: Your copy of the cleanup script is older than change orders in the export, and does not know their folders. It deletes nothing there. **Fix**: Download the current script from this page and replace your copy. **Symptom**: The Lambda fails with `complete=false`. **Likely cause**: The prefix holds more state files than one run can read in 15 minutes. **Fix**: Raise `COGRAM_CLEANUP_WORKERS`, or run the script on a machine with no time limit, for example an ECS scheduled task. ## Support Write to with questions about the state files or the script. Include the summary line and the log lines of the keys in question. Cogram cannot see your bucket or your AWS account. ## Next steps * [Continuous Data Export (S3 Push)](/organization-administration/continuous-data-export.md): the modes, the runs and **Verify bucket**. * [Set up an S3 bucket for Continuous Data Export](/organization-administration/continuous-data-export-aws-setup.md): the bucket and the key that Cogram uses. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.cogram.com/organization-administration/continuous-data-export-cleanup.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.