Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions TOC-tidb-cloud-lake.md
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@
- [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md)
- [FLASHBACK TABLE](/tidb-cloud-lake/sql/flashback-table.md)
- [ALTER TABLE](/tidb-cloud-lake/sql/alter-table.md)
- [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md)
- [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md)
- [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md)
- [ATTACH TABLE](/tidb-cloud-lake/sql/attach-table.md)
- [SHOW CREATE TABLE](/tidb-cloud-lake/sql/show-create-table.md)
Expand Down Expand Up @@ -525,8 +525,10 @@
- [SHOW TABLE FUNCTIONS](/tidb-cloud-lake/sql/show-table-functions.md)
- [SHOW PROCESSLIST](/tidb-cloud-lake/sql/show-processlist.md)
- [SHOW METRICS](/tidb-cloud-lake/sql/show-metrics.md)
- [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table-sql.md)
- [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table-sql.md)
- [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md)
- [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md)
- [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md)
- [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md)
- [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md)
- [VACUUM VIRTUAL COLUMN](/tidb-cloud-lake/sql/vacuum-virtual-column.md)
- [EXECUTE IMMEDIATE](/tidb-cloud-lake/sql/execute-immediate.md)
Expand Down
12 changes: 0 additions & 12 deletions tidb-cloud-lake/guides/data-management.md

This file was deleted.

141 changes: 47 additions & 94 deletions tidb-cloud-lake/guides/data-purge-and-recycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,135 +5,88 @@ summary: In {{{ .lake }}}, data is not immediately deleted when you run DROP, TR

# Data Purge and Recycle

## Overview

In {{{ .lake }}}, data is not immediately deleted when you run `DROP`, `TRUNCATE`, or `DELETE` commands. This enables {{{ .lake }}}'s time travel feature, allowing you to access previous states of your data. However, this approach means that storage space is not automatically freed up after these operations.

```
Before DELETE: After DELETE: After VACUUM:
+----------------+ +----------------+ +----------------+
| Current Data | | New Version | | Current Data |
| | | (After DELETE) | | (After DELETE) |
+----------------+ +----------------+ +----------------+
| Historical Data| | Historical Data| | |
| (Time Travel) | | (Original Data)| | |
+----------------+ +----------------+ +----------------+
Storage not freed Storage freed
```

## VACUUM Commands and Cleanup Scope

{{{ .lake }}} provides three VACUUM commands with **different cleanup scopes**. Understanding what each command cleans is crucial for data management.
This document describes how to reclaim storage space in {{{ .lake }}} after deleting rows or dropping tables. It also explains how to manage temporary spill files and inactive temporary-table sessions.

```
VACUUM DROP TABLE
├── Target: Dropped tables (after DROP TABLE command)
├── S3 Storage: ✅ Removes ALL data (files, segments, blocks, indexes, statistics)
├── Meta Service: ✅ Removes ALL metadata (schema, permissions, records)
└── Result: Complete table removal - CANNOT be recovered

VACUUM TABLE
├── Target: Historical data and orphan files for active tables
├── S3 Storage: ✅ Removes old snapshots, orphan segments/blocks, indexes/stats
├── Meta Service: ❌ Preserves table structure and current metadata
└── Result: Table stays active, only history cleaned

VACUUM TEMPORARY FILES
├── Target: Temporary spill files from queries (joins, sorts, aggregates)
├── S3 Storage: ✅ Removes temp files from crashed/interrupted queries
├── Meta Service: ❌ No metadata (temp files don't have any)
└── Result: Storage cleanup only, rarely needed
```
## Overview

---
Deleting rows or dropping a table does not necessarily release its storage immediately. {{{ .lake }}} retains historical data and dropped objects for recovery. `VACUUM` reclaims storage once data becomes eligible for cleanup. Cleaned history and dropped objects cannot be recovered.

> **🚨 Critical**: Only `VACUUM DROP TABLE` affects the meta service. Other commands only clean storage files.
## Choose a Cleanup Scope

## Using VACUUM Commands
| Command | Cleanup scope | Effect |
|---------|---------------|--------|
| [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) | One writable FUSE table | Removes eligible history while preserving the table and current data. |
| [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md) | Writable FUSE tables in a specified database, or all non-system databases in the current catalog | Performs the same historical cleanup in bulk. |
| [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) | Dropped objects in a specified database, or all databases in the current catalog, including dropped databases | Removes eligible dropped objects, their storage, and their metadata. |
| [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md) | Tenant temporary spill files and inactive temporary-table sessions | Cleans temporary storage. |
| [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md) | Table history, dropped objects, then temporary files | Runs the three cleanup steps in order using their respective retention rules. |

The VACUUM command family is the primary method for cleaning data in {{{ .lake }}}.
Single-table cleanup requires `SUPER` access to the table. Database-scoped batch or dropped-object cleanup requires `SUPER` access to that database. Batch table and dropped-object cleanup without FROM, VACUUM ALL, and temporary-file cleanup require global `SUPER` privilege.

### VACUUM DROP TABLE
Batch table cleanup skips non-FUSE and read-only tables. Ordinary per-table failures are logged and other tables are processed; cancellation and errors listing databases or tables can stop the operation. These commands do not return result sets.

Permanently removes dropped tables from both storage and metadata.
## Clean Table History

```sql
VACUUM DROP TABLE [FROM <database_name>] [DRY RUN [SUMMARY]] [LIMIT <file_count>];
VACUUM TABLE default.my_table;
```

**Options:**

- `FROM <database_name>`: Restrict to a specific database
- `DRY RUN [SUMMARY]`: Preview files to be removed without actually deleting them
- `LIMIT <file_count>`: Limit the number of files to be vacuumed

**Examples:**
Compaction combines small blocks and segments. To compact first and then reclaim eligible historical storage:

```sql
-- Preview files that would be removed
VACUUM DROP TABLE DRY RUN;
OPTIMIZE TABLE default.my_table COMPACT;
VACUUM TABLE default.my_table;
```

-- Preview summary of files that would be removed
VACUUM DROP TABLE DRY RUN SUMMARY;
For batch cleanup:

-- Remove dropped tables from the "default" database
VACUUM DROP TABLE FROM default;
```sql
-- One database
VACUUM TABLES FROM default;

-- Remove up to 1000 files from dropped tables
VACUUM DROP TABLE LIMIT 1000;
-- All non-system databases in the current catalog
VACUUM TABLES;
```

### VACUUM TABLE

Removes historical data and orphan files for active tables (storage-only cleanup).
## Clean Dropped Objects

```sql
VACUUM TABLE <table_name> [DRY RUN [SUMMARY]];
```
-- One database
VACUUM DROPPED OBJECTS FROM default;

**Options:**
-- All databases in the current catalog, including dropped databases
VACUUM DROPPED OBJECTS;
```

- `DRY RUN [SUMMARY]`: Preview files to be removed without actually deleting them
This removes eligible dropped objects and their metadata as well as storage. They can no longer be recovered with `UNDROP`.
Comment thread
lilin90 marked this conversation as resolved.

**Examples:**
## Clean Temporary Files or Run All Steps

```sql
-- Preview files that would be removed
VACUUM TABLE my_table DRY RUN;
VACUUM TEMPORARY FILES;
```

-- Preview summary of files that would be removed
VACUUM TABLE my_table DRY RUN SUMMARY;
To run table-history, dropped-object, and temporary-file cleanup in sequence:

-- Remove historical data from my_table
VACUUM TABLE my_table;
```sql
VACUUM ALL;
```

### VACUUM TEMPORARY FILES
A failure propagated by a step prevents later steps from running. Cleanup already completed is not rolled back.

## Retention and Protection

Removes temporary spill files created during query execution.
For table history and dropped objects, use `data_retention_time_in_days` (1 day by default). For example, set a 2-day retention period for the current session:

```sql
VACUUM TEMPORARY FILES;
SET data_retention_time_in_days = 2;
SHOW SETTINGS LIKE 'data_retention_time_in_days';
```

> **Note:**
>
> Rarely needed during normal operation since {{{ .lake }}} automatically handles cleanup. Manual cleanup is typically only required when {{{ .lake }}} crashes during query execution.
Active-table cleanup preserves snapshots and data referenced by unexpired snapshot tags, including tags with no expiration. Expired tags no longer protect history; VACUUM attempts to remove them without aborting cleanup if tag deletion fails.

## Adjusting Data Retention Time

The VACUUM commands remove data files older than the `DATA_RETENTION_TIME_IN_DAYS` setting. By default, {{{ .lake }}} retains historical data for 1 day (24 hours). You can adjust this setting:
Temporary spill files have a separate retention period of 3 days by default. Override it with RETAIN; this option does not set the lifetime of temporary-table sessions:

```sql
-- Change retention period to 2 days
SET GLOBAL DATA_RETENTION_TIME_IN_DAYS = 2;

-- Check current retention setting
SHOW SETTINGS LIKE 'DATA_RETENTION_TIME_IN_DAYS';
VACUUM TEMPORARY FILES RETAIN 2 DAYS;
```

| Edition | Default Retention | Maximum Retention |
| ---------------------------------------- | ----------------- | ---------------- |
| {{{ .lake }}} Community & Enterprise Editions | 1 day (24 hours) | 90 days |
| {{{ .lake }}} (Personal) | 1 day (24 hours) | 1 day (24 hours) |
| {{{ .lake }}} (Business) | 1 day (24 hours) | 90 days |
6 changes: 4 additions & 2 deletions tidb-cloud-lake/sql/administration-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,10 @@ This page provides reference information for the system administration commands
| Command | Description |
|---------|-------------|
| **[VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md)** | Reclaim storage space from tables |
| **[VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md)** | Clean up dropped table data |
| **[VACUUM TEMP FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md)** | Remove temporary files |
| **[VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md)** | Clean up eligible dropped objects and their storage and metadata |
| **[VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md)** | Clean table history across a database or the current catalog |
| **[VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md)** | Clean table history, dropped objects, and temporary files |
| **[VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md)** | Remove temporary files |
| **[VACUUM VIRTUAL COLUMN](/tidb-cloud-lake/sql/vacuum-virtual-column.md)** | Remove obsolete virtual column files |
| **[SHOW INDEXES](/tidb-cloud-lake/sql/show-indexes.md)** | Display table indexes |

Expand Down
6 changes: 4 additions & 2 deletions tidb-cloud-lake/sql/ddl-table-overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,15 @@ This page provides a comprehensive overview of table operations in {{{ .lake }}}
| [TRUNCATE TABLE](/tidb-cloud-lake/sql/truncate-table.md) | Removes all data from a table while preserving the table's schema | [FLASHBACK TABLE](/tidb-cloud-lake/sql/flashback-table.md) |
| [DROP TABLE](/tidb-cloud-lake/sql/drop-table.md) | Deletes a table | [UNDROP TABLE](/tidb-cloud-lake/sql/undrop-table.md) |
| [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md) | Permanently removes historical data files of a table (Enterprise Edition) | Not recoverable |
| [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) | Permanently removes data files of dropped tables (Enterprise Edition) | Not recoverable |
| [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) | Permanently removes eligible dropped objects and their storage and metadata (Enterprise Edition) | Not recoverable |
| [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md) | Cleans eligible history from tables across a database or the current catalog (Enterprise Edition) | Not recoverable |
| [VACUUM ALL](/tidb-cloud-lake/sql/vacuum-all.md) | Cleans table history, dropped objects, and temporary files (Enterprise Edition) | Not recoverable |

## Table Optimization

| Command | Description |
|---------|-------------|
| [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) | Compacts or purges historical data to save storage space and enhance query performance |
| [OPTIMIZE TABLE](/tidb-cloud-lake/sql/optimize-table.md) | Compacts segments and blocks to enhance query performance |
| [SET CLUSTER KEY](/tidb-cloud-lake/sql/set-cluster-key.md) | Configures a cluster key to enhance query performance for large tables |

> **Note:**
Expand Down
4 changes: 2 additions & 2 deletions tidb-cloud-lake/sql/drop-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,13 @@ DROP TABLE [ IF EXISTS ] [ <database_name>. ]<table_name>

This command only marks the table schema as deleted in the metadata service, ensuring that the actual data remains intact. If you need to recover the deleted table schema, you can use the [UNDROP TABLE](/tidb-cloud-lake/sql/undrop-table.md) command.

For completely removing a table along with its data files, consider using the [VACUUM DROP TABLE](/tidb-cloud-lake/sql/vacuum-drop-table.md) command.
For completely removing a table along with its data files, consider using the [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md) command.

## Examples

### Deleting a Table

This example highlights the use of the DROP TABLE command to delete the "test" table. After dropping the table, any attempt to SELECT from it results in an "Unknown table" error. It also demonstrates how to recover the dropped "test" table using the UNDROP TABLE command, allowing you to SELECT data from it again.
This example highlights the use of the DROP TABLE command to delete the "test" table. After dropping the table, any attempt to SELECT from it results in an "Unknown table" error. It also demonstrates how to recover the dropped "test" table using the `UNDROP TABLE` command, allowing you to SELECT data from it again.

```sql
CREATE TABLE test(a INT, b VARCHAR);
Expand Down
13 changes: 10 additions & 3 deletions tidb-cloud-lake/sql/optimize-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,13 @@ summary: Optimizing a table in {{{ .lake }}} involves compacting or purging hist

# OPTIMIZE TABLE

Optimizing a table in {{{ .lake }}} involves compacting or purging historical data to save storage space and enhance query performance.
`OPTIMIZE TABLE` compacts small segments and blocks to improve query performance. To reclaim storage occupied by eligible historical data after compaction, use [VACUUM TABLE](/tidb-cloud-lake/sql/vacuum-table.md).

<details>
<summary>Why Optimize?</summary>
<div>{{{ .lake }}} stores data in tables using the Parquet format, which is organized into blocks. Additionally, {{{ .lake }}} supports time travel functionality, where each operation that modifies a table generates a Parquet file that captures and reflects the changes made to the table.</div><br/>

<div>As a table accumulates more Parquet files over time, it can lead to performance issues and increased storage requirements. To optimize the table's performance, historical Parquet files can be deleted when they are no longer needed. This optimization can help to improve query performance and reduce the amount of storage space used by the table.</div>
<div>As a table accumulates small blocks and segments, queries may need to read more files and metadata. Compaction merges them into larger units to reduce this overhead. Historical files are retained until they become eligible for cleanup by VACUUM TABLE.</div>
</details>

## {{{ .lake }}} Data Storage: Snapshot, Segment, and Block
Expand Down Expand Up @@ -156,7 +156,7 @@ OPTIMIZE TABLE [database.]table_name COMPACT [LIMIT <segment_count>]

Compacts the table data by merging small blocks and segments into larger ones.

- This command creates a new snapshot (along with compacted segments and blocks) of the most recent table data without affecting the existing storage files, so the storage space won't be released until you purge the historical data.
- This command creates a new snapshot (along with compacted segments and blocks) of the most recent table data without affecting the existing storage files. To reclaim storage from eligible historical data, run [`VACUUM TABLE`](/tidb-cloud-lake/sql/vacuum-table.md) after compaction.

- Depending on the size of the given table, it may take quite a while to complete the execution.

Expand All @@ -169,3 +169,10 @@ Compacts the table data by merging small blocks and segments into larger ones.
```sql
OPTIMIZE TABLE my_database.my_table COMPACT LIMIT 50;
```

To compact and then clean up eligible historical files:

```sql
OPTIMIZE TABLE my_database.my_table COMPACT;
VACUUM TABLE my_database.my_table;
```
30 changes: 30 additions & 0 deletions tidb-cloud-lake/sql/vacuum-all.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
---
title: VACUUM ALL
summary: Runs VACUUM TABLES, VACUUM DROPPED OBJECTS, and VACUUM TEMPORARY FILES in order to reclaim storage.
---

# VACUUM ALL

`VACUUM ALL` cleans table history, dropped objects, and temporary files. `VACUUM ALL` runs the following cleanup operations in order:

1. [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md): Cleans eligible history from writable FUSE tables across non-system databases in the current catalog.
2. [VACUUM DROPPED OBJECTS](/tidb-cloud-lake/sql/vacuum-dropped-objects.md): Cleans eligible dropped objects across databases in the current catalog, including dropped databases.
3. [VACUUM TEMPORARY FILES](/tidb-cloud-lake/sql/vacuum-temporary-files.md): Cleans the tenant's temporary spill files and inactive temporary-table sessions using the default retention and no explicit limit.

Each step follows its own retention and protection rules. Current data in active tables is preserved; cleaned history and dropped objects cannot be recovered. Temporary spill-file retention is separate from `data_retention_time_in_days`.

## Syntax

```sql
VACUUM ALL
```

Requires global `SUPER` privilege. There is no database filter or command option. The command does not return a result set.

A failure that propagates from one step stops execution before the following steps. Per-table errors handled by batch cleanup retain the behavior described in [VACUUM TABLES](/tidb-cloud-lake/sql/vacuum-tables.md). Cleanup already completed is not rolled back.

## Example

```sql
VACUUM ALL;
```
Loading