Skip to content

Document collect_schemas tuning options and partition counting #39408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
databergs wants to merge 1 commit into
base: master
Choose a base branch
from
+56 −1
Draft

Document collect_schemas tuning options and partition counting #39408

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 52 additions & 0 deletions hugo/content/en/database_monitoring/schema_explorer.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,58 @@ instances:
# dbname: '<DB_NAME>'
```

### Tuning schema collection

@OliviaShoup OliviaShoup Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you could use our tabs shortcode for the PostgreSQL/SQL Server/MySQL stuff. hugo/content/en/database_monitoring/agent_integration_overhead.md is a good example:

{{< tabs >}}
{{% tab "Postgres" %}}
...
{{% /tab %}}
{{% /tabs %}}

converting would match the docs, improve scannability, and make each engine's section self-contained (helpful for AI chunking, since right now nothing but bold text separates the three)


The `collect_schemas` options available and their defaults differ by database engine.

**PostgreSQL**

| Option | Default | Description |
|---|---|---|
| `enabled` | `true` | Set to `false` to disable schema collection. |

@OliviaShoup OliviaShoup Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

integrations-core's postgres/datadog_checks/postgres/data/conf.yaml.example documents enabled as @param enabled - boolean - optional - default: false, same as SQL Server and MySQL. this row says the default is true

Suggested change
| `enabled` | `true` | Set to `false` to disable schema collection. |
| `enabled` | `false` | Set to `true` to enable schema collection. |

| `max_tables` | `300` | Maximum number of tables the Agent collects from the instance. Tables beyond this limit are not collected. |
| `max_columns` | `50` | Maximum number of columns the Agent collects per table. |
| `max_query_duration` | `60` | Maximum duration, in seconds, of the query that collects schema information. |
| `collection_interval` | `600` | Interval, in seconds, between schema collection runs. |

```yaml
collect_schemas:
enabled: true
max_tables: 1000
```

<div class="alert alert-warning">Each partition of a partitioned table counts as a separate table toward PostgreSQL's <code>max_tables</code> limit. For example, if <code>table_1</code> is partitioned into <code>partition_a.table_1</code> and <code>partition_b.table_1</code>, it counts as two tables, not one. Heavily partitioned databases can reach the default limit of 300 with far fewer logical tables than expected. If tables are missing from the Schemas page, raise <code>max_tables</code> to account for the total partition count. See <a href="/database_monitoring/setup_postgres/advanced_configuration/#handling-many-relations">Handling many relations</a> for more information.</div>

@OliviaShoup OliviaShoup Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓question❓

tracing postgres/datadog_checks/postgres/schemas.py: PG_TABLES_QUERY_V10_PLUS (the query max_tables limits) explicitly excludes partitions (WHERE c.relkind IN ('r','p','f') AND c.relispartition != 't'), and partition count is tracked separately as num_partitions metadata on the parent table rather than as additional rows. that suggests each partitioned table counts as one table toward max_tables, not one per partition. could someone from database-monitoring confirm before this merges? I may be missing a second code path, but this line hasn't changed since 2025-11-21


Raising `max_tables` increases the cost of each collection run. On instances with a large number of tables, also consider raising `max_query_duration` and `collection_interval` to reduce load on the database.

@datadog-prod-us1-3 datadog-prod-us1-3 Bot Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Longer query timeout does not reduce database load

Users can increase database resource use while they try to reduce it on large PostgreSQL instances.

Assertion details
  • Input: A user follows the new advice for a PostgreSQL instance with many tables and raises max_query_duration to reduce load.
  • Expected: Describe a longer timeout as an option that lets large schema collections finish. Describe a longer collection interval as the option that reduces collection frequency and load.
  • Actual: The text says that a longer query timeout helps reduce database load. A longer timeout lets the query use resources for more time before cancellation.

Was this helpful? React 👍 or 👎
🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest


**SQL Server**

| Option | Default | Description |
|---|---|---|
| `enabled` | `false` | Set to `true` to enable schema collection. |
| `max_tables` | `300` | Maximum number of tables the Agent collects from the instance. Tables beyond this limit are not collected. |
| `collection_interval` | `600` | Interval, in seconds, between schema collection runs. |

```yaml
collect_schemas:
enabled: true
max_tables: 1000
```

**MySQL**

| Option | Default | Description |
|---|---|---|
| `enabled` | `false` | Set to `true` to enable schema collection. |
| `collection_interval` | `600` | Interval, in seconds, between schema collection runs. |
| `max_execution_time` | `60` | Maximum duration, in seconds, of the query that collects schema information. |

```yaml
collect_schemas:
enabled: true
collection_interval: 300
```

## Tables overview

The Tables overview lists all tracked tables across your databases, grouped by table name, with the following columns:
Expand Down
5 changes: 4 additions & 1 deletion hugo/content/en/database_monitoring/setup_postgres/advanced_configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ instances:
replace_digits: true
```

Partitioning also affects schema collection. Each partition counts individually toward the `max_tables` limit for `collect_schemas`, so partitioned databases may need a higher limit for full coverage. See [Tuning schema collection][2] for details.

@OliviaShoup OliviaShoup Aug 21, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this repeats the partition-counting claim that doesn't match what I found in postgres/datadog_checks/postgres/schemas.py


## Raising the sampling rate

If you have queries that are relatively infrequent or execute quickly, raise the sampling rate by lowering the `collection_interval` value to collect explain plans more frequently.
Expand Down Expand Up @@ -75,8 +77,9 @@ instances:
| --- | --- | --- |
| `enabled` | `false` | Set to `true` to enable column statistics collection. |
| `collection_interval` | `3600` | Lower for more responsive statistics (at the cost of more queries against `pg_stats`); raise on very large or busy clusters to reduce query load. |
| `max_tables` | `500` | Raise if you monitor a database with more than 500 tables and want full coverage; lower to cap collection cost. |
| `max_tables` | `500` | Raise if you monitor a database with more than 500 tables and want full coverage; lower to cap collection cost. This limit is separate from the `max_tables` option under `collect_schemas`, which defaults to `300`. |

For column statistics to populate, the underlying tables must have had `ANALYZE` (or autoanalyze) run against them at least once — `pg_stats` is empty for tables with no collected statistics.

[1]: /database_monitoring/setup_postgres/selfhosted/#create-the-column-statistics-function
[2]: /database_monitoring/schema_explorer/#tuning-schema-collection
Loading