OpenBao Features - Recursive Lists (SCAN) & Filtering | OpenBao
Skip to main content<br>This is the fifth part of a multi-part series on OpenBao's features.
Last time we talked about declarative<br>plugin configuration and how it made deploying and adopting plugins much<br>easier. With OCI-based distribution operators can deploy plugins with just a<br>few configuration snippets, mirroring OpenTofu's approach.
We hinted at addressing two of the most-requested features<br>in HashiCorp Vault: recursive list support<br>and filtering of list responses.
As mentioned there by Vault community members,<br>we've supported recursive lists since<br>OpenBao v2.2.0 and<br>filtered lists since OpenBao v2.4.0.<br>And, for any plugin developers out there, we support it in our external plugin<br>SDK<br>including storage helpers which should work on Vault<br>as well.
Question<br>What other places need recursive list support?Reach out to us if we've missed one!
Overview
OpenBao has always supported a custom HTTP verb, LIST, for listing entries<br>under a path. This is also supported via the ?list=true query<br>parameter on a regular GET-verb request, for use<br>when clients cannot support custom verbs.
In designing recursive listing, we realized<br>many endpoints (like KVv2's list entries)<br>made sense as both LIST and SCAN operations. Rather than forcing authors<br>to implement a new endpoint design--for example, moving from a layout like<br>LIST /secrets/metadata/:path to LIST /secrets/metadata-recursive/:path--we<br>opted to introduce a new verb, SCAN for this. This allowed us to extend ACL<br>policies to allow policy authors control over recursive lists without having<br>to investigate a plugin's layout.
For instance, the policy:
path "secrets/*" {
capabilities = ["read", "create", "update", "list", "patch", "delete"]
gives users the ability to perform mostly cheap operations, while restricting<br>their ability to do recursive lists (scan operations). However, if they<br>also add the scan operation:
path "secrets/*" {
capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
this would be more expensive and maybe should only be allowed on specific<br>subdirectories within a KVv2 layout:
path "secrets/metadata/my-app/*" {
capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
Usage
As we discussed in past blogs, OpenBao has support for both<br>pagination and transactional<br>storage. When coupled with<br>SCAN support, this gives users a powerful consistency tool to inspect<br>a large number of secrets at once. For instance, the API call:
SCAN secrets/detailed-metadata/my-app
would return a list response with metadata about each secret as well. While<br>this would usually be a 1+n operation in Vault (list all secret and then<br>fetch its metadata), OpenBao will return them in a single operation, with a<br>transaction to ensure internal consistency of the results.
The same holds true for namespaces: bao namespace list will show all<br>top-level children of the current namespace, but bao namespace scan will<br>recurse and show children-of-children and the full hierarchy.
OpenBao also supports the scan operation via the bao scan command or<br>in the Go API via client.Logical().Scan(...)<br>and related operations.
Security
This ties into another commonly requested feature in HashiCorp<br>Vault: restricting list (and<br>now scans!) to entries which the user can view.
In OpenBao, we implemented a policy keyword,<br>list_scan_response_keys_filter_path, which takes a<br>text/template expression for limiting<br>visible results. Visible results are entries which (when templating is applied<br>according to the filter path) have list access for entries ending in a / or<br>read access otherwise. This means the policy author must know the<br>corresponding type of the plugin and where to map list entries to. For<br>example in KVv2, one could either map entries in a list or scan to the data<br>(/data/) or metadata (/metadata/) paths.
Consider an ACL policy like:
# Allow listing secrets broadly but limit to visible results (read or list):
path "secrets/metadata/*" {
capabilities = ["list", "scan"]
# See also: https://openbao.org/docs/concepts/policies/#filtering-list-or-scan-results
list_scan_response_keys_filter_path = "{{ .path }}{{ .key }}"
# Allow reading secrets and metadata in shared:
path "secrets/data/shared/*" {
capabilities = ["read"]
path "secrets/metadata/shared/*" {
capabilities = ["read"]
# But allow full access to a personal space.
path "secrets/data/personal/*" {
capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
path "secrets/metadata/personal/*" {
capabilities = ["read", "create", "update", "list", "patch", "scan", "delete"]
In this example, a call to the scan endpoint would show entries under shared/<br>and personal/ but hide entries under private/ due to the filtering on the<br>list result.
Use of text/template allows for advanced functionality like changing the path<br>as well; for instance, if metadata wasn't widely used but direct secret...