Added blog article on search exclusion features

This commit is contained in:
squidfunk 2021-09-26 17:53:29 +02:00
parent 3c4bc8af7a
commit 830ae7cc1e
8 changed files with 382 additions and 28 deletions

View File

@ -1,3 +1,12 @@
mkdocs-material-7.3.0+insiders-3.1.1 (2021-09-26)
* Fixed ordering bug in search exclusion logic
mkdocs-material-7.3.0+insiders-3.1.0 (2021-09-26)
* Added support for excluding pages, sections, and elements from search
* Fixed #2803: Code block annotations not visible when printing
mkdocs-material-7.3.0 (2021-09-23)
* Added support for sticky navigation tabs

View File

@ -0,0 +1,216 @@
---
template: overrides/main.html
description: >
How we rebuilt client-side search, delivering a better user experience while
making it faster and smaller at the same time
disqus: mkdocs-material
search:
exclude: true
---
# Excluding content from search
__The latest Insiders release brings three new simple ways to exclude
dedicated parts of a document from the search index, allowing for more
fine-grained control.__
<aside class="mdx-author" markdown="1">
![@squidfunk][1]
<span>__Martin Donath__ · @squidfunk</span>
<span>
:octicons-calendar-24: September 26, 2021 ·
:octicons-clock-24: 5 min read ·
[:octicons-tag-24: 7.3.0+insiders-3.1.1](../../insiders/changelog.md#311-_-september-26-2021)
</span>
</aside>
[1]: https://avatars.githubusercontent.com/u/932156
---
Two weeks ago, Material for MkDocs Insiders shipped a [brand new search
plugin][2], yielding [massive improvements in usability][3], but also in [speed
and size][4] of the search index. Interestingly, as discussed in the previous
blog article, we only scratched the surface of what's now possible. This
release brings some useful features that enhance the writing experience,
allowing for more fine-grained control of what pages, sections and blocks of a
Markdown file should be indexed by the built-in search functionality.
_The following section discusses existing solutions for excluding pages and
sections from the search index. If you immediately want to learn what's new,
skip to the [section just after that][5]._
[2]: search-better-faster-smaller.md
[3]: search-better-faster-smaller.md#whats-new
[4]: search-better-faster-smaller.md#benchmarks
[5]: #whats-new
## Prior art
MkDocs has a rich and thriving ecosystem of [plugins][6], and it comes as no
surprise that there's already a fantastic plugin by @chrieke to exclude specific
sections of a Markdown file the [mkdocs-exclude-search][7] plugin. It can be
installed with:
```
pip install mkdocs-exclude-search
```
__How it works__: the plugin post-processes the `search_index.json` file that
is generated by the built-in search plugin, giving the author the ability to
exclude certain pages and sections by adding a few lines of configuration to
`mkdocs.yml`. An example:
``` yaml
plugins:
- search
- exclude-search:
exclude:
- page.md
- page.md#section
- directory/*
- /*/page.md
```
It's easy to see that the plugin follows a configuration-centric approach, which
adds support for advanced filtering techniques like infix- and suffix-filtering
using wildcards. While this is a very powerful idea, it comes with some
downsides:
1. __Exclusion patterns and content are not co-located__: exclusion patterns
need to be defined in `mkdocs.yml`, and not as part of the respective
document or section to be excluded. This might result in stale exclusion
patterns, leading to unintended behavior:
- When a headline is changed, its slug (permalink) also changes, which might
suddenly match (or unmatch) a pattern, e.g., when an author fixes a typo
in a headline.
- As exclusion patterns support the use of wildcards, different authors
might overwrite each other's patterns without any immediate feedback since
the plugin does only report the number of excluded documents not _what_
has been excluded.[^1]
[^1]:
When the log level is set to `DEBUG`, the plugin will report exactly which
pages and sections have been excluded from the search index, but MkDocs will
now flood the terminal with debug output from its core and other plugins.
2. __Exclusion control might be too coarse__: The [mkdocs-exclude-search][7]
plugin only allows for the exclusion of pages and sections. It's not possible
to exclude parts of a section, e.g., content that is irrelevant to search but
must be included as part of the documentation.
[6]: https://github.com/mkdocs/mkdocs/wiki/MkDocs-Plugins
[7]: https://github.com/chrieke/mkdocs-exclude-search
## What's new?
The latest Insiders release brings fine-grained control for [__excluding pages,
sections, and blocks__][8] from the search index, implemented through front
matter, as well as the [Attribute List][9] extension. Note that it doesn't
replace the [mkdocs-exclude-search][7] plugin but _complements_ it.
[8]: ../../setup/setting-up-site-search.md#search-exclusion
[9]: https://python-markdown.github.io/extensions/attr_list/
### Excluding pages
An entire page can be excluded from the search index by adding a simple
directive to the front matter of the respective Markdown file. The good thing
is that the author now only has to check the top of the document to learn
whether it is excluded or not:
``` bash
---
search:
exclude: true
---
# Document title
...
```
### Excluding sections
If a section should be excluded, the author can use the [Attribute List][9]
extension to add a __pragma__ called `{ data-search-exclude }` at the end of a
heading. The pragma is not included in the final HTML, as search pragmas are
filtered by the search plugin before the page is rendered:
=== "`docs/page.md`"
``` markdown
# Document title
## Section 1
The content of this section is included
## Section 2 { data-search-exclude }
The content of this section is excluded
```
=== "`search_index.json`"
``` json
{
...
"docs": [
{
"location":"page/",
"text":"",
"title":"Document title"
},
{
"location":"page/#section-1",
"text":"<p>The content of this section is included</p>",
"title":"Section 1"
}
]
}
```
### Excluding blocks
If even more fine-grained control is desired, the __pragma__ can be added to
any [block-level element][10] or [inline-level element][11] that is officially
supported by the [Attribute List][9] extension:
=== "`docs/page.md`"
``` markdown
# Document title
The content of this block is included
The content of this block is excluded
{ data-search-exclude }
```
=== "`search_index.json`"
``` json
{
...
"docs": [
{
"location":"page/",
"text":"<p>The content of this block is included</p>",
"title":"Document title"
},
]
}
```
[10]: https://python-markdown.github.io/extensions/attr_list/#block-level
[11]: https://python-markdown.github.io/extensions/attr_list/#inline-level
## Conclusion
The latest release brings three simple ways to control more precisely what goes
into the search index and what doesn't. It complements the already very powerful
[mkdocs-exclude-search][7] plugin, allowing for new methods of shaping the
structure, size and content of the search index.

View File

@ -20,7 +20,8 @@ smaller at the same time.__
<span>__Martin Donath__ · @squidfunk</span>
<span>
:octicons-calendar-24: September 13, 2021 ·
:octicons-clock-24: 15 min read
:octicons-clock-24: 15 min read ·
[:octicons-tag-24: 7.2.6+insiders-3.0.0](../../insiders/changelog.md#300-_-september-13-2021)
</span>
</aside>

View File

@ -6,6 +6,25 @@ search:
# Blog
<h2>Excluding content from search</h2>
__The latest Insiders release brings three new simple ways to exclude dedicated
parts of a document from the search index, allowing for more fine-grained
control.__
Two weeks ago, Material for MkDocs Insiders shipped a brand new search plugin,
yielding massive improvements in usability, but also in speed and size of the
search index. Interestingly, as discussed in the previous blog article, we only
scratched the surface of what's now possible. This release brings some useful
features that enhance the writing experience, allowing for more fine-grained
control of what pages, sections and blocks of a Markdown file should be indexed
by the built-in search functionality.
[Continue reading :octicons-arrow-right-24:][2]{ .md-button }
[2]: 2021/excluding-content-from-search.md
<h2>Search: better, faster, smaller</h2>
__This is the story of how we managed to completely rebuild client-side search,

View File

@ -6,6 +6,15 @@ template: overrides/main.html
## Material for MkDocs Insiders
### 3.1.1 <small>_ September 26, 2021</small>
- Fixed ordering bug in search exclusion logic
### 3.1.0 <small>_ September 26, 2021</small>
- Added support for excluding pages, sections, and elements from search
- Fixed #2803: Code block annotations not visible when printing
### 3.0.1 <small>_ September 19, 2021</small>
- Added support for using literal `h1-6` tags for search plugin

View File

@ -141,6 +141,7 @@ The following features are currently exclusively available to sponsors:
- [x] [Rich search previews :material-new-box:][36]
- [x] [Tokenizer with lookahead :material-new-box:][37]
- [x] [Advanced search highlighting :material-new-box:][38]
- [x] [Excluding content from search :material-new-box:][39]
- [x] [Social cards][34]
- [x] [Cookie consent][33]
- [x] [Linking content tabs][32]
@ -195,14 +196,14 @@ the public for general availability.
- [x] [Custom admonition icons][31]
- [x] [Linking content tabs][32]
[30]: ../setup/setting-up-site-search.md#boosting-a-page
[30]: ../setup/setting-up-site-search.md#search-boosting
[31]: ../reference/admonitions.md#changing-the-icons
[32]: ../reference/content-tabs.md#linking-content-tabs
#### $ 7,000 Royal Gold
- [x] [Cookie consent][33]
- [ ] Exclude pages from search
- [ ] Was this page helpful?
- [ ] Link cards
[33]: ../setup/setting-up-site-analytics.md#cookie-consent
@ -221,11 +222,13 @@ the public for general availability.
- [x] [Rich search previews][36]
- [x] [Tokenizer with lookahead][37]
- [x] [Advanced search highlighting][38]
- [x] [Excluding content from search][39]
[35]: ../blog/2021/search-better-faster-smaller.md
[36]: ../blog/2021/search-better-faster-smaller.md#rich-search-previews
[37]: ../blog/2021/search-better-faster-smaller.md#tokenizer-lookahead
[38]: ../blog/2021/search-better-faster-smaller.md#accurate-highlighting
[39]: ../setup/setting-up-site-search.md#search-exclusion
### Goals completed
@ -300,17 +303,17 @@ implemented behind feature flags; all configuration changes are
backward-compatible. This means that your users will be able to build the
documentation locally with Material for MkDocs and when they push their changes,
it can be built with Insiders (e.g. as part of GitHub Actions). Thus, it's
recommended to [install Insiders][39] only in CI, as you don't want to expose
recommended to [install Insiders][40] only in CI, as you don't want to expose
your `GH_TOKEN` to users.
[39]: ../publishing-your-site.md#github-pages
[40]: ../publishing-your-site.md#github-pages
### Payment
_We don't want to pay for sponsorship every month. Are there any other options?_
Yes. You can sponsor on a yearly basis by [switching your GitHub account to a
yearly billing cycle][40]. If for some reason you cannot do that, you could
yearly billing cycle][41]. If for some reason you cannot do that, you could
also create a dedicated GitHub account with a yearly billing cycle, which you
only use for sponsoring (some sponsors already do that).
@ -318,7 +321,7 @@ If you have any problems or further questions, don't hesitate to contact me at
sponsors@squidfunk.com. Note that one-time payments are not eligible for
Insiders, but of course, very appreciated.
[40]: https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/changing-the-duration-of-your-billing-cycle
[41]: https://docs.github.com/en/github/setting-up-and-managing-billing-and-payments-on-github/changing-the-duration-of-your-billing-cycle
### Terms
@ -327,7 +330,7 @@ commercial project. Can we use Insiders under the same terms and conditions?_
Yes. Whether you're an individual or a company, you may use _Material for MkDocs
Insiders_ precisely under the same terms as Material for MkDocs, which are given
by the [MIT license][41]. However, we kindly ask you to respect the following
by the [MIT license][42]. However, we kindly ask you to respect the following
guidelines:
- Please __don't distribute the source code__ of Insiders. You may freely use
@ -338,7 +341,7 @@ guidelines:
- If you cancel your subscription, you're removed as a collaborator and will
miss out on future updates of Insiders. However, you may __use the latest
version__ that's available to you __as long as you like__. Just remember that
[GitHub deletes private forks][42].
[GitHub deletes private forks][43].
[41]: ../license.md
[42]: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/removing-a-collaborator-from-a-personal-repository
[42]: ../license.md
[43]: https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/removing-a-collaborator-from-a-personal-repository

View File

@ -238,7 +238,7 @@ For setup instructions, refer to the [official documentation][19].
## Usage
### Boosting a page
### Search boosting
[:octicons-file-code-24: Source][20] ·
:octicons-note-24: Metadata ·
@ -259,7 +259,103 @@ search:
```
[20]: ../insiders/index.md
[21]: ../../reference/meta-tags/#metadata
[21]: ../reference/meta-tags.md#metadata
### Search exclusion
[:octicons-file-code-24: Source][20] ·
:octicons-beaker-24: Experimental ·
[:octicons-heart-fill-24:{ .mdx-heart } Insiders only][20]{ .mdx-insiders }
#### Excluding pages
Sometimes, it's necessary to exclude a page from the search index, which can be
achieved by adding the following front matter using the [Metadata][21]
extension:
``` bash
---
search:
exclude: true
---
# Document title
...
```
#### Excluding sections
With the help of the [Attribute List][22] extension, it's possible to exclude a
specific section of a page from search by adding the `{ data-search-exclude }`
pragma after the Markdown heading:
=== "`docs/page.md`"
``` markdown
# Document title
## Section 1
The content of this section is included
## Section 2 { data-search-exclude }
The content of this section is excluded
```
=== "`search_index.json`"
``` json
{
...
"docs": [
{
"location":"page/",
"text":"",
"title":"Document title"
},
{
"location":"page/#section-1",
"text":"<p>The content of this section is included</p>",
"title":"Section 1"
}
]
}
```
[22]: ../reference/images.md#attribute-list
#### Excluding blocks
If even more fine-grained control is needed, content blocks can be excluded
from search sections with a `{ data-search-exclude }` pragma, so they will not
be included in the search index:
=== "`docs/page.md`"
``` markdown
# Document title
The content of this block is included
The content of this block is excluded
{ data-search-exclude }
```
=== "`search_index.json`"
``` json
{
...
"docs": [
{
"location":"page/",
"text":"<p>The content of this block is included</p>",
"title":"Document title"
}
]
}
```
## Customization
@ -273,12 +369,12 @@ your needs.
### Query transformation
[:octicons-file-code-24: Source][22] ·
[:octicons-file-code-24: Source][23] ·
:octicons-mortar-board-24: Difficulty: _easy_
When a user enters a query into the search box, the query is pre-processed
before it is submitted to the search index. Material for MkDocs will apply the
following transformations, which can be customized by [extending the theme][23]:
following transformations, which can be customized by [extending the theme][24]:
``` ts
export function defaultTransform(query: string): string {
@ -311,7 +407,7 @@ export function defaultTransform(query: string): string {
If you want to switch to the default behavior of the `mkdocs` and `readthedocs`
themes, both of which don't transform the query prior to submission, or
customize the `transform` function, you can do this by [overriding the
`config` block][24]:
`config` block][25]:
``` html
{% extends "base.html" %}
@ -331,19 +427,19 @@ customize the `transform` function, you can do this by [overriding the
The `transform` function will receive the query string as entered by the user
and must return the processed query string to be submitted to the search index.
[22]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/transform/index.ts
[23]: ../customization.md#extending-the-theme
[24]: ../customization.md#overriding-blocks-recommended
[23]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/transform/index.ts
[24]: ../customization.md#extending-the-theme
[25]: ../customization.md#overriding-blocks-recommended
### Custom search
[:octicons-file-code-24: Source][25] ·
[:octicons-file-code-24: Source][26] ·
:octicons-mortar-board-24: Difficulty: _challenging_
Material for MkDocs implements search as part of a [web worker][26]. If you
Material for MkDocs implements search as part of a [web worker][27]. If you
want to switch the web worker with your own implementation, e.g. to submit
search to an external service, you can add a custom JavaScript file to the
`docs` directory and [override the `config` block][23]:
`docs` directory and [override the `config` block][24]:
``` html
{% block config %}
@ -361,8 +457,8 @@ format using _discriminated unions_, i.e. through the `type` property of the
message. See the following interface definitions to learn about the message
formats:
- [:octicons-file-code-24: `SearchMessage`][27]
- [:octicons-file-code-24: `SearchIndex` and `SearchResult`][28]
- [:octicons-file-code-24: `SearchMessage`][28]
- [:octicons-file-code-24: `SearchIndex` and `SearchResult`][29]
The sequence and direction of messages is rather intuitive:
@ -371,7 +467,7 @@ The sequence and direction of messages is rather intuitive:
- :octicons-arrow-right-24: `SearchQueryMessage`
- :octicons-arrow-left-24: `SearchResultMessage`
[25]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker
[26]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
[27]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker/message/index.ts
[28]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/_/index.ts
[26]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker
[27]: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
[28]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/worker/message/index.ts
[29]: https://github.com/squidfunk/mkdocs-material/blob/master/src/assets/javascripts/integrations/search/_/index.ts

View File

@ -219,4 +219,5 @@ nav:
- Blog:
- blog/index.md
- 2021:
- blog/2021/excluding-content-from-search.md
- blog/2021/search-better-faster-smaller.md