org-zettel-ref-mode

中文版说明

Main Features

Invoke the command `M-x org-zettel-ref-init` to open the “Overview Window,” displaying the notes recorded in the original document and the marked text.

1. Each time an overview is generated, a literature note is automatically created and saved to a folder of your choice.
2. Quick note-taking: use `M-x org-zettel-ref-add-quick-note` to directly input notes.
3. When reviewing literature notes, you can jump directly back to the corresponding location in the original text to reread the context.
4. Offers a method to convert documents in other formats into org format.
5. Provides quick markup functionality to easily add bold, italic, underline, and other formatting to text in the source file.
6. Supports integration with knowledge management tools like org-roam and denote.
7. Flexible file association mechanisms that support multiple knowledge management modes (Normal, Denote, Org-roam).
8. Directly call external Python scripts from within Emacs to convert various document formats into org files.

Demo

As shown, the left window displays the original text, while the right window displays the overview.

The overview windows contains that marked text and notes from original text, and it will save as a note file standalone.

Applicable Scenarios

`org-zettel-ref-mode` is only effective when org-mode is activated:

1. Directly targeting org files
2. Other user-defined text formats processed in org-mode, such as md, txt, etc. In these cases, the functionality of the major mode for those formats may be affected.

However, I generally convert materials directly into org format for saving, so the second scenario is rare.

Value: A Reading Method That Balances Breadth and Depth

TL;DR Version:

• Simply saving, excerpting, or copying materials is not enough; information needs to be processed to be transformed into useful knowledge.
• The Zettelkasten method emphasizes summarizing/reviewing in your own words and establishing connections, providing multiple opportunities for information processing. However, many introductions overlook Luhmann’s method of handling a large volume of literature notes.
• Literature notes are an efficient and in-depth method that records key points and inspirations, facilitating quick review and deep reading, while also helping distinguish between existing and new information.

Full Version:

As a longtime note-taking enthusiast and writer, I’ve gradually realized some “counterintuitive” insights:

• Simply saving is almost useless.
• Simply excerpting is almost useless.
• Simply copying is almost useless.

The reason is that merely transporting material only increases the amount of information without reprocessing it. Remember the classic hierarchy? Data -> Information -> Knowledge -> Wisdom.

The Zettelkasten method always emphasizes summarizing in your own words, frequently reviewing past notes, and increasing the connections between notes. From a methodological standpoint, it offers at least 4-7 opportunities for information processing.

Even so, the literature and videos introducing the Zettelkasten method often get caught up in the craze of double-linking, falling into the trap of merely saving data—essentially ignoring the method Niklas Luhmann used to handle a massive amount of literature notes.

Let me share a number: among the more than 90,000 index cards Luhmann left behind, over 10,000 were literature notes.

Luhmann’s astounding productivity came from an exaggerated amount of data processing, and behind that was his efficiency in handling this data—achieved through the creation of literature notes.

Luhmann had a habit of taking literature notes while reading. His books or materials had no underlining, no margin notes, and were incredibly clean, almost as if they hadn’t been read. Each literature note was essentially an index of the material. He only excerpted the original text from the book when absolutely necessary.

However, after understanding how researchers create literature notes, I discovered that Luhmann’s literature notes are almost identical to standard research literature notes. They are also annotated in one’s own words, while recording the specific location of inspiration in the paper, for future in-depth reading.

In other words, this method of taking literature notes balances efficiency and depth.

When it’s unnecessary to deeply understand a material, literature notes can record key points (not the important content, but the insights useful to oneself). When a deep understanding is needed, the literature notes can quickly point to the corresponding context for in-depth reading and thinking, without wasting time re-reading from the beginning.

Besides balancing efficiency and depth, literature notes also have the advantage of easily distinguishing between existing and new information. Concepts or key points that have been annotated similarly before are existing information, and it is unnecessary to annotate them again when encountered in another material. Conversely, concepts, data, or ideas that have not been encountered before are worth annotating and recording their sources, making the discovery of new knowledge easier.

BTW:

A good intro about Zettelkstan: Introduction to the Zettelkasten Method

Installation

Installation Steps

1. Download the `org-zettel-ref-mode.el` file.
2. Place the file in your Emacs load path (e.g., `~/.emacs.d/lisp/`).
3. Add the following to your Emacs configuration file (such as `~/.emacs` or `~/.emacs.d/init.el`):

Example Configuration:

(use-package org-zettel-ref-mode
  :ensure nil
  :load-path "~/Documents/emacs/package/org-zettel-ref-mode/"
  :init
  (setq org-zettel-ref-overview-directory "~/Documents/notes/source-note/")
  :config
  (setq org-zettel-ref-mode-type 'denote)  
  ;; (setq org-zettel-ref-mode-type 'org-roam)  
  ;; (setq org-zettel-ref-mode-type 'normal)  
  (setq org-zettel-ref-python-file "~/Documents/emacs/package/org-zettel-ref-mode/convert-to-org.py")
  (setq org-zettel-ref-temp-folder "~/Documents/temp_convert/")
  (setq org-zettel-ref-reference-folder "~/Documents/ref/")
  (setq org-zettel-ref-archive-folder "/Volumes/Collect/archives/")
  (setq org-zettel-ref-debug t)
  )

Basic Usage

Custom Note Saving Modes

(Updated 2024-08-29) org-zettel-ref-mode provides three modes: normal, org-roam, and denote, allowing note files to be saved in the corresponding format. For example, after selecting org-roam mode, the saved note files will automatically include an ID, making them easier to retrieve.

Configuration Method:

(setq org-zettel-ref-mode-type 'normal) ; Options: 'normal, 'denote, 'org-roam)

Activating the Mode

In any org-mode buffer, run: `M-x org-zettel-ref-init`

Clean Up <<>> in Source Files

Since the core functionality of adding notes involves adding <<>> target links in the original text, many materials converted to org format come with a lot of <<>> text.

Before annotating or marking text in the org file for the first time, you can use `org-zettel-ref-clean-targets` to clean up the format and ensure the quick note feature works correctly.

Adding Quick Notes

1. Place the cursor where you want to add a note
2. `M-x org-zettel-ref-add-quick-note`
3. Enter the note name and content

Quick Markup

1. Select the text in the source file
2. `M-x org-zettel-ref-quick-markup`
3. Choose the markup style you prefer

Sync Overview Files

Automatic sync by default: Automatically runs when saving the source file. Manual sync: `M-x org-zettel-ref-sync-overview`

Manage Source Files

1. Launch Panel

M-x org-zettel-ref-list

Reminder: The following commands are all executed within the panel interface.

1. Rename Source File (“r”)

M-x org-zettel-ref-list-rename-file

Rename according to the fixed format AUTHOR__TITLE==KEYWORDS.org.

1. Edit/Add Keywords (“k”)

M-x org-zettel-ref-list-edit-keywords

Independently add one or more keywords to the source file.

1. Delete Source File

Delete a single file (“d”) M-x org-zettel-ref-list-delete-file

Delete multiple files (“D”) Press “m” in the list to mark multiple files, then execute M-x org-zettel-ref-list-delete-marked-files

If the marked files are incorrect, press “u” to clear the marked status, and press “U” to clear all marked statuses.

1. Use Filters

Simple Filter (“/ r”): Use Author, Title, Keywords as filter conditions, only one filter condition can be applied at a time M-x org-zettel-ref-filter-by-regexp

Complex Filter (“/ m”): Multiple filter conditions can be applied using Author, Title, Keywords as conditions

⚠️ Caution

1. Do not casually change the filename of note files. If you do, adding quick notes/markups again in the source file will generate duplicate notes during sync.

Advanced Features

Custom Text Marking Types and Highlight Styles

Reference the following example:

(setq org-zettel-ref-highlight-types
      (append org-zettel-ref-highlight-types
             '(("warning" . (:char "w"
                           :face (:background "#FFA726" 
                                 :foreground "#000000" 
                                 :extend t)
                           :name "warning"
                           :prefix "⚠️"))
               ("success" . (:char "s"
                           :face (:background "#66BB6A" 
                                 :foreground "#FFFFFF" 
                                 :extend t)
                           :name "success"
                           :prefix "✅")))))

Highlight type configuration. Each type should include:

• :char Single character identifier for the type
• :face Face attributes for highlighting
• :name Display name for the type
• :prefix Symbol shown in the overview

File Association Mechanism

org-zettel-ref-mode now supports multiple file association mechanisms and no longer fully relies on the “-overview” suffix in filenames:

• Normal Mode: Still uses the “-overview” suffix (for backward compatibility).
• Denote Mode: Follows Denote’s naming conventions.
• Org-roam Mode: Follows Org-roam’s naming conventions and ID attributes.

If you’re upgrading from an older version, your existing “-overview” files will still work. However, for new files, we recommend using the new association mechanisms.

Debugging in org-roam Mode

The `M-x org-zettel-ref-check-roam-db` function checks the status of the org-roam database.

Custom Overview File Location

(setq org-zettel-ref-overview-directory "~/my-notes/overviews/")

Adjusting Auto-Sync Behavior

Disable Auto-Sync:

(org-zettel-ref-disable-auto-sync)

Enable Auto-Sync:

(org-zettel-ref-enable-auto-sync)

Enabling Debug Mode

If you encounter issues during use, you can enable debug mode to get more information:

(setq org-zettel-ref-debug t)

Using Scripts to Convert Documents in PDF, ePub, HTML, MD, TXT Formats to Org Files

Script: convert-to-org.py

org-zettel-ref-mode now supports directly calling external Python scripts from within Emacs to convert various document formats into org files.

Key Features

1. Multi-format Support:
   • Supports converting PDF, EPUB, HTML, Markdown, and TXT formats to Org format.
   • Can handle both electronic and scanned PDFs, supporting mixed Chinese and English documents.
2. OCR Functionality:
   • Uses OCR technology to process scanned PDFs, supporting Chinese and English recognition.
3. File Management:
   • Automatically checks file size to prevent processing overly large files.
   • After conversion, it can automatically archive the source file.
4. Flexible Configuration:
   • Supports custom paths for temporary files, reference materials, and archives.
   • You can choose to use the system Python, Conda environment, or virtual environment.

Usage Instructions

1. Configure Python Environment:

   (setq org-zettel-ref-python-environment 'conda)  ; or 'system, 'venv
   (setq org-zettel-ref-python-env-name "your-env-name")  ; If using Conda or venv
       

2. Set Script Path and Folders:

   (setq org-zettel-ref-python-file "~/path/to/document_convert_to_org.py")
   (setq org-zettel-ref-temp-folder "~/Documents/temp_convert/")
   (setq org-zettel-ref-reference-folder "~/Documents/ref/")
   (setq org-zettel-ref-archive-folder "/Volumes/Collect/archives/")
       

3. Run Conversion Script: Use the command `M-x org-zettel-ref-run-python-script` to execute the conversion.

⚠️ Caution

• Ensure that all necessary Python libraries (e.g., PyPDF2, pdf2image, pytesseract, etc.) are installed.
• For scanned PDFs, the conversion process may be slow, and the results may not be as good as for electronic versions.
• It’s recommended to use this script primarily for converting electronic PDFs, EPUB, Markdown, and TXT documents.

Workflow Recommendations

1. Use a browser extension (e.g., Markdownload) to save web pages as Markdown files.
2. Use org-zettel-ref-mode’s Python script to convert Markdown files to Org format.
3. For audio files, you can first convert them to text using Whisper and then use the script to convert them to Org format.

This feature significantly expands the application range of org-zettel-ref-mode, making it a more comprehensive knowledge management tool.

⚠️ Caution

It is recommended to use this script for converting ePub, markdown, txt, and electronic PDF documents.

It is not recommended to use this script to convert scanned PDFs due to slow conversion speed and suboptimal conversion quality.

Available Commands

Here are the main commands provided by org-zettel-ref-mode:

• `M-x org-zettel-ref-init`: Initialize org-zettel-ref-mode, create or open an overview file
• `M-x org-zettel-ref-add-quick-note`: Add a quick note at the current position
• `M-x org-zettel-ref-sync-overview`: Manually sync the overview file
• `M-x org-zettel-ref-quick-markup`: Quickly add markup to selected text
• `M-x org-zettel-ref-clean-multiple-targets`: Clean up excess targets in the source file
• `M-x org-zettel-ref-enable-auto-sync`: Enable auto-sync
• `M-x org-zettel-ref-disable-auto-sync`: Disable auto-sync
• `M-x org-zettel-ref-check-roam-db`: Check org-roam database status
• `M-x org-zettel-ref-run-python-script`: Run the specified Python script

Configurable Variables

Here are the main configurable variables for org-zettel-ref-mode:

• `setq org-zettel-ref-overview-directory “~/org-zettel-ref-overviews/”`: Set the overview file storage directory
• `setq org-zettel-ref-mode-type ‘normal`: Set the mode type (options: ‘normal, ‘denote, ‘org-roam)
• `setq org-zettel-ref-include-empty-notes nil`: Set whether to include empty quick notes
• `setq org-zettel-ref-include-context nil`: Set whether to include more context in the overview
• `setq org-zettel-ref-quick-markup-key “C-c m”`: Set the shortcut key for quick markup
• `setq org-zettel-ref-python-environment ‘system`: Set the Python environment type (options: ‘system, ‘conda, ‘venv)
• `setq org-zettel-ref-python-env-name nil`: Set the Python environment name
• `setq org-zettel-ref-python-file “~/path/to/script.py”`: Set the Python script file path
• `setq org-zettel-ref-temp-folder “~/Documents/temp_convert/”`: Set the temporary folder path
• `setq org-zettel-ref-reference-folder “~/Documents/ref/”`: Set the reference materials folder path
• `setq org-zettel-ref-archive-folder ”Volumes/Collect/archives”`: Set the archive folder path
• `setq org-zettel-ref-debug nil`: Set whether to enable debug mode
• `setq org-zettel-ref-overview-width-ratio 0.3`: Set the overview window width ratio
• `setq org-zettel-ref-overview-min-width 30`: Set the overview window minimum width
• `setq org-zettel-ref-highlight-types`: Set text marking types and highlight styles
• `setq org-zettel-ref-overview-image-directory=”~/Documents/org-zettel-ref-images/”`: Set the image save path for overview notes

FAQ

Q: How do I use org-zettel-ref-mode across multiple projects? A: You can set different overview directories for each project, dynamically changing the value of `org-zettel-ref-overview-directory` when switching projects using `let-bound`.

Q: What should I do if the overview file becomes too large? A: Consider splitting the overview file by topic or time period. You can customize the `org-zettel-ref-create-or-open-overview-file` function to achieve this.

Q: How do I back up my notes? A: Include both the source files and overview files in your version control system (e.g., Git). Additionally, regularly perform file system-level backups.

Q: How can I check the status of the org-roam database? A: You can use the `M-x org-zettel-ref-check-roam-db` command to check the status of the org-roam database, including version information, number of nodes, etc.

Troubleshooting

If you encounter issues:

1. Ensure you are using the latest version of org-zettel-ref-mode.
2. Check your Emacs configuration to ensure there are no conflicting settings.
3. Try to reproduce the issue in a clean Emacs configuration (`emacs -q`).
4. Check the `*Messages*` buffer for any error messages.
5. If the issue is related to the Python script or Conda environment, check your Python environment configuration.
6. Enable debug mode (set `org-zettel-ref-debug` to `t`) to get more detailed log information.

If the issue persists, please submit an issue on the GitHub repository, including a description of the problem, steps to reproduce it, and debug logs.

Contributions

We welcome community contributions! Here are some ways you can get involved:

• Report bugs or suggest new features.
• Submit patches or pull requests.
• Improve documentation or write tutorials.
• Share your experiences and tips for using org-zettel-ref-mode.

Changlog

Version 0.5 (2024-11-12)

• Upgrade: Major upgrade to marking and note system (see #Demo for changes after upgrade)
  • Decoupled from org-mode’s built-in styles
  • Automatic note ID numbering
  • Automatic highlighting of marked content
  • Content under overview headlines won’t be cleared
  • Mark images and sync them to overview notes
    • Must run org-zettel-ref-add-image command to add images to overview notes
    • Requires setting org-zettel-ref-overview-image-directory configuration
  • Overview note style upgrades:
    • Note titles now display note IDs
    • Uses org-mode Headlines style
    • Note icon prefixes to distinguish note types
  • New custom configuration options (customize text marking types and highlight styles, see #Advanced Features):
    • org-zettel-ref-highlight-types defines/adds marking types and highlight styles
    • org-zettel-ref-overview-image-directory defines image save path for overview notes
  • Painless upgrade, maintains familiar commands
  • Note: When executing org-zettel-ref-mark-text, please don’t select note type or image type
    • For quick notes, continue using the previous org-zettel-ref-add-quick-note command
    • This design choice is to provide highlight styles for quick notes and image note

Version 0.4.4 (2024-11-09)

• Fixed:
  • The issue where org-zettel-ref-watch-directory reports an error after running org-zettel-ref-list-rename-file

Version 0.4.3 (2024-11-08)

• Optimized:
  • The display method of the overview file window.
    • Added a configuration item to define the width of the overview window: org-zettel-ref-overview-width-ratio, default 0.3
    • Added a configuration item to define the minimum width of the overview window: org-zettel-ref-overview-min-width, default 30

Version 0.4.2 (2024-11-08)

• Fixed:
  • The error in org-zettel-ref-db-init #15
  • The issue where the cursor position is lost after executing org-zettel-ref-init
  • The issue where the overview file failed to synchronize correctly due to improper index file retrieval

Version 0.4.1 (2024-11-06)

• Optimized convert_to_pdf.py
  • Dropped using OCR to convert PDF

Version 0.4 (2024-11-04)

• Attention!
  • If you’ve previously used org-zettel-ref-mode, you need to run M-x org-zettel-ref-migrate the first time you use the new version to upgrade the data structure in the hash table.
• New Feature: Provides a visual management panel for source files
  • org-zettel-ref-list (see Basic Usage -> Manage Source Files for details):
    • Visualization: Provides a reference management panel
    • Multi-column list: Displays the current references in a list format, with key columns such as Title, Author, and Keywords
    • Rename: Allows renaming files in the format AUTHOR__TITLE==KEYWORDS.org within the panel
    • Sorting: Click on the column name to sort the list alphabetically
    • Filtering: Filter source file entries by conditions, such as Author, Title, or Keywords. Currently, only one condition can be filtered at a time.
• Upgraded the data structure of the hash table in org-zettel-ref-db.el
• Upgraded org-zettel-ref-clean-multiple-targets
• Fixes:
  • Restored the accidentally deleted custom configuration item org-zettel-ref-debug
• Reminder:
  • Due to the upgrade of the hash table storing the mapping between source files and overview files to version 2.0, the following functions are deprecated:
    • org-zettel-ref-check-and-repair-links, org-zettel-ref-maintenance-menu, org-zettel-ref-refresh-index, org-zettel-ref-rescan-overview-files, org-zettel-ref-status.

Version 0.3.3 Stable release (2024-09-28)

• Backend optimizations to further enhance code robustness, modularity, and improve plugin stability.
• Fixed an issue in version 0.3.2 where rapid updates to the overview file caused synchronization errors with quick notes and marked text.
• Fixed an issue in version 0.3.2 where the file naming strategy led to frequent re-creation of overview files in Denote mode.
• Fixed an issue in version 0.3.2 where the overview file failed to synchronize correctly due to improper index file retrieval.

After this period of development, the code for org-zettel-ref-mode has finally become modular and robust. No new features will be introduced before version 0.4. Instead, the focus will be on further componentization of the code and providing more customization options.

Version 0.3.2 (2024-09-24)

• Improved compatibility with Org-roam v2: Update the record of literature notes in the overview file to the Org-roam database.
• Improved file naming
• Fix a bugs that causes Emacs crash
• Refined code, modularized

Version 0.3.1 (2024-09-15)

• Compatible with Emacs 30 and later versions.
• Overview files now have more elegant names, reducing repetitive occurrences of the word “overview.”
• Fixed an intermittent (setq, 5) error.
• Removed the dependency on conda.el in org-zettel-ref-mode.el, and the detection of the Python environment is now entirely handled by convert-to-org.py.
  • Automatically sets up a virtual environment via the python venv command and installs required libraries.
    • Note: After updating to this version, running convert-to-org.py will reinstall third-party libraries. If you prefer a clean environment, you may want to manage this manually.
• Improved synchronization mechanism for overview files, preventing multiple overview files from being created for the same source file. Also improved the robustness and stability of this feature.
  • A hash table is now used to map source files to overview files. One great thing is that you don’t need to manually set the hash table file location.
  • The overview file header now includes a new property block: #+SOURCE-FILE: to confirm the mapping.
  • New commands:
    • org-zettel-ref-check-and-repair-links - Check and repair links between source files and overview files.
    • org-zettel-ref-maintenance-menu - Display a menu for org-zettel-ref-mode maintenance operations.
    • org-zettel-ref-refresh-index - Manually refresh the overview index.
    • org-zettel-ref-rescan-overview-files - Rescan the overview directory and update the index.
    • org-zettel-ref-status - Display the current status of org-zettel-ref-mode.

Version 0.3 (2024-09-03)

• Improved integration with org-roam:
  • Added conditional loading and error-handling mechanisms for better stability
  • Optimized database operations for increased efficiency
  • Enhanced file handling for greater compatibility
  • Added a database status check feature for easier debugging
• Enhanced support for Conda environments:
  • Provided more flexible Python environment configuration options
  • Improved the initialization and activation process for Conda environments
• Refined logic for filename generation and processing:
  • Added a filename cleanup feature for greater robustness
  • Optimized file naming strategies across different modes
• Optimized overview file synchronization:
  • Implemented selective updates, only refreshing changed sections
  • Improved buffer handling to reduce file I/O operations
  • Enhanced content generation for increased efficiency
• Added debugging features:
  • Included detailed log output for easier troubleshooting
  • Provided more error messages and status check options

Acknowledgments

org-zettel-ref-mode was inspired by my friend @lijigang’s org-marked-text-overview. Due to extensive modifications, I decided to release it separately as org-zettel-ref-mode after discussing it with him.

Future Plans

• ✅ Improve performance, optimizing handling of large files
• ✅ Integrate with other knowledge management packages, such as org-roam and denote
• Support more file formats (possibly)
• ✅ Further optimize Python script integration
• Add more customization options
• Optimize file association mechanisms, reducing reliance on specific filename suffixes

If you like it, please Star.