Markdown

[dazappa]

Here's something I know will not change, but I still want to complain about anyway.

Markdown is great. Markdown for a novel is annoying, for one reason: paragraphs. Pressing Enter twice to create a paragraph is nothing more than annoying when you come from a traditional word processor like LibreOffice Writer, Microsoft Word, or even the beloved wordgrinder. I have 78 paragraphs in one chapter; this adds up quick over time.

Markdown is also more difficult to "get right" for a new user. For example, having one chapter with multiple scenes self-contained (because I find a standalone document with just a title annoying)

Incorrect

## *Example
### Hungry
% synopsis: I am hungry and eat food
I grab an apple from the table and eat it.
### Sleepy
% synopsis: I am tired and go to bed
After yawning, I crawl into bed.

Correct

## *Example

### Hungry
% synopsis: I am hungry and eat food
I grab an apple from the table and eat it.

### Sleepy
% synopsis: I am tired and go to bed
After yawning, I crawl into bed.

If you have incorrect Markdown syntax, an exported document will also be incorrect. At least with items such as headings, bold, italics, and strikethrough there is some immediate feedback whether or not you did things correct. In the above incorrect example, the content from the two scenes will be joined together. This is an easy mistake to make.

Coming from other programs, the necessity to manually add Headings for Chapters and Scenes is a bit confusing. Does a Document really need a different Label from its Heading? The Document's type already shows us if it is a scene or a chapter, why should we have to add the correct heading level and heading to a Document at all?

Limiting editing to Markdown also means we lose some traditional rich text editing features, such as indenting the first line of a paragraph and the spacing between paragraphs. In theory, these could still be added to the Editor (as a global Editor setting, if you don't want per-document settings) so writing would be nicer. Likewise, if you did a complete overhaul of the editor and removed Markdown formatting, Headers, Bold, Italic, and Strikethrough are all still represented and obvious. It would be possible for the storage format to still be Markdown, but this would almost certainly mean you would have to make some RTF -> Markdown converter.

Mostly I think having to learn a new formatting "language" will put the general writer with limited tech background experience off trying to use novelWriter.

And for myself, as silly as it sounds, I am thinking of not using novelWriter simply because I don't want to press enter twice all the time.

[vkbo]

It's perfectly fair to not want to use markdown, but then novelWriter obviously isn't the right app. In its first iteration, it was a WYSIWYG editor doing more or less what you describe, and I wasn't happy with it. I thought I may as well use Libre Office then, or FocusWriter, which is an excellent distraction-free editor for single documents. GhostWriter is also great for single-document full markdown support. novelWriter is filling a gap in-between these, and drawing some inspiration from yWrite which is just plain text, as well as Zim Wiki.

In any case, the file layouts are mainly there so you can tag the documents so you know what they're for. They have some effect on exports too, but mostly related to text alignment and page breaks on title pages.

The GUI design draws inspiration from other apps, like Zim. It has the same logic with document label and title. Since you can have multiple titles in a document, or none at all, they cannot be automatically the same. Zim does the same. I often have different labels on the documents and the titles within. I've always thought of them as file names and titles with no one-to-one relation. Although I am considering making an optional link between label and first line of the document. Zim does the same.

The reason you can split the scenes into separate files is that you can then freely move them around. Same with chapters. You may want to move the chapter separators around between scenes. You don't have to use this structure, but that is the point of it.

Limiting formatting is also a "feature, not a bug". It's a distraction from writing, and distraction-free writing is a key point of novelWriter. I'm adding additional formatting options to the build tool instead. Adding an option to generate paragraph indenting is on my list. It's coming when I start on the manuscript export feature, where it is needed.

In one of the recent version of Qt5, markdown support was added to the QTextEdit widget, so it may not be a huge amount of work to make it a rich editor with markdown in the background. The tag and comment syntax could still be added as the tags are designed to not conflict with standard markdown syntax. I may look into that at some point, and just let the user choose which mode to use the editor in.

[vkbo]

Regarding Headings and file type, at the end of the day I think you could just commit to a decision. Either force certain things to be separate (every chapter and every scene in its own file) or allow everything to be together (as it is currently).

I definitely don't want to do that. Personally, I start by writing my draft scenes, in terms of headings and a paragraph, in one file per act, and when I've established the plot breakdown, split that file into individual scenes. I don't add chapters at all until much later in the process. I know other people use a similar work flow. That flexibility is an important feature.

[dazappa]

Your notes about the Layout are fair. I got too focused on chapters and scenes.

I could in principle just merge scene, book and chapter into a layout named "story", and instead use different icons for them based on the level of the first heading within. I've considered dropping these layouts for a while. They've been there since I first started working on novelWriter, and were supposed to affect formatting more than they ended up doing in the end.

I believe I have changed my mind. You can probably leave Layout alone, or make the change in the above quote.

Forcing chapters and scenes to be structural and limiting a scene Layout to a single scene is more of the yWriter approach. Your current flexible method of multiple chapters / scenes in a file is more of the Scrivener approach. I don't have a strong preference either way, just throwing out ideas.

[vkbo]

Oh, absolutely. I love this kind of feedback. There are numerous feature ideas that have come out of such discussions. This got me re-thinking the layout functionality, which I haven't done in a long while. The current layout structure is basically a bit out of date with the current functionality. Even the code works around it by merging the layouts internally. I have just avoided changing it front-end because it would imply forcing a conversion of people's projects. But I think I can find a way to do it without any major impact.

It helps a lot that I now have an index class that contains a lot of the information that the earlier versions of novelWriter expected the user to provide. For instance, the structure of headers within the file. The early versions didn't care about those until tokenization before export.

Anyway, I think I'll make that merge, and made issue #614 for it. I expect to put it into version 1.2 together with proper Open Document export, since they both involve a lot of work on the build tool.

[peter88213]

There is something enlightening to me about this discussion. When I'm not making much progress with my novel writing, I look for a better tool, a writing software that supports my way of thinking and working in a perfect way. After a long search, I then decide to program my own solution that fits everything.

Then I enter the phase that we know from professional life as process analysis and optimization. I come up with an architecture that is supposed to translate my way of working creatively into data structures. All of a sudden, there's a demand for a decision, so for example, like here: how do I break the text down into chapters and scenes? Do I demand a fixed file structure that is technically easier to process, or do I allow for a more sketchy approach in which structural information is loosely inserted into the running text as I write?

For both approaches, there are already a ton of solutions that you just need to download and install. But they all don't really fit, so I'm doing my own thing. In fact, I want to be able to do both: Set markup and chapter and scene separators in the writing flow, and then have a clear directory tree with separate files, metadata display, statistics, logging, navigation, drag/drop, etc. when revising.

So I construct a hybrid that glues things together using custom markup, multiple files, indexes and tokenizers. This requires conventions, additional data, background processes, synchronization, which in turn needs to be configured. Programming-wise rather suboptimal, and not exactly easy to explain to others, but it does indeed promise to support my creative writing process, so it's worth the effort.

After writing and editing, it's time to publish. After all, I want standard pages, EBooks, PDF in book format, and so on. There are plenty of solutions for that too, but now I've already started, so I'm integrating such features as well, and exposing myself to public discussion.

In this way, I reflect on my way of thinking and working, I get excited, solve problems, see success. There's only one drawback to all this: I'm still not making any progress with my novel writing now ...

Don't get me wrong, I admire your work, you really put something great together. And the example of yWriter shows how Simon can develop his quite popular software and publish a lot of novels at the same time. I've been following the development of yWriter for quite a while, and I think the secret of Simon's success is that he's pretty consistent with his concept, and rather cautious about feature requests. And most importantly: yWriter is closed source, no one else can tamper with it.

As a yWriter user, I have adjusted to this and created a superstructure with Python scripts for missing features that seem important to me. That seemed easier than starting my own writing software project. However, now I am considering switching from Windows to Linux, and unfortunately yWriter is not really made for that. That's how I came across novelWriter. At the moment I'm also thinking of writing a very own novel software that basically manages structure and metadata, while writing, layouting and publishing are outsourced to Libre/OpenOffice and QuarkXpress.

At the moment, I see two options: Build the program from the bottom up, possibly based on tkinter, or strip down a nice and powerful solution like novelWriter to the basic functions needed and retrofit it to the yWriter file format.
What do you think, can you give me some advice on this?

Keep up the good work!
Cheers,
Peter

[vkbo]

Making software for writing is definitely a distraction from writing. Since I don't depend on writing for an income, it is not really a big issue, and since I get my income from writing code, having this and other projects has a net gain in terms of experience.

Personally, I'm torn between plain text and rich text for writing text. I've written several novels worth of text using plain text formatting like Markdown, restructuredText and LaTeX. Both technical and scientific work. I am very comfortable with this approach, but I also tend to spend a lot of time tinkering with how the rendered text looks to get it just right (it is for instance why I wrote my own ODT exporter class rather than use the one provided by the Qt framework).

I too tested a lot of tools for writing fiction, and never found one that was just right. I ended up using Markdown or LaTeX instead, but wanted the ability to easily reorganise the elements of the story, and track meta information and notes about characters, etc. At first, I started making a sort of compiler that would build a novel and various visualisations from a set of Markdown files. I quickly realised it would be rather clunky, so I instead decided to wrap a GUI around it. That is novelWriter. I also discussed over many lunch breaks how to build such a tool with one of my former colleagues who was writing a huge novel in LaTeX source.

Due to how the idea of novelWriter came to be, there are clear limitations on what feature requests can be accepted. I have a sticky issue listing the most common ones I will not take. But I have received a ton of very good suggestions from users that I've implemented because they fit well with the philosophy behind novelWriter. @dazappa here made some very good observations that I have implemented, like dropping the layouts, simplifying the Markdown syntax, and adding the header for new files etc.

Still, there are a bunch of neat requests that I cannot easily accommodate because novelWriter is plain text and not rich text. They can in principle be added by using LaTeX-style markup, and there are Markdown extension that add complex features in similar and complex ways, but they are really too clunky for what I want to do here so I have mostly held off on those changes.

That said, many of the suggestions are good, and I think I have a better idea now of how I want a writer's app to work than I did when I started on novelWriter. Because of this, I am working on a rich text sister app to novelWriter whenever I have the time. With it, I want to avoid some of the unnecessary complexity, and keep the neatest features. We'll see how it goes.

[peter88213]

I am working on a rich text sister app to novelWriter whenever I have the time.

My, you are busy! I actually have to agree with @dazappa about Markdown. Having to enter line breaks twice would be a cutoff criterion for me, too. But this is not a fundamental problem at all, something like this could be made configurable, couldn't it? The only markup I need for prose text is italics and bold. When converting to ODT, I convert this to "emphasize" and "strong emphasize" tags, whereby I then use a character style sheet with small caps for "strong emphasize". Generally the layout in OpenOffice is set by document templates.

The display in the text editor must be clear and eye-friendly, i.e. have unobtrusive coloring, a reasonable font and sufficient line spacing, that's all I need. Markdown does not bother me at all. Not much to ask, is it? But as I've read elsewhere here, the qt textbox doesn't seem to have a line spacing setting.

The basic concept of novelWriter really appeals to me. I especially appreciate the "plot" tree, something like that can be used very nicely for dramaturgical structures, story lines or character arcs. Maybe novelWriter will be my yWriter successor after all, who knows?

I started writing a converter to and from the yWriter format yesterday. With the detour via the yWriter novel metamodel, all other converters and application synchronizers could then be realized effortlessly.

However, I am not very happy with the novelWriter file format. The project name is only visible in the parent directory name and within the "nwProject.nwx" file. When writing back to this format, the directory structure may need to be touched, there is redundant data, and in the middle of it there can be user dropped files lying around. If you then synchronize the whole thing in the cloud, inconsistencies of all kinds can arise.

Well, in daily work such a thing is not so noticeable, but if you want to make a round trip with third party software, the novelWriter format looks rather disadvantageous. What a huge step forward in this respect was the transition from the yWriter 5 format with one file per scene to the yWriter 7 format, where the entire project tree is in a single XML file.

Anyway, I've already learned something useful from novelWriter: I'm currently upgrading my yWriter conversion software so that it can expand the document structure based on part, chapter and scene separators inserted into the text. This is a feature that really helps me with fluent writing using OpenOffice as a text editor.

Cheers,
Peter

[peter88213]

and because I work as a programmer, and expanding my skills is beneficial to me professionally. I haven't done a lot of work in C++.

I completely understand your motivation. However, as far as writing goes, it's not just procrastination, as you mentioned yourself, but there's also a danger that you're getting further and further away from the novelist's point of view when it comes to features, isn't there?

[vkbo]

There is a misunderstanding again, I'm afraid. I am not concerned here with rich text features that affect the exported document, but with the organized presentation during writing.

First line indents in the exported document should generally be controlled by paragraph style sheets. This is purely a matter for the exporter.

Then there is no misunderstanding. That's what I'm talking about. Exports are text snapshots, which is easy as there is nothing to keep up to date. Same for the document viewer panel. It's static content. It's the dynamic nature of the editor itself that is the challenge. The features we're discussing are rich text editor features. The editor is plain text, and so are the files that it saves the data to. I.e., the .nwd files.

Paragraph indents are already a feature request for exports in #906, and on my todo list.

a proper rich text editor (I think that's what yWriter did too at some point?)

Yes, as far as I know, yWriter uses a modified dotnet rich text box that internally works with RTF format. Very tricky about character encoding, as the history shows. Bedides, RTF is quite horrible to convert.

I wrote a lot of code with Visual Basic 6, before .NET, and have done battle with RTF. Not a favourite. I quite like the ODF standard. I'm considering adapting a subset of the text standard for my Collett document format. I cannot use it directly as I need more data in my files, but it would make it a lot easier to support ODT exports.

But don't get me wrong: I myself am not interested in richt text. As I mentioned earlier, I only need bold and italic for prose text, and I can enter that with Markdown. All the rest, like line alignment, I think belongs in the styles applied during or after export.

The alignment and indent feature is mostly there to allow the author to block quote text. It's useful for citing letters, speeches, quotes, signs, whatever. The novelWriter export is only intended to produce a manuscript anyway, not a print-quality document. But it's useful to have these things stand out in the manuscript as well.

For notes, I use Zim, if only for the diary. And if I'm not mistaken, there's even a plugin for drawing tables for it.

Zim is excellent. There are a bunch of details in novelWriter inspired by Zim actually. I only stopped using Zim because I felt it would never reach a polished final product, so I switched to Joplin.

As soon as I start adding loops over content in the editor, the processing time starts running into several minutes.

Is there no possibility to catch the "Enter" key event or do some auto-replacing? However, this wouldn't work with copy/paste, so at least the clipboard should be taken care of.

I already capture every keystroke. so that's not an issue. It's how the auto-replace feature is implemented, among other things.

Actually, the block format (i.e. paragraph format) is inherited from the previous block automatically. The initial document can be populated one block at a time, which is slow, and much slower the when I let the C++ library handle it internally like now, but that too is doable. The main issue is to maintain correct formatting on existing text when block types change (heading, meta data, comments, etc) and with copy/paste as you mention. But the real tricky one is actually undo/redo. In order to automatically apply formatting I have to replace the undo/redo stack with a custom implementation because any formatting change goes onto the undo stack regardless of whether the user does it or code does it. That's one more reason why a hybrid plain/rich editor is tricky.

If the syntax highlighter could apply a shadow block format, all would be good. The syntax highlighter is a tool stacked on top of the document itself, not the editor, and it alters the character format of each character in a block when a block is changed. This character format is not actually a part of the document, so it doesn't affect the undo/redo stack. It is merged with the "real" format when the text is rendered. This is where the spell checking happens as well. Otherwise you could undo the squiggly line under a misspelled word. While you can access the underlying block format from the syntax highlighter, you cannot apply a shadow format like for the text itself. It's one of the many approaches I've tried. It would be nice if Qt would add it.

[vkbo]

I completely understand your motivation. However, as far as writing goes, it's not just procrastination, as you mentioned yourself, but there's also a danger that you're getting further and further away from the novelist's point of view when it comes to features, isn't there?

Well, I do use my app for writing, although writing is just a hobby. A bunch of features I've added come directly from using the app for writing fiction, and discussing it with other writers both online and offline. I do have some guiding principles too. If a feature isn't strictly for writing prose, they will not be included. I've gotten plenty of requests for features for technical writing, screen writing, pretty document formatting, etc that I've rejected because they're out of scope.

Even when considering a rich text editor, the formatting scope will be narrow. I don't plan to allow for any more formatting features than are in novelWriter now. You would just be able to emphasise and align text without having to add formatting codes. The paragraph layout would of course conform to rich text editing norms rather than plain text. I also want to add some meta data features that cannot be easily implemented in plain text, like footnotes and highlighting + writer's notes.

Although I'm not a huge fan of the current minimalist UI trend, especially when it actually becomes a limitation and constant annoyance for getting work done, I still don't like bloat and over-engineered apps. I'm aiming for a middle ground, and staying on target.

[peter88213]

Alright, I realize that the idea with the tab inserted doesn't lead further.

As for yWriter, in version 5 the editor box had sort of direct access to the scenes stored in single RTF files. Now, in the yWriter 7 format, the scene content is stored in a proprietary yet easy to convert tagged format, and converted when loaded to/saved from the RTF editor box. This works pretty well and leads to a neat file format. Unlike Markdown, Simon's format has different opening and closing tags, allowing conversion without a special parser or regular expressions.

Thus, it can be said that yWriter actually uses a rich text editor, but its features are mainly for appearance while writing. For users who are used to wysiwyg, this is sometimes irritating, e.g. when they do not find the set font or line spacing in the exported document. For me, however, it is just the right thing.

Since yWriter's default output format for manuscripts is RTF (there are also others for ebooks and latex, but I don't use them) , I have written my own converters to e.g. ODT and ODS, which I have put online as Open/LibreOffice extensions. So you can load yWriter documents comfortably from within LibreOffice. Now that I have written a novelWriter/yWriter converter, it would be easy to set up a LibreOffice extension for importing novelWriter projects as well. However, such an extension would not support all formatting options offered by novelWriter. Providing and maintaining such an extension would probably not be worthwhile due to the low demand.

[peter88213]

indent feature is mostly there to allow the author to block quote text. It's useful for citing letters, speeches, quotes, signs, whatever.

This is a good point. In the meantime, I have included this feature in my extension. However, I implemented an extra OpenOffice macro to format paragraphs prefixed with ">" on demand. This allows the prefixes to be preserved for re-export to yWriter.