[Lazarus] Code Structure / SourceEdit and SyneEdit [Re: Mouse Link in SynEdit (only link-able items)]

[Martin Friebe]

There are several meanings by the word "view" I have tried to put some 
clarification into this. Having finished this mail, I thought I copy 2 
fragments up here to the top. If they are not clear in itself, please 
read on first. And let us find a common approach to name those things....

----
Maybe for clarifications. I started this using the word "view". But I 
can see there are 2 ways to read this word.
- "physical view": Like a painter. The final output of the combined 
text/style information. Most common drawing it to a canvas. But it could 
be a reader too.
- "logical view": (I guess what you called the grid?). A module that 
takes the raw-text, and converts it into a structure suitable for the 
"physical view", applying style/highlight info on the way.

----
I do use the word "view" to describe the logical-view of a source-file 
transformed into a grid.
I do  not  use the word "view" to describe the painter. ( I propose 
"physical view")
I do  not  use the word "view" to describe the high level dual 
visibility of the same source-file in 2 windows (or a splitted window) ( 
I propose "user view" ?? imho a weak description, not good)


Some of the answers where given before I understood that "view" in your 
text sometimes refers to "multi window display".  I tried to amend them 
in a 2nd run through my answer. I hope I didn't miss any. If my answers 
do not seem to match your text, I probably read view in a different way 
from what you meant


Hans-Peter Diettrich wrote:
> Martin Friebe schrieb:
>   
>> Then how to you handle double width chars? This si one of the problems I 
>> still have to address.
>> Even in a proportional font, some chars (Chinese, and other) have twice 
>> the width of a normal char. They will 2 positions in the grid.
>> Actually it is the same issue as tabs.
>>     
> Are these characters inside the Unicode BMP?
>   
Yes, and there are quite a few Lazarus users waiting for a solution.
In utf16 they occupy 1 code point ( they are encoded in  a single 16 bit 
word), but need display twice the display width. Actually it seems they 
are called "full width", while all others  (like western chars) are 
called half width.
> Full Unicode requires assistance by some sophisticated library, that can 
> deal with all the oddities of character sequences (ligatures...), and 
>   
True, and so far we only talk about the display, it also needs to be 
edited. Anyway there is much other work before I get there, so I think I 
defer the details on this part.
>> Column Blocks, as well as horizontal caret movement have to deal with 
>> this anyway, since you may allow to be in the middle of a tab. But you 
>> can not permit to have the caret in the middle of a chinese char.
>>     
> No problem, my tabs have a special display encoding, that forces the 
> caret to move to the next ordinary character cell.
>
> Is it really still a column block, when a double-width character hangs 
> out on it's right boundary?
>   
Well it is the users option (with tabs) to either have ragged column 
blocks, or cut tabs into spaces or ....
With Chinese (and I believe Arabic, and a few other language) there are 
no options (the char can not be cut).
>> The question still is how do your painters to all that. IMHO the mapping 
>> into a grid requires a lot of info (folded/ word wrapped/ tabs,...) as 
>> well as highlighting info. The painter as I see it collects all this 
>> info, but the info is provided by other objects.
>>     
> Right, I left the implementation of the highlighters and folding to 
> dedicated objects. The classes have to implement only the very slim 
> interface of the base class, everything else is open end.
>   
Right that sounds similar to my plans. The folded info is stored in a 
FoldTree (well at the moment it is split, and some is still stored on 
the raw-text-lines, work in progress)
The mapping is done in FoldedView.

 From all I read the difference is, that I put a viewer-class into a 
stack. You seem to have this in your grid-class, or a specialized 
inherited grid-class.
Probably both approaches have their benefits. In order to compare them 
we would have to deeply analyse both of them.
> BTW, I stored "characteristic" info in fixed size records, which can 
> easily saved and exchanged together with the source file or the global 
> settings. No encapsulation, but easy to use, and little chances for 
> coding errors.
>   
You are speaking of the highlighting info? Maybe easiest to give a 
usecase or example? Also when you say characteristic, do you mean: the 
details of the characteristics (e.g. numbers are blue or comments are 
bold), or do you mean whih characteristic apply to a char/group of chars 
(e.g. format the next 3 chars with the format for numbers).
>> The View port for example does not do the painting, it is a helper class 
>> to map the right code into the grid. It is used by the painters, but 
>> also used outside.
>> It allows (without accessing the painter) to check if a char or the 
>> caret is in the visible area. ( There still is the question if it will 
>> be the painter or the viewport who defines the size of each grid cell 
>> (basically the font size))
>>     
>
> In my model nobody has to know about the painting, all required 
> information resides in the line buffer and viewport outline. Apart from 
> the basic client and gutter size (in pixels) and the font size, the 
> viewport outline contains the overall grid dimension, corresponding to 
> the line/row count of the document after block folding, the visible area 
> is described by a scrollable offset within the grid, or (0,0) for 
> painting in client address space, and the current extent of the viewport 
> (client area of the control), in both fully and partially visible 
> characters.
Sounds identical to my idea. Only the painters (gutter and text) will 
know about painting, and they get provided by a text fragment fully 
transformed into a grid, with all char and highlight info provided.

In other words I could replace the painters by other modules, saving a 
bitmap, voice-reading, writing html or richtext or pdf. (though some of 
them would probably prefer the text before it is transformed into a 
grid, but maybe with some of the other transformations (folding or 
wrapping?) already applied...)

>  The separation into fully (page size) and partially visible 
> (viewport size) characters allows to e.g. scroll by (fully visible) 
> pages in both directions, while painting and display of the caret stops 
> only at the viewport margins - the latter (caret display) is not 
> properly implemented in the current LazEdit!
>   
Please fill in a bug (or feature) report for the caret stuff.
Besides, I agree the way the caret is currently implemented is horrible.
>>> ACK. A MVC (model-view-controller) approach migth be better. The model 
>>> holds the source files, the view manages painting and user interface 
>>> (mouse and keyboard), and the controller updates the document upon input 
>>> or other commands, and synchronizes the related view(s) afterwards.  
>>>       
>> The view IMHO is more than one class, that gradually apply the mapping 
>> from a Source-Holder (TStringList) to a char-grid. The Painter then 
>> transfers each char to from the grid to the canvas.
>>     
>
> All that has to be encapsulated in every single view(er). When the code 
> explorer is a view, it's internals have almost nothing in common with 
> the text viewer. A gutter also is kind of an viewer, coupled with the 
> text viewer only by a common TopLine, but independent otherwise.
>
> Of course the various viewers are related to a distinct document and 
> helper objects, from which they obtain all the information to be 
> displayed, but the kind of required information depends on their 
> individual tasks.
>
> The document base class has (virtual) functions for the conversion 
> between stored (file based) and visible (possibly folded) coordinates, 
> so that the details of folding are encapsulated. When the gutter 
> painter/viewer requires information about blocks, it also can obtain 
> that information from the according methods in the document base class.
>   
The last paragraph sound very much like what I have started.

One of the problems in this mail-thread is that we had different 
association for the names each other used to describe their classes. And 
another problem (introduced by me) is that at some point I tried to 
simplify thinks by referring to my structures with the names I took from 
your description (that was when I switched from "view" to 
"grid-provider") that was likely more confusing than helpful.

Maybe for clarifications. I started this using the word "view". But I 
can see there are 2 ways to read this word.
- "physical view": Like a painter. The final output of the combined 
text/style information. Most common drawing it to a canvas. But it could 
be a reader too.
- "logical view": (I guess what you called the grid?). A module that 
takes the raw-text, and converts it into a structure suitable for the 
"physical view", applying style/highlight info on the way.

The 2 classes go hand in hand. The desired output of the logical-view is 
a grid-matrix, for the kind of physical-view we currently have in mind. 
(It may slightly differ for a reader (text to voice)).
Ideally the logical-view should be divided into a part that is 
independent of the physical-view, and a part that is allowed to depend.
>> That is what I am currently trying to do
>>
>> the painter looks at a "grid-provider",  a grid provider may read either 
>> the source or another grid-provider as input. I currently call those 
>> grid-provider "View".
>>     
>
> Okay - with the subtle difference that in my model the painter is 
> provided by the grid-provider with all required information, eliminating 
> the need for any callbacks to other objects. The extent of information, 
> required by an painter, can be determined easily, and only that 
> information has to be passed to the painter.
>   
Rereading my last paragraph. I seem to have mixed up some terminology. 
Yes the painter is provided by the "logical viewer" (aka grid-provider) 
with all information it needs.
> Did you realize that mutliple views of the same source file can have 
> different TopLines, viewport sizes, word-wrap settings, and much more?
Yes, I realized that. (And I just realize you do not talk about view as 
i have read this before. I have read "logical view". You write "user 
view, in 2 actual windows)

>  I really cannot see how one grid-provider can fit these different needs of 
> multiple views. A meaningful separation IMO were:
> - file provider: access in file coordinates (by characters or lines).
> - block manager: access to visible (expanded) blocks, in display 
> lines/columns.
> - view: mapping of visible lines into display (row/col) coordinates.
> - painter: mapping of viewport into pixel coordinates.
>   
See my comments above. When I moved to the wording "grid-provider" I 
attempted to adapt to your terminology. This went wrong and didn't help 
making my point clear.
In my initial description I spoke of views (meaning logical views, not 
the painter). I also included the actual raw-text (file-provider) in the 
list of views. Because in my stacked organization they share some part 
in their interface. A logical view reads it's input from either another 
logical view or from the file provider (hence the file provider must be 
able to look like a logical view)

For some clarification (now that I got this as "user view across 2 
windows). Each "user-view" will have it's complete own stack of logical 
views. But each stack reading the same instance of the the file-buffer
> The latter (pixel mapping) occurs in two places, in the grid itself from 
> mouse coordinates in grid coordinates, and in the painter from grid 
> coordinates into canvas coordinates (different directions). The required 
> information is stored in the viewport outline
>> One thing must change, currently the PaintLines code, combines the 
>> highlight info with the grid-view result. But that means mapping the 
>> highlight info.
>> The highlight info must be applied to the unmodified source, and then 
>> share the way through the grid-providers
>> (That's actually something I realized from this discussion => good)
>>     
>
> That's why keep both the characters and their text attributes in the 
> line buffer, passed to the painter. I started with separate regions, 
> describing the begin and length of tokens, selection etc., but it turned 
> out that the handling of overlapping regions is accomplished easier by a 
> direct mapping of the text attributes to every single character. The 
> text attributes then can be used to e.g. determine, whether the mouse is 
> over a (character in a) hyperlink.
>   
The code for handling overlapping regions has yet to be refactored, it 
does already handle overlapping region. But yes this needs to and will 
move out of the painting code
>> But for me the 
>> (as an example) the tab-view/expander is not a subclass of the painter 
>> (or grid). The tab-view/expander class is a class of it's own 
>> (inheriting from an abstract TextView/GridMapper).
>>     
>
> Why should tab expansion require a separate class, extending or 
> descending from any other class? The tab settings are global (IDE wide), 
> and can be reflected in a commonly used data structure or tab-expander 
> singleton.
>   
You wrote above:
> Did you realize that mutliple views of the same source file can have 
> different TopLines, viewport sizes, word-wrap settings, and much more?
>   
Which could include tab settings....
Anyway, there also is the quest for those elastic tabs (not really a 
source editor feature) but it doesn't hurt if it can be provided easily.

And collecting tab-handling code in a single place, makes maintenance 
easier too.
>> All the individual view/grid/mapping classes are organized in a stack. 
>> You can at anytime add/exchange mebers of the stack to archive new 
>> functionality.
>>     
>
> Sounds good, but I doubt that this is feasable. The modules are so 
> tigthly coupled, that an implementation in distinct units will be almost 
> impossible. This way adding new functionality will require to edit the 
> common unit, so that it doesn't matter in which class (common or 
> separate) the functionality is implemented.
>   
Well yes there will be problems to overcome. But I still think it is 
possible. And I also see a benefit in it, if where changes to the common 
unit can not be avoided, they can be minimized.

>> See above. You always speak of your Grid in singular, as one class. For 
>> me this is a list of classes (the stack), plus the helper classes 
>> (Highlighter and Markup)
>>     
>
> My design is bottom up, with open end. The base class implements a 
> default behaviour, that can be modified in a derived class, as can be 
> seen in the TTextViewer class. The base classes and the base unit(!) 
> never must be touched when the functionality is extended.
>   
Extending by inheritance is exactly where I see the problem.

Lets say I provided to or more extension of how word wrapping should be 
handled. I put each of my extension into a new subclass of your class.

Now  I want to provide 2 or more extensions to tab handling (or anything 
else). Each of those extensions should and could extend each of the 
word-wrap behaviours.

How do I do that? (Let's say I had 2 word-wrappings and 3 tab handling).
Do I write 3 subclasses for each of the 2 ward wraps?

>>> TopLine and LeftChar are not of any interest outside the view. A 
>>> ScrollIntoView method will be sufficient for the outer world, with 
>>> document based coordinates, perhaps with an anchor (alTop, alBottom, 
>>> alCenter).
>>>   
>>>       
>> The View here being a SynEdit drawing a (possible shared) Textbuffer? 
>> True TopLine should not be needed outside, but it is needed for Caret 
>> Control.
>>     
>
> A shared text buffer with a shared caret or scroll position does make no 
> sense to me. It is debatable whether bookmarks or block folding should 
> be the same in multiple views of some file, with regards to the amount 
> and management of such block trees, but the user must be allowed to move 
> to different places in every view, select different parts of the text, 
> have different (hyperlink) history lists, insert/overwrite modes etc.
>   
There must be a misunderstanding. I never said that a caret should be 
shared between 2 user-views of the same text.
Of course they must not.
> Thus caret control has to be private to every view. Please try to 
> separate all your intended helper objects, with regards to their later 
> use, as being bound to a single document, a single view, or whatever 
> else. Also keep in mind what has to be saved and restored when the user 
> tabs through the file list of a notebook.
>   
I meant to say caret/topline a needed within each single SynEdit (where 
two SynEdits can be seen a 2 user-views of the same raw-text (file) 
buffer).
But being visible throughout each such instance (user-view), means 
caret/topline are visible outside the logical-views described earlier.

I do use the word "view" to describe the logical-view of a source-file 
transformed into a grid.
I do not use the word "view" to describe the painter. ( I propose 
"physical view")
I do not use the word "view" to describe the high level dual visibility 
of the same source-file in 2 windows (or a splitted window) ( I propose 
"user view" ?? imho a weak description, not good)

Imho 2 source files seen in 2 windows, should act like to independent 
instances of SynEdit (no matter how they are structured internally) that 
share only selected information. This selected information is the 
TextBuffer, containing the source file.  They will update according to 
changes made by the other instance(s).
They may if the user wishes share additional info, such as folding or 
the selected block. But they do not by default.


Best regards
Martin