[Lazarus] Multi-threading support in IDE

[Vincent Snijders]

Martin schreef:
> Mattias Gärtner wrote:
>> No. Just write a ThreadedProcess. That would work pretty much like 
>> TAsynProcess.
>> And I guess it will suffer the same bug.
> I don't know what you mean by "ThreadedProcess" unless you meant yes, 
> but wanted to indicate a different form of implementation?
> 
> IMHO, if the process running fpc is started from a different thread, 
> then the IDE can be kept as functional, as it can with TAsyncProcess.
> In fact if the TProcess is in started from a thread of it's own, it does 
> not want to be Async, it wants to be blocking, so you don't need to put 
> a lot of sleeps into your code.
> 
> As separate thread *can* also solve one more issue.
> 
> Currently  even the AsyncProcess is done in a loop. It does use the 
> OnDataRead event, but only to fill a cache, which is then used in a loop.
> It should not be in a loop at all. It should be purely event driven. 
> Then it would continue to run, even if a "search in files" is started 
> while compiling....
> 
I have looked a bit more why outputfilter is so slow at parsing a lot of compiler 
output (e.g. compiling a simple LCL app with -va). The pipe buffer is rather small, 
NumBytesAvailable is not bigger than 1280 bytes. So OnAsyncReadData reads only 1280 
bytes at a time. OnAsyncReadData is only called after an application.handlemessage. 
So on line 319 of outputfilter.pas, the buffer is most times empty, but 
NumBytesAvailable = 1280, and we are sleeping until Application.HandleMessage is 
called.

I had some success with the following patch. It does a sleep(0) to give the compiler 
the chance to write to the pipe again and tries to read it again.
--- ide/outputfilter.pas
+++ ide/outputfilter.pas
@@ -1206,9 +1206,12 @@ var
    Count: LongWord;
  begin
    if fProcess=nil then exit;
-  Count:=TAsyncProcess(fProcess).NumBytesAvailable;
-  if Count>0 then
-    FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
+  repeat
+    Count:=TAsyncProcess(fProcess).NumBytesAvailable;
+    if Count>0 then
+      FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
+    sleep(0);
+  until Count=0;
  end;

  function TOutputFilter.CreateScanners(ScannerOptions: TStrings): boolean;

> If you create a class TThreaded Process (just to simulate an 
> AsyncProcess within a (hidden) Thread), then you still must transform 
> the current loop into an event driven model.
> 
> 
> However this 2 issues can be addressed on their own. even using a 
> blocking thread, you can go (almost) event drive. Put the read on a 
> timer (every 10 milliseconds) or OnIdle, The read will block, but after 
> each read, it will return to the event loop => events will be processed 
> normally (instead of ProcessMessages), and then the next read happens on 
> timer, or idle
> 
> 
> In fact if you make everything eventdriven, then you could introduce 
> some method to mark events as thread-save, and the event-loop could 
> automatically deploy threads, if a lot of events occur => that would be 
> cool.

It is not clear to me why we can't check NumBytesAvailable (available since fpc 
2.2.2 or earlier) on unix and windows and just forget about TAsyncProcess for compiling?

Vincent