[Lazarus] Multi-threading support in IDE

[Martin]

Vincent Snijders wrote:
> I have looked a bit more why outputfilter is so slow at parsing a lot 
> of compiler output (e.g. compiling a simple LCL app with -va). The 
> pipe buffer is rather small, NumBytesAvailable is not bigger than 1280 
> bytes. So OnAsyncReadData reads only 1280 bytes at a time. 
> OnAsyncReadData is only called after an application.handlemessage. So 
> on line 319 of outputfilter.pas, the buffer is most times empty, but 
> NumBytesAvailable = 1280, and we are sleeping until 
> Application.HandleMessage is called.
>
> I had some success with the following patch. It does a sleep(0) to 
> give the compiler the chance to write to the pipe again and tries to 
> read it again.
> --- ide/outputfilter.pas
> +++ ide/outputfilter.pas
> @@ -1206,9 +1206,12 @@ var
>    Count: LongWord;
>  begin
>    if fProcess=nil then exit;
> -  Count:=TAsyncProcess(fProcess).NumBytesAvailable;
> -  if Count>0 then
> -    FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
> +  repeat
> +    Count:=TAsyncProcess(fProcess).NumBytesAvailable;
> +    if Count>0 then
> +      FAsyncOutput.Push(TStream(TAsyncProcess(fProcess).Output),Count);
> +    sleep(0);
> +  until Count=0;
>  end;
>
>
Makes sense. Why the sleep(0)?
Maybe the buffersize can also be increased?

>>
>> In fact if you make everything eventdriven, then you could introduce 
>> some method to mark events as thread-save, and the event-loop could 
>> automatically deploy threads, if a lot of events occur => that would 
>> be cool.
>
> It is not clear to me why we can't check NumBytesAvailable (available 
> since fpc 2.2.2 or earlier) on unix and windows and just forget about 
> TAsyncProcess for compiling?
>
NumButesAvailable seems to block on TProcess.
 From my understanding on a none blocking socket/pipe you can call all 
the operations like read/write/peek and they will always return immediately.
On a blocking socket they will always wait until data is there.
On a blocking socket one has to call select (fpselect), which should 
indicate the availabilty of data without blocking.

But Imho we should avoid sitting in a loop and sleeping.

That is what TASyncProcess.OnReadData is there fore. Read and handle all 
data in the event, exit the event and wait for the event to fire again.
Then you can even to find in files in parallel with compiling (it wil be 
slow, since both access the disk) and it will work.

Martin