[Lazarus] filesystem timing Linux vs Win

[Max Vlasov]

On Sat, Nov 27, 2010 at 10:31 PM, Mattias Gaertner <
nc-gaertnma at netcologne.de> wrote:

> On Sat, 27 Nov 2010 20:14:38 +0100
> Florian Klämpfl <florian at freepascal.org> wrote:
>
> > Am 27.11.2010 18:53, schrieb Henry Vermaak:
> > > Also, the features of the filesystems
> > > are so different, you can't even compare them.  fat and ntfs are stuck
> > > in the dark ages compared to ext*.
> >
> > Modern NTFS implementations have some really nice and advanced features
> > none of the ext* has: snapshoting (saves file server admins a lot of
> > time :)) and transactions. I guess especially snapshotting makes ntfs
> > slow: CoW semantics simply requires a lot of coping of files and
> > fragments probably the disc. I made a similiar experience with btrfs
> > (whsich has CoW semantics by default as well) on linux, working with fpc
> > on a btrfs partition made fpc really slow. I didn't test further by
> > turning off CoW for btrfs, but it might be reason.
>
> Florian, the reading/writing of files seems to be ok under Windows. But
> the file access functions are much slower than under Linux (file exists,
> date, open, close).
> fpc already caches these attributes, but of course it has to read them
> every time it starts. It reads over 80 directories. Maybe
> someone can check how much time this costs under windows.
>
>

Mattias, it's challenging issue indeed. I might do some research, it's also
interesting to read the discussion that already was about the the file
scanning on Windows.

Two observations that come to mind:
- If one looks at any directory scanning utility in Windows, he or she will
notice that usually it doesn't consume much cpu, at least first scanning.
That's because all file operations are made with wait objects. So somewhere
in the kernel, the request is made to read something (part of file,
directory and so on) and the process that requested it just waits for a
signal not consuming cpu at all. So as a indirect measurement the more
percentage of total time cpu spend (can be queried with GetProcessTimes)
comparing to total time passed the less file system shortcoming affected the
compilation. Is it possible for ppc386 (maybe with a special option), report
with final message ("..compiled successfully") the total time +
(user+kernel) time. If the difference is significant most of the time, we
have to do something.
btw, the same logic can be applied for example for fragmentation of sqlite
databases. The less cpu spent on the query, the more the database is
fragmented internally

- When modifying a file, windows also changes the corresponding attribute of
the parent directory (and only direct parent). Since fpc looks only to
directories it knows, it could bypass scanning the files themselves, only
parent directories. I sometimes use this trick to find a file I worked with
recently in Far Manager since this allow me to work one level up in my
searching saving some time.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20101127/d43e7054/attachment-0003.html>