[Lazarus] Adding codepage-support to the RTL (making LConvEncoding obsolete)

[Marco van de Voort]

On Fri, Dec 03, 2010 at 11:49:28AM +0100, Guy Fink wrote:
> I have opened issue #0018144 in the bugtracker and uploaded a new version of my codepages unit.
> 
> My description on this :
> 
> In September we had a discussion on the Lazarus-mailing list to rewrite LConvEncoding and move the functionality to the RTL (Thread: rewriting of LConvEncoding).
> 
> Since there I did a lot of coding to implement an effective algorithm, both for Singlebyte- as for Doublebyte-Codepages. A first release was on the mailing list mid-October, mainly as a base for further discussions. But there were no comments or suggestions on this.
> 
> So here is a nearly final release with many changes to the first version.
> 
> Major points:
>  - The unit supports Single- and Double-bytecodepages trough the same functions
>  - Widestringsupport (configurable)
>  - UTF8 and UTF16 support (UTF16 needs widestrings)
>  - Direct conversion from CP to CP without intermediate string
>  - Uppercase and Lowercase support
>  - Underlying Unicodes as of V 6.0.0 (October 11, 2010)
>  - A converter-application to convert Unicodedefinitions to a complete
>    pascal unit. The cp_* units are entirely generated by this app.
>  - Conversion up to 80% faster for SBCS. For DBCS up to 100 times

Major problems that I see from a quick look:
- does not integrate with FPC's existing systems.
   - FPC already has a CP_ generator and loader system (creumap and charset)
   
- Introduces an own enumeration for charsets. No possibilities to integrate
     this with system codepage enumeration. (or whatever substitute FPC will 
     define for this). I know that it doesn't help that FPC hasn't made a 
     decision about this yet wrt cp_newstr 
- when using these routines for simple UTF8 operations, the large table with
   descriptions of unit unicodemappings is always linked in.
- no attempt to use system codepage routines and tables. (this could be done
   on widestr manager level for the platform though)

None of these faults prevent inclusion in say packages/ as a backup solution
for systems without codepages, but for rtl/ it is not modular enough (too
big, too much a standalone system that doesn't integrate with system)