[Lazarus] data matrix with thousands of columns

[Max Vlasov]

On Wed, Mar 27, 2013 at 1:24 PM, Andrea Mauri <andrea.mauri.75 at gmail.com>wrote:

>
> After calculations the app/user should be able to search for one/more
> samples (or for one/more columns) getting all/some values for the
> sample/column. Briefly I need to be able to rapidly get some values from
> this huge data matrix.
>
>

Andrea, looking at your description I probably have a suggestion, although
we're are more and more drifting from Pascal :)

Once I wanted to extract CIA Factbook data and when noticed there are about
170 columns decided to try Triplestore (Object-Propery-Value) similar to
rdf. Basically the data was contained in a table of Sqlite having very few
columns. Changing to your case this can be

CREATE TABLE [Data] (
... ,
[Row] INTEGER, [Column] INTEGER, [Value] TEXT
...
)

Additional tables were used for naming countries and properties (rows and
columns in this case) and indexes was used for quick querying. Although in
my case 170 was much less than your tens of thousands, this approach can
work. Your I noticed that queries for such table were similar two general
queries, only extra join was usually added. Moving from sql columns helped
a lot in thinking abstractly about the data and queries became more
flexible.

Max
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20130327/e2d95916/attachment-0003.html>