[ale] In case you missed it

[Ed Cashin]

Data sets often have more columns than are supported by relational
databases like Oracle (1000) or Postgres (1600).  I tried using scidb,
which is a database by Mike Stonebraker, the database architecture guru,
but it wasn't at all polished, and I probably wouldn't use it again.

You can use Python's numpy or something, and let your RAM constrain the
size of your data, loading all the data at once from a file.  You can try
to use sparse data structures ...

I don't see a lot of easy answers for persistent-storage-backed arbitrarily
shaped tensor data with fast on-demand paging to RAM (that is smarter than
the Linux page cache would be).  Usually people use RAM, loading the data
into RAM and doing what they want next.  There are some promising
developments in Tensorflow, though:
https://www.tensorflow.org/guide/data_performance

All that to say that the story reflects the dangers of using a tool beyond
its limits, and all tools have limits, including databases.



On Wed, Oct 7, 2020 at 2:27 PM Pete Hardie via Ale <ale at ale.org> wrote:

> Once again, I repeat my frequent assertion that a spreadsheet is not a
> database, and your admin assistant probably does not have a CS/Programming
> degree
>
> On Wed, Oct 7, 2020 at 2:19 PM Jim Kinney via Ale <ale at ale.org> wrote:
>
>> Excel blew up in the faces of UK healthcare.
>>
>>
>> https://www.theregister.com/2020/10/05/excel_england_coronavirus_contact_error/
>> --
>> Computers amplify human error
>> Super computers are really
>> cool_______________________________________________
>> Ale mailing list
>> Ale at ale.org
>> https://mail.ale.org/mailman/listinfo/ale
>> See JOBS, ANNOUNCE and SCHOOLS lists at
>> http://mail.ale.org/mailman/listinfo
>>
>
>
> --
> Pete Hardie
> --------
> Better Living Through Bitmaps
> _______________________________________________
> Ale mailing list
> Ale at ale.org
> https://mail.ale.org/mailman/listinfo/ale
> See JOBS, ANNOUNCE and SCHOOLS lists at
> http://mail.ale.org/mailman/listinfo
>


-- 
  Ed Cashin <ecashin at noserose.net>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.ale.org/pipermail/ale/attachments/20201007/0015d583/attachment.html>