AboutKrishna Prasad Expertise Data Warehousing and Data Modelling, Cube Building, Data Migration Strategies and Middleware aspects
Experience Several large scale data warehousing and data migration projects. Built a data warehouse of 7 Terrabytes for an organization. Was involved in cradle-to-grave (data architecture to implementation and testing) of a large scale data warehouse effort, in a control position.
Education/Credentials Bachelor of Technology and MBA
Expert: Krishna Prasad Date: 11/7/2007 Subject: What database handles the largest files?
Question QUESTION: I know that certain database system (e.g. ACCESS) defaults at 2 gigabytes. I need a database system that will handle between 8-12 terabytes worth of data. The program I want to design I cannot break up the data into "chunks" very easily. In its smallest form it runs about 200 gigs.
Actually, I'd like to take this huge file (200 gigs) and have it search out multiple databases totally between 10-14 terabytes.
Any suggestions?
ANSWER: For that large a database, you need an industrial strength database. I have used Oracle, Sybase and DB2 in the past to hold and manage that kind of data volumes. All of them can handle the requirements.
I know that there are other Database systems out there (maybe MS SQL Server?) but I personally have not built anything that big on these.
So please take this as a partial list of the options you have.
---------- FOLLOW-UP ----------
QUESTION: Of the three systems you mentioned about, which would be the easiest for someone to learn if they do not have much programming experience? Also assuming that programming expertise isn't at issue, which would provide the best and fastest indexing? Thanks
Answer Unfortunately, with power comes complexity. All these systems are complex and require knowledge to use. They are also expensive. (An inexpensive - free - midrange database will be MySQL. It is not as powerful as these three, but it is free since it is Open Source).
As for performance, these days all of them are continually evolving and increasing in performance. The most popular is undoubtedly Oracle, as it runs on most Operating Systems (and always has, since it was created) and since it has really sophisticated features.
The good news is this. Though you need to know a lot to use these systems efficiently, to start using to create tables and indexes does not require deep knowledge. You can learn the online interfaces for each of this (SPUFI for DB2, PL/SQL based screens for Oracle, etc) quite quickly.
