Tech - NOSQL Tutorial - Part 1 (Introduction)

Several people and organizations have cited the growing size and consumption of data in the current digital age. I have deliberately used the word 'digital' instead of 'internet'. There is growing convergence of internet, media (video, audio) and devices. One can today, browse a newspaper on a Smart TV and conversely view a movie on a phone or tablet. There is of course the 'social' chatter on sites such as Facebook, Flickr and Twitter to name a few.

The implication of all this is of course transmission and storage of data. I am going to cover the latter aspect in this series of articles.

I have worked on various RDBMS starting from databases like SQL Server, Oracle and right up to DB2 and Teradata. I have always been curious about how data and it's application. So, I have been researching on some of the technologies that power large scale data. A key component has been the rising use 'NOSQL' databases. The NO - doesn't stand for No as in Dr. No but for 'Not Only'. This should avoid any perception that these databases should replace the traditional RDBMS. Now that we have that sorted out, hopefully I will not have the hordes of database enthusiasts despairing on me!

To start with, there are no standards yet that have emerged among NOSQL databases. It has so far been powered by Open Source products and organizations. In general, the most common types of NOSQL databases are the following -
  1. Key - Value based - Think of it as a hash with key:value pairs of data. Examples - Amazon Dynamo, Voldemort.
  2. Column based - A multi-dimensional sorted map. Examples - Google Big Table, HBase.
  3. Document based - Schema free, document based databases. Examples - Apache CouchDB, MongoDB.
  4. Graph - Inspired by the Graphy theory (yes, from Mathematics). Examples - Neo4j

While the above databases are different in their own way, they all share some common characteristics, like -
  1. NOSQL has been designed from the ground up to be hugely scalable and very fast.
  2. They are very web friendly with API's to access and manipulate data.
  3. They may not support ACID (Atomicity, Consistency, Isolation, Durability) but lean more towards CAP (Consistency, Availability, Partition Tolerance).
As most people have been used to RDBMS, comparisons are inevitable. Since, we don't have a standard yet for the NOSQL databases, I will refrain from it for now.

Some of the popular websites that use the various NOSQL databases (among other databases) are - Twitter (Cassandra, Hbase), Facebook (Cassandra), Google (Big Table), LinkedIn (Voldemort). Typically, the data that is generated on these sites run into several terabytes per day. The kind of data that is generated is quite varied too - right from search terms to tweets to photos. Imagine doing a sentiment or cohort analysis on a database with millions or even billions of rows and you get the picture.

One trend seems to be emerging, the era of using only an RDBMS for all data storage purposes is over. NOSQL databases seem to be emerging as a credible option for some of the applications. As they say, horses for the courses. For now, I am hooked into this new breed of databases. 

In the next part, I will start looking at one of these databases and dig a bit deeper.

Click here for - Part 2..
Location: Welwyn Garden City, Hertfordshire, UK

Comments

Post a Comment

Popular posts from this blog

Meta Prompting for Success

Image
Meta Prompt  Regardless of the Generative AI model used, sometimes the model may refuse to provide a response for a valid prompt. Often the reasons cited are - "I don't know", "I am not trained for the task", "violation of safety/responsible/ethical guardrails". This is sometimes more acutely felt when using Generative AI models for creating images. Overcoming this behavior can be frustrating and time consuming. An approach is to ask the AI model itself to generate prompts so that the prose and nuances of prompting are aligned with how the AI model has been built. This is called meta-prompting . The theory is that the prompts created by the AI model would have better chance of working. Let's dive in to see if and how this meta-prompting works in Gemini .  How to Create a Meta Prompt As with all good prompts, start with a persona/situation that describes the overall context. Next, describe the task in detail. In this case, it is about creating some ...
Read more

Book Review - The Price of Being Fair

Image
Overview  As the FairPrice group in Singapore completes 50 years, "The Price of Being Fair" book is a timely story of it's birth and evolution. The book has been authored by Sue-Ann Chia and Shing Huei Peh. While being true to history, the book has nuggets of "behind the scenes" information, given the time the authors have spent with various leaders at FairPrice. The book is already a best seller according to Kinokuniya book store. At under 200 pages, it makes for a breezy read. The Usual Suspects Having worked in retail industry for over two decades, the travails of  NTUC FairPrice seem familiar to me. A retail CEO once remarked that retail is a pretty "public" industry. There is not much behind which a retail business can hide. As such, retailers often have to follow the aspirations and concerns of the communities and customers they serve. The journey of FairPrice from being an EDLP (everyday low price) retailer to one having multiple formats, busin...
Read more

How to create a big picture

Image
Typically, you need wide view or fisheye lenses to capture a panoramic picture. An alternate method (though may not give the same quality) is to use software to create a panoramic image. There are several software available which you can use to create a panoramic image. The concept is called - "stitching". Some of the software available are - PTGui, Panavue, PAX-it, AutoStitch and so on. I sometimes use AutoStitch to create the panoramas since I have a 50mm lens for now. Each of these software vary in price and features. The added advantage is that you can take the pictures using any kind of camera - digital, phone or SLR. Steps: 1. Visualize the picture you want to take. 2. Use your camera and "divide" your image into rows and columns. An example being - 2 rows and 5 columns. 3. Take 10 pictures to cover your entire image. 4. Use any of the software to "stitch" them together. Here is an example of a panorama created using AutoStitch. I to...
Read more