Cloudera Quick Start VM in Hyper-V

Overview

Marc Andreessen penned his now famous essay, “Why Software Is Eating the World” in Wall Street Journal in 2011. However, if we dig under the covers, it's really virtualization that is powering this phenomenon. Today, literally every hardware component such as - processor, memory, network, storage, network load-balancer, network router and switch, firewall are virtualized. In fact, even a user is now "virtualized" especially in scenarios such as load testing and synthetic monitoring. Finally, as a concept, money is also virtualized in the form of crypto-currencies.

In this post, let's go through how we can implement virtualization in a local machine and get better understanding of the concepts of virtualization. 

Virtualization 

One key aspect, driver and benefit of virtualization is sharing of the physical resources. Before we dive in, let's define some concepts.
  • Host Machine - This can be a server rack, single server or a desktop/laptop. In our case, it is a laptop. Typically, virtualization has to be configured in the hardware BIOS.
  • Host Resources - These are the "physical" resources in the host machine such as cpu, memory, network cards, disk, display. peripheral devices.
  • Host OS - The operating system of the host machine. In addition, virtualization software/drivers need to be installed int he Host OS too.
  • Guest VM - This is the "virtual machine" that will be created. So, one could create multiple VM's in a single physical machine.
  • Guest Resources - Each guest VM can have it's own set of resources. For example, it is possible to create a VM without a display attached to it.
  • Guest OS - In each VM, one can have different operating systems and applications.
  • Hypervisor - Type 1 or Type 2

Hypervisor

The key construct of virtualization is a hypervisor. Hypervisor is the component that allocates and manages the various physical resources among the virtual machines. There are broadly two types of hypervisors - Type 1 and Type 2.

Image Source: IBM
As seen in the figure above, Type 1 hypervisors (Hyper-V, VMWare ESX etc.) are more closely integrated with the host OS. Type 2 hypervisors (Oracle Virtual Box, VMWare Player etc.) run as an "application" on to of the Host OS. For running applications in "live" environment, Type 1 is typically used. Type 2 is more used in the "development" environment. However, now with Hyper-V from Microsoft, it is now possible to have a Type 1 hypervisor that can be used in a Windows desktop/laptop.

Exercise

In this example, the following are going to be used - 
  • Host Machine - Dell Inspiron 15 7000 Gaming laptop.
  • Host Resources - The laptop has 8 logical cores (i7 Processor). 16 GB RAM, 1 TB Hard Disk and 1 Network Card.
  • Host OS - Windows 10 Professional.
  • Guest VM - 1 Guest VM.
  • Guest Resources - Guest VM has 4 cores, 10 GB RAM and virtual hard disk assigned to it.
  • Guest OS - Cent OS 6.
  • Hypervisor - Type 1 (Windows Hyper V)

Prerequisites

  • Ensure that the virtualization is enabled in the desktop/laptop BIOS.
  • Open the "Windows Features" option under "Control Panel". Enable Hyper-V option as shown below. Restart the desktop/laptop.



Setup

In this example, let's deliberately take a complex use case. We need a big data framework to be setup. Let's look at Cloudera distribution. Cloudera internally consists of various software components such as HDFS, HBase, Impala, Solr, Spark etc. It is possible to setup these components separately, but it is time consuming and error prone. Here comes virtualization to the rescue!

The steps are -
  • Download the VMWare image of Cloudera Quick VM. Copy it to a folder say "D:\Temp".
  • Install Oracle Virtual Box.
  • Open the Virtual Box folder in Command Prompt. Run this command to convert the image from VMWare to Hyper-V format - vboxmanage clonehd “D:\Temp\cloudera-quickstart-vm-5.13.0-0-vmware\cloudera-quickstart-vm-5.13.0-0-vmware.vmdk” “D:\Temp\cloudera-quickstart-vm-5.13.0-0-vmware.vhd" --format vhd
  • Open the Hyper-V Manager and create a new VM. Create a new external virtual switch and call it 'External Switch'.
  • Create a new VM with following settings - 
    • Generation 1 VM 
    • 4 Cores 
    • Min 8 GB, Max - 10 GB RAM (with dynamic allocation)
    • Instead of creating a new virtual hard disk, assign the virtual hard disk created above
    • Assign the 'External Switch' 
  • Start the VM.
  • The user id and password for the VM and all services is cloudera/cloudera.
  • Depending on the amount of CPU and memory assigned to the VM, allow the various Cloudera services to start. It may take anywhere from 3-5 minutes.
  • Once all the services are up, the Cloudera instance can be used by connecting to the VM.

Tests

You can also get the IP address of the VM and open http://[VM IP Address]:7180 in the host machine browser to access Cloudera Manager.



This is a big data framework after all. Let's now see if we can ingest some sample data.
  • Open Hue from http://[VM IP Address]:8888
  • User ID/password is cloudera/cloudera.
  • Using Hue, one can create table in Impala by running this command - CREATE TABLE default.t1 (x INT, y STRING);
  • Next step is to insert multiple rows using - INSERT INTO default.t1 VALUES (1, 'one'), (2, 'two'), (3, 'three');
  • Finally, query the table using - SELECT * FROM default.t1; 




Benefits

From a development perspective, here are the advantages of virtualization -
  • It saves the time of having to setup multiple components for a complex software.
  • A VM can be setup in minutes to run any kind of application
  • If, for some reason, the VM is corrupted, it can always be re-installed.
  • Each developer in the team can have their own full fledged Cloudera instance for developing, testing, troubleshooting etc.

Location: Bengaluru, Karnataka, India

Comments

  1. Logan30 July 2019 at 05:09

    initramfs and dev/disk/by-uuid not found, and dracut emergency shell issues. I've tried this with both the VirtualBox and the VMWare .OVA files extracted to VHDK and converted to VHD.

    ReplyDelete
  2. Alice4 November 2019 at 20:26

    Such a great guide for Virtualization newbies. Thanks!

    ReplyDelete
  3. Laura Bush18 March 2021 at 10:10

    I just need to say this is a well-informed article which you have shared here about software development. It is an engaging and gainful article for us. Continue imparting this sort of info, Thanks to you. Android developer tools for getting started

    ReplyDelete

Post a Comment

Popular posts from this blog

Meta Prompting for Success

Image
Meta Prompt  Regardless of the Generative AI model used, sometimes the model may refuse to provide a response for a valid prompt. Often the reasons cited are - "I don't know", "I am not trained for the task", "violation of safety/responsible/ethical guardrails". This is sometimes more acutely felt when using Generative AI models for creating images. Overcoming this behavior can be frustrating and time consuming. An approach is to ask the AI model itself to generate prompts so that the prose and nuances of prompting are aligned with how the AI model has been built. This is called meta-prompting . The theory is that the prompts created by the AI model would have better chance of working. Let's dive in to see if and how this meta-prompting works in Gemini .  How to Create a Meta Prompt As with all good prompts, start with a persona/situation that describes the overall context. Next, describe the task in detail. In this case, it is about creating some ...
Read more

Book Review - The Price of Being Fair

Image
Overview  As the FairPrice group in Singapore completes 50 years, "The Price of Being Fair" book is a timely story of it's birth and evolution. The book has been authored by Sue-Ann Chia and Shing Huei Peh. While being true to history, the book has nuggets of "behind the scenes" information, given the time the authors have spent with various leaders at FairPrice. The book is already a best seller according to Kinokuniya book store. At under 200 pages, it makes for a breezy read. The Usual Suspects Having worked in retail industry for over two decades, the travails of  NTUC FairPrice seem familiar to me. A retail CEO once remarked that retail is a pretty "public" industry. There is not much behind which a retail business can hide. As such, retailers often have to follow the aspirations and concerns of the communities and customers they serve. The journey of FairPrice from being an EDLP (everyday low price) retailer to one having multiple formats, busin...
Read more

How to create a big picture

Image
Typically, you need wide view or fisheye lenses to capture a panoramic picture. An alternate method (though may not give the same quality) is to use software to create a panoramic image. There are several software available which you can use to create a panoramic image. The concept is called - "stitching". Some of the software available are - PTGui, Panavue, PAX-it, AutoStitch and so on. I sometimes use AutoStitch to create the panoramas since I have a 50mm lens for now. Each of these software vary in price and features. The added advantage is that you can take the pictures using any kind of camera - digital, phone or SLR. Steps: 1. Visualize the picture you want to take. 2. Use your camera and "divide" your image into rows and columns. An example being - 2 rows and 5 columns. 3. Take 10 pictures to cover your entire image. 4. Use any of the software to "stitch" them together. Here is an example of a panorama created using AutoStitch. I to...
Read more