Ideas to think different

By default Apache spark has FIFO (First In First Out) scheduling. To configure Fair Schedular in Spark 1.1.0, you need to do the following changes - 1. In spark home, there is a conf folder. In the conf folder. 2. Open the spark-env.sh and the following lines -  SPARK_JAVA_OPTS="-Dspark.scheduler.mode=FAIR" SPARK_JAVA_OPTS+="-Dspark.scheduler.allocation.file=$SPARK_HOME_PATH/conf/fairscheduler.xml"  Path of the fair scheduler xml being mentioned. sample Contents of the fairscheduler.xml <?xml version="1.0"?> <allocations> <pool name= "test1" > <schedulingMode> FAIR </schedulingMode> <weight> 1 </weight> <minShare> 2 </minShare> </pool> <pool name= "test2" > <schedulingMode> FIFO </schedulingMode> <weight> 2 </weight> <minShare> 3 </minShare> </pool> ...

If we can store a class object in hbase and retrieve it back, then we can easily resolve the problem of persistant storage of variables. As a result we don't need the object to be reinitialized again. Pick it it from the previous saved instant and start using it. And again save it back for future use with latest updates. This is very useful when we are using a ArrayList or Hashmap. which has to be populated again and again every time your program is intialized for the same set of values. You can save the List as it is and pick it from Hbase/HDFS. As a result, you will be getting a faster performance.  So here we are basically talking about serialization and deserialization column value of hbase/HDFS. So lets start with code part. public static String serializeObjectToString(Object object) throws Exception {         ByteArrayOutputStream arrayOutputStream = new ByteArrayOutputStream();         GZIPOutputStream...

It has been a long time, I have not written a blog. So i thought of writing one. When i started writing it. I thought....., what are the things i need to cover. I searched on the internet a lot about hbase api tutorial, which comes up with rich examples. But in vain, there were very few to be found. Then i thought why not to add the things i am learning. It will be a note for self as well who so ever wants to follow up for examples can also do the same. I will start the No-SQL database getting famous i.e: Hbase . The first question that comes into mind is, What is Hbase? As " wikipedia " says - " HBase is an open source , non- relational , distributed database modeled after Google's BigTable and is written in Java . It is developed as part of Apache Software Foundation 's Apache Hadoop project and runs on top of HDFS (Hadoop Distributed Filesystem) , providing BigTable-like capabilities for Hadoop. That is, it provides a fault-tolerant way of storing...