How to create Translucent and Shaped Windows in Java?
Posted on | October 14, 2013 | No Comments
Overview: In this article we will discuss about the transparency and different shaped windows in Java. In java 7 swing supports this feature and making swing UI components more flexible and user friendly.
Introduction: In some application, transparent window is a requirement to support its functionality. Now in Java, transparency can be implemented. A translucent window is created by altering its opacity by implementing a method called setOpacity on a JFrame component. But we must understand that a translucent window is only possible if the underlying operating system supports it. And we also need to make sure that the window is not decorated. To make a window undecorated, you need to call setUndecorated (true) method. Some time it is also required to change the shape of a window UI. To implement it, we need to call setShape method within the componentResized method. It will recalculate the shape when the window is resized.
Back ground: In java UI, support for translucency and shaped window was a long time demand for Swing/AWT components. For native development access to these properties are available from long time back. But it was not accessible to core java components. In Java6 onward, support for translucency and shaped window is available. Even per pixel level translucency is also supports in Java7.
Type of support for translucency and transparency: Java 7 supports the following three type of support.
TRANSLUCENT: In this feature the window can have different colors but with the same level of opacity. So the alpha value is same for all the pixels. This effect is suitable for fade off a window and then gradually decreases and increases the alpha value.
PERPIXEL_TRANSLUCENT: This property supports different alpha values within the window itself. This is used to fade selected region of the window.
PERPIXEL_TRANSPARENT: This property supports different shapes of windows rather than traditional rectangle. If per pixel transparency is supported then different shapes like circle, triangle can be created.
Before we use any of the above properties we need to test the support of under lying operating system. The test can be done by using method isWindowTranslucencySupported belonging to the java.awt.GraphicsDevice class. It takes the type of transparency as input and returns true/false to indicate the support.
Let us check a sample code to implement the translucency feature. In this example we have created a rectangular translucent window on a text area. First we have disabled the decoration mode to get the effect of transparency. Then we have checked if the underlying operating system supports it or not. If supported the effect is visible on the frame window.
Listing1: Sample showing Translucency of a JFrame window
import java.awt.Color;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.GridBagLayout;
import java.awt.event.ComponentAdapter;
import java.awt.event.ComponentEvent;
import java.awt.geom.Rectangle2D;
import javax.swing.JFrame;
import javax.swing.JTextArea;
import javax.swing.SwingUtilities;
public class TransRecFrame extends JFrame {
/**
* Create a transparent rectangular frame with 85% transparency
*/
public TransRecFrame() {
super(“Translucent Rectangular Frame”);
//Set layout
setLayout(new GridBagLayout());
//Create a text area
final JTextArea txtArea = new JTextArea(5, 50);
txtArea.setBackground(Color.CYAN);
add(txtArea);
//Call to disable decoration
setUndecorated(true);
//Call setShape to resize the shape when widnow is resized
addComponentListener(new ComponentAdapter() {
@Override
public void componentResized(ComponentEvent e) {
setShape(new Rectangle2D.Double(0, 0, getWidth(), getHeight()));
}
});
//Make the window 85% transparent
setOpacity(0.85f);
//Set other parameters
setLocationRelativeTo(null);
setSize(200, 300);
setDefaultCloseOperation(EXIT_ON_CLOSE);
setVisible(true);
}
public static void main(String[] args) {
//Create graphics environment
GraphicsEnvironment genv = GraphicsEnvironment
.getLocalGraphicsEnvironment();
//Check if OS supports Translucency
if (genv.getDefaultScreenDevice().isWindowTranslucencySupported(
GraphicsDevice.WindowTranslucency.TRANSLUCENT)) {
System.out.println(“OS supports translucency”);
new TransRecFrame();
}
}
}
Features to support translucency and shape: In Java7 there are mainly three features available to support translucency and shape of a window.
- Full window translucency: In this approach full window is translucent.
- Per pixel translucency: In this approach a part of the window is translucent
- Shaped windows: Make different shaped windows like ovale, circle, rectangular etc.
Per pixel translucency:
We have already seen how to make a complete window translucent in the previous example. Now we will discuss the second part of making a set of pixels translucent by using their background color. There are some limitations to implement this scenario. The window should not be full screen and the system must support pixel level translucency. The rest of the procedure is similar to the above example.
In the following example we will see how pixel level translucency is set in a frame.
Listing2: Sample showing pixel level translucency support.
import javax.swing.*;
import java.awt.*;
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
public class PixelTranslucency extends JFrame {
public PixelTranslucency() {
super(“Set pixel level translucency”);
//Set size of the JFrame
setSize(250, 250);
//Set lay out
getContentPane().setLayout(new GridLayout(6, 6));
//Call for pixel setting
for(int i = 0; i < 16; i++){
add(new PixelPanel(255 – i * 12));
}
//Set background and other properties
setBackground(new Color(0, 0, 0, 0));
setLocationRelativeTo(null);
setVisible(true);
}
public static void main(String[] args) {
GraphicsEnvironment genv = GraphicsEnvironment.getLocalGraphicsEnvironment();
//Check if window supports translucency
if (genv.getDefaultScreenDevice().isWindowTranslucencySupported(GraphicsDevice.WindowTranslucency.PERPIXEL_TRANSLUCENT)) {
System.out.println(“OS supports translucency”);
new PixelTranslucency();
}
} private class PixelPanel extends JPanel {
private PixelPanel(int pixl) {
super();
setBackground(new Color(0, 0, 255, pixl));
}
}
}
Shaped Windows:
Now we will discuss about another important feature supported by Java7.The shaped window supports all types of shapes whatever be the requirement of the user. This feature helps you to create any shape like circle, triangle, polygon or any possible complex shape. The setShape method of the window class is available for setting the property. But again we have to remember that full screen mode should not be allowed and the operating system supports translucency.
The following example shows the usage of shaped window.
Listing3: The sample code showing the usage of shaped window.
import javax.swing.*;
import java.awt.*;
import java.awt.geom.Ellipse2D;
import java.awt.geom.GeneralPath;
public class JavaShapedWindow extends JFrame {
public JavaShapedWindow() {
super(“Set shaped Window”);
//Set undecorated OFF to get an effect
setUndecorated(true);
//Set size
setSize(new Dimension(250, 250));
//Set polygon properties
Polygon polygon = new Polygon();
polygon.addPoint(0, 100);
polygon.addPoint(50, 0);
polygon.addPoint(100, 100);
//Set the values of the shape
Ellipse2D.Double newCircle = new Ellipse2D.Double(0, 50, 1.0*100, 1.0*100);
//Set general path properties
GeneralPath gpath = new GeneralPath();
gpath.append(polygon, true);
gpath.append(newCircle, true);
//Set the path
setShape(gpath);
//Set box layout
getContentPane().setLayout(new BoxLayout(getContentPane(), BoxLayout.X_AXIS));
add(Box.createHorizontalGlue());
//Create label and set properties
JLabel newlabel = new JLabel(“Java Shaped window”);
newlabel.setForeground(Color.white);
add(newlabel);
add(Box.createHorizontalGlue());
//Set content pane background color
getContentPane().setBackground(Color.cyan);
//Set location
setLocationRelativeTo(null);
setVisible(true);
}
public static void main(String[] args) {
GraphicsEnvironment genv = GraphicsEnvironment.getLocalGraphicsEnvironment();
//Check OS support for this property
if (genv.getDefaultScreenDevice().isWindowTranslucencySupported(GraphicsDevice.WindowTranslucency.PERPIXEL_TRANSPARENT)) {
System.out.println(“OS supports translucency”);
new JavaShapedWindow();
}
}
}
We can also implement a combination of two features like translucency and shaped window. To implement this, call setOpacity method to your frame. The result window will display the combined effect. But we should remember the underlying operating system must support pixel level translucency and pixel level transparency.
Conclusion:
In this article we have covered some new features in java7.The translucency and shaped window features are really interesting and make java UI components more flexible. But at the same time we must remember that the underlying operating system should support all these properties. Otherwise the effect will not be visible. So in the application itself we check the OS support and then implement the effect. The latest feature of java allows the developers to create different types of effect on the UI side. As we all know that look and feel is an important aspect of any UI based application. So the developer is now equipped with tools to make the UI more pleasant and attractive.
When not to use Drools Guvnor?
Posted on | July 7, 2013 | No Comments
Do not use Guvnor in the following scenarios
a) If you have business rules in a database as a part of an application and no new application is needed to manage them. This is a scenario where rules exist as a part of an existing application and there is no need fot future expansion.
b) If you already have a rule management system tailored to your environment, migrating to Guvnor may not be necessary. But at the same time if you think you need more flexibility, expansion,control on business rules in future, then plan for migration.
c) If you are using rules/business logic to solve complex algorithmic problems and they are integral part of the application, don’t go for guvnor.
So the conclusion is, first try to understand your application requirement and then select the proper tool.Otherwise some time people use different tools like Guvnor, rule engine without any proper justification. As a consequence, proper advantages are not realized and the application does not get any benefit.
When to use Drools Guvnor?
Posted on | July 7, 2013 | No Comments
Drools Guvnor is used if you have the following requirement
a) You need to manage versions/deployment of rules.This is a scenario where you have a huge number of business rules and they need to be managed using versioning.
b) You have multiple users at different skill levels and they will access and edit rules.
c) You have lot of business rules and they are growing.And you do not have any existing infrastructure for rules management.
Guvnor can be used as stand alone or integrated with IDE.
What is Reference Architecture and Fit/Gap Analysis
Posted on | May 2, 2013 | No Comments
Description
Before defining Reference Architecture, it is better to have some idea about software architecture. Software architecture has various definitions which are derived from different sources. It ranges from designing, creating and maintaining all the code of large computer systems to incorporating all the functioning of the physical and logical components of enterprise systems as well. It is a combination and interaction of all the components (software and hardware) to make a complete system specific to different business needs.
The Reference Architecture and Fit/Gap analysis is a short of document stating the Reference Architecture to be used as the basis for the current project’s architecture and including the rationale for this decision. It is also a analysis process to understand the reference architecture and its implementation.
A Reference Architecture is a predefined architectural pattern designed for use in particular business and technical contexts.
Source of Reference Architectures within an organization can be any proven framework and asset used in earlier projects. The work product is generic and applies equally to selection of Reference Architectures from other sources.
The most important part of a Reference Architecture is to understand the fitness and gaps found during analysis.
A Reference Architecture Fit/Gap Analysis tabulates the key factors involved in selecting a Reference Architecture:
- Business scenarios
- Business drivers
- Architecture characteristics
Asset selection involves tradeoffs. Reference Architecture Fit/Gap Analysis documents the differences between the desired project architecture and the reference architecture in a statement-of-fit and identifies modifications required for the project.
Purpose
Now let us check the purposes Fit/Gap analysis in Reference Architecture model. The Reference Architecture Fit/Gap Analysis are used for the following
- To assist in determining the scope of a proposed solution development project
- As an input to other architecture work products. Depending on the size of the gaps, the chosen Reference Architecture and associated fit/gap analysis may form the basis for the whole architecture.
- To highlight any gaps where the Reference Architecture does not fit project requirements. These indicate areas of risk or substantial architectural work required in the project. It also helps to select the reference architecture for the proposed project.
Impact of Not Having Reference Architecture
Selecting a Reference Architecture reduces the risk by reusing proven, best-practice solutions. Not applying a standard Reference Architecture increases project effort and cost, and increases the risk of project failure.
The best way to use architectural assets is to start by selecting a Reference Architecture. This will form the context for a more detailed selection of assets and guidance.
Reasons for Not Using Reference Architecture
On asset-based projects, this product should always be produced.
If appropriate Reference Architectures are not available to a project or a project decides not to use a Reference Architecture, then at most the Executive Summary, including reasons why a Reference Architecture was not selected, should be completed. Often even this is unnecessary. A note can be added to the documentation of the choice of work products.
How Fit/Gap analysis is documented
A Reference Architecture Fit/Gap Analysis is basically textual document, with tables assessing the requirements characteristic profile and subject area coverage. The tabular form describes all the required points to get a clear picture of the referred architecture and its fitness in the proposed project.
Executive Summary: Brief statement of selected Reference Architecture and any major issues/risks.
Introduction: Introductory comments about the reference architecture.
Key Drivers: A brief statement of the business objectives or constraints.
Architecture Requirements Checklist: The document consists of a standard set of questions and potential answers regarding architectural requirements, which allows a systematic collection of the architectural requirements for the project. This set is used as a checklist to prompt for customer architectural requirements and associated issues, which are documented in a table, as below. This is a sample table and it can be of a different format according to the standard of the organization.
| Requirement | Reference Architecture Characteristic | Required Value for Project | Risks/Issues/Comments Points to Investigate |
| Requirement name | Relevant range of values for this architecture | Range applicable to the customer | Any gaps, risks, likely future issues, topics for subsequent analysis |
Subject Area Checklist:Each Reference Architecture encompasses several subject areas (sometimes referred to as “domains”). Subject areas are specific areas of concern, for example, the web based delivery channel is a subject area consisting of Web browsers, domain name servers, etc. A simple and clear way of examining the “fit” of a Reference Architecture to customer requirements is to compare it to the subject area coverage. Following is a table for capturing the details.
Subject Area |
Present in Architecture? | Required by Customer? | Issues/Risks/Notes |
| Name of subject area | Yes/No | Yes/No |
Statement of Fit:A summary of the extent of “fit” between the Reference Architecture and customer requirements. Each Reference Architecture is described at several levels of abstraction, and guidance is provided at each level on the characteristics supported. For each area where the Reference Architecture does not meet requirements, describe briefly:
- The unmet requirement. This is very important for selecting the reference architecture.
- The type and extent of delta that would be requisite for the Reference Architecture to meet the requirement. Also mention if it needs to be extended, modified, or receive a new subject area?
- An estimate of the cost and risk of the delta
Development Approach
The following steps should be taken in generating this work product:
- Identify the most likely Reference Architecture candidate. As there are a small number of Reference Architectures, each with a well-defined focus, it should be simple to identify one or two as relevant to the customer’s project goals and future business model.
- Review the description of the asset context, which is provided in both textual and illustration form, with the customer to assess fit. The illustrations, in particular, are a powerful way of checking customer understanding of the proposed approach. During this process, the customer may decide to modify his previously stated requirements, for example, through recognition that the architectural pattern provided by the Reference Architecture is superior to the one he had previously envisaged.
- Use Business Drivers, IT Principles and Requirements lists to characterize customer requirements Check with the customer on whether there are any other important requirements that have not been covered by the standard characteristics.
- Examine the fit of the requirements with the characteristics provided by the Reference Architecture at various levels of abstraction.
- From the asset base, identify the subject areas covered by the Reference Architecture. Compare these with the subject areas required by the customer.
- Analyze the fit between the Reference Architecture and the customer requirements. Document all gaps, and make initial assessments with the customer on how these shortcomings will be resolved. This may involve:
- Extending the architecture (e.g., linking to an additional subject area)
- Modifying the Reference Architecture (i.e., changing its internal structure or behavior)
- Document the rationale for each of these decisions in terms of assumptions, business justification, risks, etc.
- Make an overall assessment of viability and risk based on the information available at this point.
Validation and Verification
Perform the following checks:
- Verify that the customer has approved the criteria used to compare and choose the Reference Architectures.
- Verify that all major requirement areas of the customer have been considered.
- Verify that critical dependencies of the architecture in the target environment can be met.
- Validate any tradeoffs being made with the customer and Asset Providers with the following criteria.
- Verify that the architecture is resilient to anticipated changes.
How to set up Hadoop on single node and multi node?
Posted on | March 5, 2013 | No Comments
We will describe Hadoop setup on single node and multi node. The Hadoop environment setup and configuration will be described in details. First you need to download the following software (rpm).
- Java JDK RPM
- Apache Hadoop 0.20.204.0 RPM
A) Single node system Hadoop setup
1) Install JDK on a Red Hat or CentOS 5+ system.
$ ./jdk-6u26-linux-x64-rpm.bin.sh
Java is installed and set JAVA_HOME to /usr/java/default
2) Install Apache Hadoop 0.20.204.
$ rpm -i hadoop-0.20.204.0-1.i386.rpm
3) Setup Apache Hadoop configuration and start Hadoop processes.
$ /usr/sbin/hadoop-setup-single-node.sh
The setup wizard will guide you through a list of questions to setup Hadoop. Hadoop should be running after answering ‘Y’ to all questions.
Create a user account on HDFS for yourself.
$ /usr/sbin/hadoop-create-user.sh -u $USER
B) Multi-nodes Hadoop setup
1) Install both the JDK and Hadoop 0.20.204.0 RPMs on all nodes
2) Generate hadoop configuration on all nodes:
$ /usr/sbin/hadoop-setup-conf.sh \
--namenode-url=hdfs://${namenode}:9000/ \
--jobtracker-url=${jobtracker}:9001 \
--conf-dir=/etc/hadoop \
--hdfs-dir=/var/lib/hadoop/hdfs \
--namenode-dir=/var/lib/hadoop/hdfs/namenode \
--mapred-dir=/var/lib/hadoop/mapred \
--datanode-dir=/var/lib/hadoop/hdfs/data \
--log-dir=/var/log/hadoop \
--auto
Where ${namenode} and ${jobtracker} should be replaced with hostname of namenode and jobtracker.
3) Format namenode and setup default HDFS layout.
$ /usr/sbin/hadoop-setup-hdfs.sh
4) Start all data nodes.
$ /etc/init.d/hadoop-datanode start
5) Start job tracker node.
$ /etc/init.d/hadoop-jobtracker start
6) Start task tracker nodes.
$ /etc/init.d/hadoop-tasktracker start
7) Create a user account on HDFS for yourself.
$ /usr/sbin/hadoop-create-user.sh -u $USER
C) Setup Environment for Hadoop
$ vi ~/.bash_profile
In INSERT mode set path for JAVA_HOME
Export JAVA_HOME
Save file by clicking esc:wq
Run the .bash_profile
$ source ~/.bash_profile
Set JAVA_HOME path in Hadoop Environment file
$ /etc/hadoop/hadoop-env.sh
D) Configuration for Hadoop
Use the following:
conf/core-site.xml:
| <configuration> |
| <property> |
| <name>fs.default.name</name> |
| <value>hdfs://localhost:9000</value> |
| </property> |
| </configuration> |
conf/hdfs-site.xml:
| <configuration> |
| <property> |
| <name>dfs.replication</name> |
| <value>1</value> |
| </property> |
| </configuration> |
conf/mapred-site.xml:
| <configuration> |
| <property> |
| <name>mapred.job.tracker</name> |
| <value>localhost:9001</value> |
| </property> |
| </configuration> |
E) Hadoop Commands
$ hadoop
$ hadoop namenode –format (Format the namenode, If ask to
answer press ‘Y’)
$ hadoop namenode (Start the namenode)
$ find / -name start-dfs.sh (find the file in directory)
$ cd usr/sbin (Go to respective directory directly)
$ start-dfs.sh
$ start-mapred.sh
$ hadoop fs –ls / (Shows the HDFS root folder)
$ hadooop fs –put input/file01 /input/file01 (Copy local input/file01 to
HDFS root /input/file01)
How to use drools decision table in spreadsheet format to execute rules?
Posted on | February 13, 2013 | 4 Comments
The following example code will describe the process of using decision table to execute rules.There are two ways
a) Use the spread sheet APIs directly to execute the rules.
b) Convert the excel spread sheet into a drl file and then execute the rules.
I will describe the second process (b).
** The main java class is ‘DroolExcelDemo.java’.First it will convert the excel sheet into a (.drl) file and then execute the rules.
package DROOLS;
import java.io.BufferedWriter;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.FileWriter;
import java.io.IOException;
import java.io.InputStream;
import java.util.Collection;
import org.drools.KnowledgeBase;
import org.drools.KnowledgeBaseFactory;
import org.drools.builder.KnowledgeBuilder;
import org.drools.builder.KnowledgeBuilderFactory;
import org.drools.builder.ResourceType;
import org.drools.decisiontable.InputType;
import org.drools.decisiontable.SpreadsheetCompiler;
import org.drools.definition.KnowledgePackage;
import org.drools.event.rule.DebugAgendaEventListener;
import org.drools.event.rule.DebugWorkingMemoryEventListener;
import org.drools.io.ResourceFactory;
import org.drools.runtime.StatefulKnowledgeSession;
/**
* This class will create a drl file from excel sheet
* and then execute the rules.
*/
public class DroolExcelDemo {
public static final void main(final String[] args) {
// Create knowledge builder
final KnowledgeBuilder kbuilder = KnowledgeBuilderFactory.newKnowledgeBuilder();
// Create drl file from excel sheet
InputStream is =null;
try {
is= new FileInputStream(“D:/Workspace/DroolsDemo/src/DROOLS/DroolExcel.xls”);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
// Create compiler class instance
SpreadsheetCompiler sc = new SpreadsheetCompiler();
// Compile the excel to generate the (.drl) file
StringBuffer drl=new StringBuffer(sc.compile(is, InputType.XLS));
// Insert dialect value into drl file
drl.insert(drl.indexOf(“DROOLS”)+40,”dialect \”mvel\”"+”\n”);
// Check the generated drl file
System.out.println(“Generate DRL file is showing below–: “);
System.out.println(drl);
// writing string into a drl file
try {
BufferedWriter out = new BufferedWriter(new FileWriter(“D:/Workspace/DroolsDemo/src/DROOLS/RuleFile.drl”));
out.write(drl.toString());
out.close();
}
catch (IOException e){
System.out.println(“Exception “);
}
// Wait before using the drl file in the next section.
try {
Thread.sleep(10000);
} catch (InterruptedException e) {
e.printStackTrace();
}
// End creation of drl file from excel sheet
// Using DRL file
kbuilder.add(ResourceFactory.newClassPathResource(“RuleFile.drl”, DroolMessage.class ), ResourceType.DRL );
// Check the builder for errors
if ( kbuilder.hasErrors() ) {
System.out.println(“kbuilder has errors”);
System.out.println( kbuilder.getErrors().toString());
}
// get the compiled packages (which are serializable)
final Collection pkgs = kbuilder.getKnowledgePackages();
// add the packages to a knowledgebase (deploy the knowledge packages).
final KnowledgeBase kbase = KnowledgeBaseFactory.newKnowledgeBase();
kbase.addKnowledgePackages(pkgs);
// Create stateful session
final StatefulKnowledgeSession ksession = kbase.newStatefulKnowledgeSession();
// Set event listeners
ksession.addEventListener(new DebugAgendaEventListener());
ksession.addEventListener(new DebugWorkingMemoryEventListener());
// Create message text
DroolMessage messagetxt = new DroolMessage();
messagetxt.setMessage(“FlightNumber”);
DroolMessage messagetxt1 = new DroolMessage();
messagetxt1.setMessage(“FlightCode”);
// Insert into session and fire rules
System.out.println(“insert into session”);
ksession.insert(messagetxt);
ksession.insert(messagetxt1);
System.out.println(“before firing rules”);
ksession.fireAllRules();
System.out.println(“after firing rules”);
ksession.dispose();
System.out.println(“after dispose”);
}
}
* *The second file is a ‘DroolMessage.java’ pojo class. It holds the values set by the main java class described above.
package DROOLS;
/**
* This is a POJO for messages.It will be used by the drl file.
* The message values will be set from the DroolExcelDemo class
*/
public class DroolMessage {
// Create variable
private String message;
// Constructor
public DroolMessage() {
System.out.println(“initializing Message”);
}
// Setter and getter methods
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
// test method to test the rule execution
public void test(String test) {
System.out.println(“Parsing start and end position: “+test);
}
}
** Following is a rule file generated by the ‘DroolExcelDemo.java’ class.
package DROOLS;
//generated from Decision Table
dialect "mvel"
import DroolMessage;
// rule values at B12, header at B7
rule "DroolExcelRule_12"
when
msg:DroolMessage(message == "FlightNumber")
then
msg.test("1,6");
end
// rule values at B13, header at B7
rule “DroolExcelRule_13″
when
msg:DroolMessage(message == “FlightCode”)
then
msg.test(“7,8″);
end
** Following is the input excel based rule file to the ‘DroolExcelDemo.java’ class above.
What is Apache Sqoop and how to use it to import/export data from Hadoop Distributed File System?
Posted on | February 11, 2013 | No Comments
Apache Sqoop is a tool used for transferring data from/to Hadoop distributed file system. Hadoop architecture can process BIG data and store it in HDFS. But if we want to use that data then we need to use some tool to import/export it efficiently. Apache Sqoop allows easy import and export of data from structured data stores such as relational databases, enterprise data warehouses, and NoSQL systems.
Apache Sqoop is very important when we think about using Hadoop for analytics and data processing.The two main aspects which Sqoop address are
a) Loading bulk(production) data into Hadoop.
b) Accessing bulk data from map/reduce applications running on large clusters.
Earlier we used to write/use scripts to import/export data between different systems.But this process is inefficient and does not ensure data consistency, accuracy and other critical points.
Sqoop uses straight forward mechanism to transfer data.The entire dataset is splitted into slices and each slice is a map-only job.Now each map-only job is responsible for transferring one slice of the data-set.
As we have discussed, Sqoop can be used to import data from a RDBMS into HDFS.The input to the import process is a database table and Sqoop reads table row by row into HDFS.The input process is performed in parallel so the output will be multiple files.These output files can be text files or other type of files containing serialized data.
There is a by-product of the Sqoop import process.It is a Java class which can encapsulate one row of the imported table.This Java class is used by the Sqoop itself durng import process.The source code of this by-product Java class is also available for customized use.
After processing the imported data, it can be exported to any relational database using Sqoop. Sqoop will read a set of delimited text files from HDFS (in parallel) and insert them as new rows to the
target table.Now these data is available for consumtion by the external applications.
Sqoop also provides some command utilities to get information about the databaes on which it is working.The list of database schemas, tables can also be viewed using Sqoop commands.Sqoop also provides primitive SQL execution shell.
Sqoop operations like import,export,code generation etc can be customized.For import, row ranges/columns can be specified.The delimiters, escape characters for file based representation can also be changes as per the requirement.The package/class name of the generated code can also be customized to meet the application requirement.
Sqoop connectors are another important part of the tool.Connectors are plugin components built on Sqoop’s extension framework.These connectors can be added to any Sqoop installation and then data can be transferred between Hadoop and external store.
Sqoop comes with default connectors for various popular databases such as MySQL, PostgreSQL, Oracle, SQL Server and DB2.Sqoop also includes a generic JDBC connector which can be used to connect to any database accessible via JDBC.
To conclude this discussion we can say that Sqoop can be used to transfer large datasets between Hadoop and external datastores efficiently.Beyond this, Sqoop also offers many advanced features like different data formats,compression,customization,working with queries etc.
What is Hadoop Streaming?
Posted on | December 20, 2012 | No Comments
Ans : Hadoop streaming is a powerful utility which comes with Hadoop distribution.The basic concept of Hadoop framework is to split the job,process it in parallel and then join it back to get the end result.So there are two main components involved in this framework.
a) Map application
b) Reduce application
The Hadoop streaming utility allows you to write Map/Reduce applications in any language that is capable of working with STDIN and STDOUT.
What is Map/Reduce in Hadoop?
Posted on | December 20, 2012 | No Comments
Ans : Processing vast amount of data (multi-terabyte data-sets) is a major concern in real life projects.As the size of data is increasing day by day, applications are finding it difficult to process it in a reliable,secured and fault-tolerant way.
Here comes the Hadoop Map/Reduce framework.It helps to write the applications easily which process vast amount of data keeping all the concerns in mind.The biggest advantage is, Map/Reduce allows parallel processing on large clusters of commodity hardware.
The main concepts behind Map/Reduce is
a) Split the job into independent tasks (known as Map tasks).
b) Process the tasks in a parallel manner.
c) Sort the output of the maps and send them as input to the reduce task (known as Reduce tasks).
So it is basically parallel processing of chunks and then joining them back to get the end result.The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.
Note : Commodity hardware is a term for affordable devices that are generally compatible with other such devices.
How to increase the performance of a JMeter test?
Posted on | December 18, 2012 | 1 Comment
A: Following are some techniques which can be applied to increase the performance of a JMeter test.
1) Reduce number of concurrent threads :The hardware and server capability limits the number of threads which can run effectively with JMeter.If lot of threads are working simultaneously then each thread has to wait more to get CPU and timing information will be inflated. So it is always better to limit the number of threads.
2) Use proxy server: Use proxy server to filter out all requests which are not important for your testing. Use exclude pattern to filter out unnecessary data.All the files are having their own extensions (like .php, .jsp, .jpeg etc). So use those extensions to set up your exclude pattern.
3) Reduce resource requirements : Following are some measures which can be followed to reduce resources
a) Use non-GUI mode.
b) Use CSV instead of XML output.
c) Use as few listeners as possible.
d) Do not use functional mode.
e) Reduce the number of samplers.