Java Latency Contributor

1.      The biggest contributor is garbage collector

2.      In-process locking and thread scheduling

3.      I/O

4.      Application algorithmic inefficiencies

Who created Osama Bin Laden?

 

Integrating Centralized Logging System to OpenNMS for Alerts

Once the solution is based on SOA, there can be a need to have a centralized logging facility where each of the service would be using log agents to send logs to a cluster of centralized log server. This can be done via various manners using log4J framework by making use of one of the following appenders:

1. SysLogD
2. JMSAppenders

JMSAppender could be used within Log agents to send logs to a messaging bus from where message could be retrieved by store agents on centralized log server.

Further to that, the message once retrieved from the Message bus would be watched for warnings or exceptions for purpose of generating alerts which could be informed to admin via SMS or email.

This can be done by integrating centralized log server with OpenNMS. This is done based on following:

1. The incoming messages is watched for exception/warning messages.
2. Once these messages are found, the event is created and send as an XML document to OpenNMS eventD plugin on a TCP socket. EventD listens on port such as 5817 by default. This port needs to be opened for external system such as centralized log server to send message using Sockets.

Following class could be used to send message using Socket:

he Code

   package com.company.onms.monitor;

 import java.io.IOException;
   import java.io.PrintWriter;
   import java.net.Socket;
   import java.net.UnknownHostException;
   import java.text.DateFormat;
   import java.util.Calendar;
   import java.util.Date;
   import java.util.HashMap;
   import java.util.Map;
   /**
    *
    * @author apaxson
    */
   public class OnmsEvent {
       private String uei;
       private String source;
       private int nodeid;
       private String time;
       private String host;
       private Map parms = new HashMap();
       public OnmsEvent(String uei, int nodeid, String host, String source) {
           setUei(uei);
           setNodeid(nodeid);
           setHost(host);
           setSource(source);
           Date currentTime = Calendar.getInstance().getTime();
           DateFormat dformat = DateFormat.getDateTimeInstance(DateFormat.FULL, DateFormat.FULL);
           dformat.setTimeZone(TimeZone.getTimeZone("GMT"));
           setTime(dformat.format(currentTime));
       }
       protected String getUei() {
           return uei;
       }
       protected void setUei(String uei) {
           this.uei = uei;
       }
       protected String getSource() {
           return source;
       }
       protected void setSource(String source) {
           this.source = source;
       }
       protected int getNodeid() {
           return nodeid;
       }
       protected void setNodeid(int nodeid) {
           this.nodeid = nodeid;
       }
       protected String getTime() {
           return time;
       }
       protected void setTime(String time) {
           this.time = time;
       }
       protected String getHost() {
           return host;
       }
       protected void setHost(String host) {
           this.host = host;
       }
       public void addParm(String key, String value) {
           parms.put(key, value);
       }
       @SuppressWarnings(value = "unchecked")
       protected Map<String, String> getParms() {
           return parms;
       }
       protected String toXml() {
           StringBuffer data = new StringBuffer();
           data.append("<log>");
           data.append("<events>");
           data.append("<event>");
           data.append("<uei>" + getUei() + "</uei>");
           data.append("<source>" + getSource() + "</source>");
           data.append("<nodeid>" + getNodeid() + "</nodeid>");
           data.append("<time>" + getTime() + "</time>");
           data.append("<host>" + getHost() + "</host>");
           data.append("<parms>");
           // Cycle through each parameter
           for (Map.Entry<String, String> e : getParms().entrySet()) {
               data.append("<parm>");
               data.append("<parmName>");
               data.append("<![CDATA[" + e.getKey() + "]]></parmName>");
               data.append("<value type=\"string\" encoding=\"text\"><![CDATA[" + e.getValue() + "]]></value>");
               data.append("</parm>");
           }
           data.append("</parms>");
           data.append("</event>");
           data.append("</events>");
           data.append("</log>");
           return data.toString();
       }
       /**
        * TODO Remember to change the hardcoded address 2.3.4.5 to your address for onms.
        *  Preferrably, place this in a config file somewhere
        */
       public void sendEvent(String xmlData) {
           //TODO check to make sure xmlData isn't null before creating socket
           //TODO make some useful logging functions here, rather than println
           Socket socket = null;
           try {
               System.out.println("Sending data");
               socket = new Socket("2.3.4.5", 5817);
               PrintWriter out = new PrintWriter(socket.getOutputStream(), true);
               out.println(xmlData);
               System.out.println("Sending data complete");
           } catch (UnknownHostException ex) {
               System.out.println("Unknown host");
           } catch (IOException ex) {
               System.out.println("I/O Exception while creating socket");
           } finally {
               try {
                   socket.close();
               } catch (IOException ex) {
                   System.out.println("I/O Exception when closing socket");
               }
           }
       }
   }

The Usage

Okay, so to use this new class, you create the object, and set the data. As such:

       OnmsEvent event = new OnmsEvent("uei.company.com/application/logFileTableLoadError",26,"test.host.com", "ServerLogMonitor");
       event.addParm("file", "logfile001.log");
       event.addParm("content","Disregard... testing only");
       event.sendEvent(event.toXml());

Lessons learnt for scaling Java EE applications

Check out following link for detailed description:

1. Use
   share nothing clustering architectures
2. Use
   scalable session replication mechanisms
3. Use
   collocated deployment instead of distributed one
4. Shared
   resources and services
5. Parallel Processing

funpiper.com - Plan Events and Parties, Broadcast and Share

I am pondering upon funpiper.com, an event management platform to take to next level with GroupOn.com like concept..anyone interested to invest!

Imagine an automated engine helping you to meet your friends with you making very little effort like answering yesy/no for a couple of times!!! Let me know if it sounds interesting to you. Write me at ajitesh at @ at gmail dot com

funpiper.com - Plan Events and Parties, Broadcast and Share

My Two Cents on SOA & Startups

An organization needs to spend a lot (COST) to achieve a true SOA based architecture just because SOA is not only limited to Service and their interactions. Following are areas which needs to be considered for achieving an “Agile SOA” platform:

- Service

- Governance

- Architecture

- Process

- People

- Engagement & Delivery Operation

Trust me, really speaking, it would not be very SMART & INTELLIGENT if we tell to our clients that we offer SOA based solutions. WE COULD SIMPLY BE CAUGHT UNAWARE! Simply because if we get an SOA expert across the table, we would simply find ourselves struggling with some real tough questions on SOA governance, process, service discovery, Delivery & Operations Management (ITIL) aspects of SOA. So, we rather should take caution (MAKE RIGHT CHOICE OF WORDS) when we speak to our clients that we offer SOA-based solution as it could simply backfire!

And, this is very natural that an organization cannot achieve “Agile SOA” status in first few years as no organization (especially start-ups) could IGNORE COST-EFFECTIVENESS of the solutions. Or, it will simply be DEAD!

Its very natural that in the initial years, organization would want to be happy with SOA based solution rated as “Ad-hoc SOA” status. That said, pl feel free to ask for various transitions for SOA and roadmap that an organization need to take to transition from “Ad-hoc SOA” to “Agile SOA” status.

Is the Java Dynasty Seeing the Beginning of the End?

This is very interesting to see that various blogs, articles have started writing about Java and its end. Well, as they say, ALL Good Things come to an end. Would this be applied to Java? I don't want to agree. But, In my past experience, I have found myself moving from C, C++ to Java and then adopt PHP and then started exploring fairly recent JVM languages such as Groovy, Scala, JRuby etc.

IT Poll: Is the Java Dynasty Seeing the Beginning of the End?

JBoss AS 6 out with JEE 6 Web Profile Specs

Following are some of key highlights of JBoss AS 6:

1. JEE 6 Web Profile Spec (JSP, EJB3, JPA, JSF, CDI). CDI stands for Contexts and Dependency Injection (JSR 299)

2. RESTEasy 2.1.0, an implementation of JAX-RS 1.1

3. Hornet 2.1.2

4. JBoss Web Services CXF

Also, Infinispan is integrated with JBoss AS 6 and is default distributed cache provider. Cache is JSR 107 compliant.

Also is delivered Mod_Cluster which dynamically discover and register to front-end apache http servers.

Performance management and Monitoring strategy for SOA

Following can be performance management and monitoring strategy that an SOA based solution architecture could adopt. Please share your thoughts.

Performance Management

- Continual performance evaluation (in TPS) of various services by making use of different stress scenarios such as following:

1. Simple strategy: The Simple Strategy runs the specified number of requests with the specified delay between each run to simulate a "breathing space" for the server.
2. Fixed rate strategy: Fixed rate strategy guarantees a fixed number of requests in a given time period.
3. Variable load strategy: This strategy related to vary the load. Following are key strategies for variable load:

• Variance: This varies the number of requests over time in a “sawtooth” manner as configured; set the Interval to the desired value and the Variance to the how much the number of requests should decrease and increase.
• Burst: this strategy is specifically designed for Recovery Testing and takes variance to its extreme; it does nothing for the configured Burst Delay, then runs the configured number of requests for the “Burst Duration” and goes back to sleep.
• Thread: This lets you linearly change the number of requests from one level to another over the run of the stress test. It’s main purpose is as a means to identify at which level certain statistics change or events occur, for example to find at which request count, the maximum TPS or BPS can be achieved or to find at which request count, at which functional testing errors start occurring.
• Grid: this strategy allows to specifically configure the relative change in number of requests over time, its main use for this is more advanced scenario and recovery testing, where you need to see the services behaviour under varying loads and load changes.
• Script: the script strategy is the ultimate customization possibility; the script you specify is called regularly and should return the desired number of requests at that current time.

- Tools such as SOAP-UI, GRINDER or Apache JMeter could be used for implementing above stress strategies.

- Java code instrumentation using custom logging framework for collecting timing for various IO (Disk/Network) tasks in any APIs.

- Distributed agents-collector based architecture to store performance metrics on a cluster of centralized log servers. This could make use of framework such as Flume (Cloudera) or Scribe (Facebook) or NIO based SocketServer implementations on centralized log server for log aggregation.

- Integration with monitoring tool such as NAGIOS for active monitoring on service performance, degradation etc.

- Web services to expose the performance metrics to internal and external consumers.

- Usage of performance metrics data for business intelligence. An ETL tool would be used for transferring data from log servers to BI server.

Monitoring

- Performance metrics monitoring based on log messages arriving in centralized log servers. This could make use of framework such as Flume (Cloudera) or Scribe (Facebook) or NIO based SocketServer implementations on centralized log server for log aggregation.

- Services and Server configuration monitoring for tracking configuration changes.

- Custom alerts for sending notifications for scenarios such as configuration changes , service degradation based on performance metrics etc.

- Web services to expose the monitoring information to internal and external consumers

- Integration with monitoring tool such as NAGIOS for active monitoring on service performance, degradation etc.

- Usage of monitoring information for business intelligence. An ETL tool would be used for transferring data from log servers to BI server.

Top five scripting languages on the JVM

1. Groovy -- Powerful, high-level, enterprise-friendly

2. JRuby -- a better Ruby than Ruby?

3: Scala -- fast, multiparadigm programming

4: Fantom -- Java and .Net plus performance

5: Jython -- Python for the JVM

Top five scripting languages on the JVM | Developer World - InfoWorld

Some cool features of Tomcat 7

1. Configuration management for Web applications in which not everything needs to reside anymore in web.xml. One could use annotation and break config instructions in multiple files.

2. Benefits of servlet 3.0 which includes cookie management and asynchronous thread support that should increase the number of connections Tomcat can handle

Tomcat 7 finalized - JavaWorld

LazyLogger for Optimal Logging

Logging has always been a concern when we are talking about application performance. What to log and which appender to use and when to flush the logs and where to log etc are some of the key concerns. One of the concern has been put a check such as isLoggingEnabled at a particular level. This is where something such as LazyLogger utility could prove very useful.

Following method does the magic in the LazyLogger:

public void log(Level level, Object... params) {
        // Check logging level before continuing
        if (logger.isLoggable(level)) {
            String message;
            if (params != null) {
                if (params.length == 1) {
                    message = String.valueOf(params[0]);
                } else {
                    StringBuilder buf = new StringBuilder();
                    for (Object param : params) {
                        buf.append(param);
                    }
                    message = buf.toString();
                }
            } else {
                message = "null";
            }
            logger.log(level, message);
        }
    }

Tip: Lazy Loading for Efficient Java Logging - Developer.com

Evaluating AppDynamics Lite

1. Start adlite-viewer by following command:

java -jar adlite-viewer.jar

2. Place following command in catalina.bat and start

set JAVA_OPTS=%JAVA_OPTS% -javaagent:E:\apps\AppDynamicsLiteBeta\AppServerAgentLiteBeta\javaagent.jar

3. Enter the login/password (default: admin/admin) to AppDynamics Lite Web App Login screen

However, I couldn't for some reason get something to be displayed on the screen. I even tried to configure what they call as business transactions. However, it didnt work.

Managing SOA based solutions from operation perspective

When we are talking about multiple apps, and caching layer being put to interact with service (sounds like a data service), I am not sure if one team & one codebase could be the most suitable answer to achieve the best results.

This looks to me as an architecture based on SOA. There is a whole notion of SOA governance, related processes and tools which could be made use of to get the best results from SOA based solutions.

InfoQ: Squid Wrangling

Design considerations: Logging framework for Distributed Service

Model 1: Customizing Log4J & Perf4J Framework

Usage of asynchronous Log4J appenders (AsyncAppender/JMSAppender or custom appenders) to push the log data as and when they arrive to a a JMSProvider from where the log messages are picked up and stored in the file-based storage and databases appropriately as per the logging rules & policies.

In above, the appenders may be termed as agent nodes, the JMSProvider may be termed as collector nodes and the file storage or database may be termed as storage nodes/repositories.

Key design considerations:

1. Asynchronous

2. a) Customization required to store the log data locally in case of failure of MQ storage; b) process the local messages when MQ comes live.

3. JMSAppender needs to be evaluated in the high stress.

4. Scalability of collector and storage need to be evaluated.

5. Clustering considerations need to be evaluated

5. Perf4J used to generate performance reports

Model 2: Flume & Log4J/Perf4J framework

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is

centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic application

Key Design Considerations:

1. Usage of FileAppenders with appropriate flush policy may lead to enhanced performance.

2. Reliable as Flume a well-defined recovery strategy if agent node crashes, and store on failure if collector/storage nodes crashes.

3. Well defined clustering considerations available for scaling out agent, collector and storage nodes.

http://archive.cloudera.com/cdh/3/flume/UserGuide.html

http://archive.cloudera.com/cdh/3/flume/Cookbook.html

Model 3: Scribe & Log4J/Perf4J framework

Scribe is a server for aggregating streaming log data. It is designed to scale to a very large number of nodes and be robust to network and node failures. There is a scribe server running on every node in the system, configured to aggregate messages and send them to a central scribe server (or servers) in larger groups. If the central scribe server isn’t available the local scribe server writes the messages to a file on local disk and sends them when the central server recovers. The central scribe server(s) can write the messages to the files that are their final destination, typically on an nfs filer or a distributed filesystem, or send them to another layer of scribe servers.

Key Design Considerations:

1. Usage of FileAppenders with appropriate flush policy may lead to enhanced performance

2. Reliability: The scribe system is designed to be robust to failure of the network or any specific machine, but does not provide transactional guarantees. If a scribe instance on a client machine (we’ll call it a resender for the moment) is unable to send messages to the central scribe server it saves them on local disk, then sends them when the central server or network recovers. To avoid overloading the central server upon a restart, the resender waits a random time between reconnect attempts, and if the central server is near capacity it will return TRY_LATER, which tells the resender to not attempt another send for several minutes.

3. Well defined clustering considerations available for scaling out agent, collector and storage nodes.

4. Architecture is based on scribe agent lying on every server and these scribe agents sending logs to scribe server.

Model 4: Custom Log4J AsyncAppender/SocketAppenders and NIO SocketServer (Non-blocking IO SocketServer)

Using this model, AsyncAppender sends message to SocketAppenders(agent) that write the data to SocketServer (collector) which writes the data to appropriate storage nodes (file based or DB).

Key Design Considerations:

1. a) Customization required to store the log data locally in case of failure of SocketServer; b) process the local messages when MQ comes live.

2. Scalability of collector and storage need to be evaluated.

3. Clustering considerations need to be evaluated

Following could be one NIO SocketServer implementation which is very similar to what I am speaking about. The socketServer from project Voldemart could be customized to meet the centralized logging requirements.

http://sna-projects.com/blog/2009/08/introducing-the-nio-socketserver-implementation/

On all of the above desigm models, one thing is pretty clear that the key component of architecture is based on following:

- Set of Agent nodes residing on the web/app servers (physical nodes) that host one or more services

- Set of collector nodes which is logically separated from agent node but may reside on same or different physical nodes as web/app servers.

- Set of storage nodes which are basically file based or database repositories

My recomendation in order of preference:

Model 4 which involved socket based communication if some staff could be allocated for customization if it is found to be good. One other advantage would be that framework could be customized as and when required.

Model 2/3: Flume/Scribe & Log4J/Perf4J framework

Model 1: Customizing Log4J & Perf4J Framework to work with AsyncAppender/JMSAppender

Method Invocation in Java

While listening to one of the webinars, I revised on following different kind of method invocations at bytecode level:

invoke_virtual: allows to call polymorphic method

invoke_special:allows to call private methods

invoke_static: allows to call static method

invoke_interface: allows to call interface methods

All of the other are static typed and an object type needs to be known.

With Java 7, another method is supported which invoke_dynamic

How Clojures are implemented in JVM languages such as groovy, scala:

Using anonymous inner classes, although these are not most efficient way of doing the same.

Distributed Logging Service - Research

Binary distribution of Facebook Scribe for Window

Is there any binary distribution of scribe for windows?. I would want to run scribe server in a windows machine without using cygwin. If there is no binary distribution . Is it possible to build one?

AsyncAppender vs JMSAppender

I was also thinking that I could use a JMSAppender. Now, I know that this is not truly asynchronous, because each logging request is not actually performed in a separate thread. But since performance is usually degraded due to IO (i.e. writing log to a file), this solution may still be somewhat asynchronous.

How to attach multiple appenders to AsyncAppender so that certain application log data should go to FileLogger and certain should go to JMSAppender..??? Should I need to specify in log4j.properties file if so pls give details how to do the same...??????

http://www.theserverside.net/discussions/thread.tss?thread_id=18735

Below defines how to add multiple file appenders in Log4J properties file and then sending logs to appropriate appender from the application code:

You could define two loggers in the log4j.properties file like this.

# Declare the appenders

log4j.logger.app.file = DEBUG, fileappender

log4j.logger.app.jms = DEBUG, jmsappender

# Now Specify the properties for the appenders, for ex:

#File - Refer to Log4J for additional properties

log4j.appender.fileAppender=org.apache.log4j.RollingFileAppender

log4j.appender.fileAppender.File=Log.xml

log4j.appender.fileAppender.MaxFileSize=100KB

#JMS - Refer to Log4J for additional properties

log4j.appender.jmsappender=org.apache.log4j.net.JMSAppender

log4j.appender.jmsappender.TopicBindingName=LOGGING_TOPIC

And finally from your application, refer to these appenders based on where you want to log your messages to. Ex:

# File logging

Logger fileLogger= Logger.getLogger( "app.file" );

fileLogger.debug("This will be logged to file" );

# JMS logging

Logger jmsLogger= Logger.getLogger( "app.jms" );

jmsLogger.debug("This will be put in queue for specified topic" );

Another Logging Framework - Logback Project

Logback is intended as a successor to the popular log4j project, picking up where log4j leaves off.

Logback's basic architecture is sufficiently generic so as to apply under different circumstances. At present time, logback is divided into three modules, logback-core, logback-classic and logback-access.

The logback-core module lays the groundwork for the other two modules. The logback-classic module can be assimilated to a significantly improved version of log4j. Moreover, logback-classic natively implements the SLF4J API so that you can readily switch back and forth between logback and other logging frameworks such as log4j or java.util.logging (JUL).

The logback-access module integrates with Servlet containers, such as Tomcat and Jetty, to provide HTTP-access log functionality. Note that you could easily build your own module on top of logback-core.

Perf4J Framework

http://perf4j.codehaus.org/devguide.html

StopWatch could be used to log the timings very easily

StopWatch stopWatch = new LoggingStopWatch();

try {

// the code block being timed - this is just a dummy example

long sleepTime = (long)(Math.random() * 1000L);

Thread.sleep(sleepTime);

if (sleepTime > 500L) {

throw new Exception("Throwing exception");

}

stopWatch.stop("codeBlock2.success", "Sleep time was <>

} catch (Exception e) {

stopWatch.stop("codeBlock2.failure", "Exception is: " + e);

}

Perf4J used for logging performancee statistics (http://perf4j.codehaus.org/devguide.html)

Following are some of the key features:

# Adding Timing Statements

# Parsing Log Files to Generate Performance Statistics

# Using the log4j Appenders to Generate Real-Time Performance Information

* Writing Graphs with the GraphingStatisticsAppender

* Exposing Performance Statistics as JMX Attributes

# Exposing Performance Graphs in a Web Application

# Unobtrusive Logging with @Profiled and AOP

* Adding the Profiled Annotation to Method Declarations

* Using AspectJ Load-Time Weaving to Integrate Timing Aspects at Run Time

* Using the AspectJ Compiler to Integrate Timing Aspects at Compile Time

* Using Spring AOP to Integrate Timing Aspects

* Using EJB Interceptors to Integrate Timing Aspects

IPL 4 - Google Search

Now that the likes of Gautam Gambhir and Yusuf Pathan is part of KKR in IPL4 season, would Shahrukh team be able to win the championship? Only time would tell if Shahrukh KKR management has done the right thing by not taking Saurabh Ganguly.

I am sure Saurabh Ganguly himself would be watching KKR performance in IPL 4 this year with a lot of curiosity from various perspective. Also thing to see is how much would be west bengal favour KKR without dada.