Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

and many more ....

Featured FREE Whitepapers

What's New Here?


JBoss BRMS Best Practices – tips for your BPM Process Initialization Layer

I have posted some articles in the past on migration strategies, taken closer looks at process layers and provided some best practices for jBPM, both touching on very specific parts of your BPM strategies. I wanted to revisit the topic of best practices but then on an Intelligent, Integrated Enterprise level where we talk about getting control over your business processes with JBoss BRMS. Introduction To start with we need to take a closer look at the landscape and then peel back the layers like an onion for a closer look at how we can provide BPM projects that scale well. Figure 1 shows that there are several component layers where we will want to focus our attention:Process Initialization Layer Process Implementation Layer Process Repository Tooling for business users & developers Console, reporting & BAM dashboards Process Interaction LayerFigure 1: Enterprise BPM landscape.The process initialization layer will be covered in this article, where I present some best practices around you, your customer and how processes are started. The process implementation layer is where the processes are being maintained, with help from the process repository, tooling, business users and developers that design them. Here you will also find the various implementation details, such as domain specific extensions to cover specific node types within our projects. Best practices in this layer will be covered at a later time.The console, reporting and BAM dashboard components are the extended tooling used in projects to provide business value or information that can be used to influence business decisions. Best practices in this area will be covered at a later time.Finally, the process interaction layer is where you processes will connect to all manner of legacy systems, back office systems, service layers, rules systems even third party systems and services. Best practices in this area will be covered in a later article.Figure 2: Getting started.Process Initialization Layer Taking a look at how to initialize your processes, I want to provide you with some of the best practices I have seen used by larger enterprises over the years. There seems to be a main theme of gathering the customer, user or system data that is needed to start your process, then just injecting it by the startProcess call. This can be embedded in your application via the BRMS jBPM API call, make use of the RESTful service or via a standard java web service call. No matter how you gather the data to initialize your process instances, you might want to think about how you would want to scale out your initialization setup from the beginning. Often the initial projects are setup without much thought as to the future, so certain issues have not been taken into consideration. Customers The customer defined here can be a person, a system or some user that provides for the initial process starting data. In figure 2 we provide a high level look at how our customers provide process data that we then package up into a request to be dropped into one of the process queues. From the queues we can then prioritize and let different mechanisms fetch these process requests and start a process instance with the provided request data. We show here EJB’s, MDB’s and clouds that represent any manner of scheduling that might be used to empty the process queues. Queues These queues can be as simple as database tables or as refined as message queues. They can be setup any way your project desires, such as Last-In-First-Out (LIFO) or First-In-First-Out (FIFO). The benefits of using message queues is that you can prioritize these from your polling mechanism. The reason for this setup is two fold. First, you have ensured that by not directly starting the process instance from the customer interface that you have persisted the customer request. It will never be lost in route to the process engine. Second, you have the ability to prioritize future processes that might not be able to meet project requirements like a new process request that has to start in 10 seconds after submission by the customer. If it gets put at the bottom of a queue that takes an hour to get to processing it, you have a problem. By prioritizing your queues you can adjust your polling mechanism to check the proper queues in the proper order each time. Java / Cloud The java icons in figure 2 are representing any JEE mechanism you might want to use to deal with the process queues. It can be EJB’s, MDB’s, a scheduler you write yourself or whatever you want to come up with to pick up process requests. The cloud icons are meant to represent services that can be used by your software to actually call the final startProcess method to initialize the process instance being requested and pass it initial data. It is important to centralize this interaction with the jBPM API into a single service thereby ensuring minimal work if the API should change, for possible version migrations in the future and should you wish to expand it in future projects to extend the service interaction with jBPM. Conclusion This article briefly walks through the high level BPM architecture and lays out the various layers of interaction. The first layer of interaction in larger enterprise BPM architectures, the initialization layer is examined to provide some insights into best practices within this layer. It is not a discussion that attempts to push implementation details, but takes a step back and introduces some of the basic elements that are repeatedly encountered in large BPM architectures. It covers the initial customer submission of a processing request, the queueing of process requests and the processing of these queues in a consistent and scalable manner. There is still more to take a look at in future articles, in the Process Implementation Layer, the Process Interaction Layer, in the Process Repository, in the Tooling and in the reporting & BAM layers. Reference: JBoss BRMS Best Practices – tips for your BPM Process Initialization Layer from our JCG partner Eric D. Schabell at the Thoughts on Middleware, Linux, software, cycling and other news… blog....

What makes parallel programming hard?

Multi-cores are here, and they are here to stay. Industry trends show that each individual core is likely to become smaller and slower (see my post to understand the reason). Improving performance of a single program with multi-core requires that the program be split into threads that can run on multiple cores concurrently. In effect, this pushes the problem of finding parallelism in the code to the programmers. I have noticed that many hardware designers do not understand the MT challenges (since they have never written MT apps). This post is to show them the tip of this massive iceberg.Update 5/26/2011: I have also written a case study for parallel programming which may interest you.Why finding parallelism is hard?Some jobs are easy to parallelize, e.g., if it takes one guy 8 hours to paint a room, then two guys working in parallel can paint it in four hours. Similarly, two software threads can convert a picture from color to grayscale 2x faster by working on different halves of the picture concurrently. Note: Programs that fall in this category are already being parallelized, e.g., scientific computing workloads, graphics, photoshop, and even open-source apps like ImageMagick.There also other programs that are sequential in nature, e.g., two guys will not be able to cook 2x faster than one guy because the task isn’t fully parallelizable: there are inter-task dependencies and the cooks end up waiting for each other at times. Unfortunately, a lot of programs have artificial inter-task dependencies because the programmers wrote them with an ST mind set. For example, consider this code excerpt from the H.264 reference code (I have removed unnecessary details to highlight my point):macroclock_output_data mb; //Global variable for (...) // The OUTER loop decode_slice(...); if(mb->last_mb_in_slice) break;void decode_slice(...) ... mb = ...Notice how the variable mb is written every iteration and no iteration uses the mb written by the previous iterations. However, mb was declared as a global variable probably to avoid its repeated allocation and deallocation. This is a reasonable ST optimization. However, from an MT standpoint, the loop iterations of the OUTER loop now have a dependency among each other and they cannot be run in parallel. To parallelize this code, the programmer has to first identify that the dependency is artificial. He/She then has to inspect 1000s of lines of code to ensure that this assumption isn’t mistaken. Lastly, he/she has to change the entire code to make mb a local per-iteration variable. All this is difficult to achieve (I parallelized H.264 for this paper).So here is the status: Leaving the artificial dependencies in the code limits parallelism, mistakenly removing a real one breaks the program, and reaching the perfect balance requires prohibitive effort. Since its hard to identify all dependencies correctly the first time, they make errors and debugging begins.Why is debugging difficult?Debugging multi-threaded code is very hard because bugs show up randomly. Consider the following:Say two threads T0 and T1 need to increment the variable X. The C/C++/JAVA code for this will beX = X + 1Their assembly code will look as follows: (instructions are labeled A-F).T0T1A Load X , R0 D Load A, R0B Increment R0 E Increment R0C Store R0, X F Store R0, AThe programmer wants X to be incremented by 2 after both threads are done. However, when the threads run concurrently, their instructions can interleave in any order. the final value of X depends on the interleaving (assume X was 0 before the two threads tried to increment). For example,ABCDEF: X = 2 (correct)DEAFBC: X=1 (incorrect)ADBCEF: X = 1 (incorrect)Basically, there is a dependency that D shall not execute before C (or A should not happen before F). The programmer has missed this dependency. However, the code does work fine half the times making it impossible to track the bug or test a fix. Moreover, traditional debugging techniques like printf and gdb become useless because they perturb the system, thereby changing the code behavior and often times masking the bug.Why is optimizing for performance so important and challenging?The sole purpose of MT is performance. It is very common that the first working version of the parallel code is slower than the serial version. There are two common reasons:Still too many dependencies (real or artificial): Programmers often iteratively remove dependencies and sometimes even re-write the whole program to reduce these dependencies.Contention for a hardware resource: Threads can also get serialized if there is contention for some hardware resource, such as a shared cache. Programmers have to identify and reduce this contention. Note that identifying these bottlenecks is especially challenging because hardware performance counters are not reliable.After several iterations, the code becomes good enough for the performance target.The work does not end here..Unlike ST code which would get faster every process generation, MT code has complex non-deterministic interactions that can make its performance swing long ways when hardware changes. For example, I had a branch-and-bound algorithm (a 16-Puzzle solver) which would slow down with more cores because the algorithm would end up on a different path when more threads were running. Even a simple kernel like histogram computation can behave very differently with different inputs or machine configuration (see this paper). Thus parallel programmers are also burdened with the task of making their code robust to changes.ConclusionMy goal here was not to teach parallel programming but merely provide a flavor of what it takes to write a good parallel program. It is indeed a complex job and I assert that it is not possible to appreciate the challenges without actually writing a parallel program. For those who have not written one yet, here is my call for action:Write a parallel program to compute the dot-product of two arrays and get it to scale perfectly, 4x speedup on 4-cores, etc. It is simple but you will learn more than you expect.Search the keywords: pthreads or winthreads to learn the syntax on linux or windows respectively. Share your experiences!Update 5/30/2011: Thank you everyones for your feedback and readership. I have written a third post on parallel programming. It describes a profile-based solution for choosing the best task granularity in parallel programs.Reference: What makes parallel programming hard? from our JCG partner Aater Suleman at the Future Chips blog....

Hello JavaFX 2.0: Introduction by Command Line

I looked at a simple JavaFX version of the ubiquitous Hello World example from a NetBeans 7.1 beta perspective in the blog post Hello JavaFX 2.0: Introduction by NetBeans 7.1 beta. In this post, I look at a slightly different version of Hello World implemented with JavaFX using only command-line tools.The JavaFX 2.0 API documentation includes the class description for the javafx.application.Application class and this is a good place to start. The Javadoc documentation for the Application class provides an example of an effective class usage description. This class description describes a JavaFX application’s life cycle and even provides a code sample with an image showing how the sample renders. I’ll work up to that same sample in this post.The Application class’s Javadoc documentation describes the central role of this class: “Application class from which JavaFX applications extend.” The start(Stage) method is the most interesting in the Application class as it is the “main entry point for all JavaFX applications.” It is an abstract method and so must be overridden by extending classes. The next code listing shows a minimal implementation that will compile but not do anything (it doesn’t even have a main function). (I: Bare Minimum) package dustin.examples;import javafx.application.Application; import javafx.stage.Stage;/** * Simple JavaFX Hello World example. * * @author Dustin */ public class HelloWorld extends Application { @Override public void start(final Stage stage) throws Exception { throw new UnsupportedOperationException("JavaFX example not supported yet."); } }The previous code snippet shows the importing of two JavaFX classes (Application and Stage) When the above code is compiled with javac without placing the JavaFX libraries on the classpath, errors similar to the following occur. error: package javafx.application does not exist import javafx.application.Application; ^ error: package javafx.stage does not exist import javafx.stage.Stage; ^ error: cannot find symbol public class HelloWorld extends Application ^ symbol: class Application error: cannot find symbol public void start(final Stage stage) throws Exception ^ symbol: class Stage location: class HelloWorld error: method does not override or implement a method from a supertype @Override ^ 5 errorsThe obvious solution is to place the apropos JavaFX library on the classpath of the compiler. In my case, the JavaFX SDK and JAR needed to build this code is C:\Program Files\Oracle\JavaFX 2.0 SDK\rt\lib\jfxrt.jar. The next code listing builds upon the previous code snippet and is adapted from the example provided in the Application class’s class-level Javadoc documentation. (II: Adapted from Application’s Javadoc) package dustin.examples;import javafx.application.Application; import javafx.scene.Group; import javafx.scene.Scene; import javafx.scene.shape.Circle; import javafx.stage.Stage;/** * Simple JavaFX Hello World example. * * @author Dustin */ public class HelloWorld extends Application { @Override public void start(final Stage stage) throws Exception { final Circle circ = new Circle(40, 40, 30); final Group root = new Group(circ); final Scene scene = new Scene(root, 400, 300);stage.setTitle("Hello JavaFX 2.0!"); stage.setScene(scene);; } }The JavaFX application shown above can be deployed to a web browser, but I’m going to instead focus on running it from the command line. To do this, a main function is added to the JavaFX application as shown in the next version. (III: Added ‘main’ Function) package dustin.examples;import javafx.application.Application; import javafx.scene.Group; import javafx.scene.Scene; import javafx.scene.shape.Circle; import javafx.stage.Stage;/** * Simple JavaFX Hello World example. * * @author Dustin */ public class HelloWorld extends Application { @Override public void start(final Stage stage) throws Exception { final Circle circ = new Circle(40, 40, 30); final Group root = new Group(circ); final Scene scene = new Scene(root, 400, 300);stage.setTitle("Hello JavaFX 2.0!"); stage.setScene(scene);; }/** * Main function used to run JavaFX 2.0 example. * * @param arguments Command-line arguments: none expected. */ public static void main(final String[] arguments) { Application.launch(arguments); } }Only a single line is required in the main function. That line is a call to the static method Application.launch(String…) with the command-line arguments passed to it. This application can now be executed and appears as shown in the screen snapshot that follows.ConclusionThis blog post has demonstrated writing and running a simple JavaFX application using only command-line tools. Proving that JavaFX 2.0 has put the ‘Java’ back into JavaFX, the examples in this post were compiled and executed with the typical Java compiler and Java launcher used for “normal” Java applications. More complex JavaFX applications may benefit from more specific tools, but this one was compiled and executed solely with the standard Java tools.Reference: Hello JavaFX 2.0: Introduction by Command Line from our JCG partner Dustin Marx at the Inspired by Actual Events blog....

Telecommunications for Dummies – Telecom Basics and Introduction to BSS

Introduction This post is intended to be a crash course for beginners who wish to understand at a “broad level” how Business Support Subsystem components work in a telecom carrier’s network and more importantly how they connect to the telecom network elements over standard protocols.Hence, the text is more “conversational” in nature rather than completely “technical”. Elementary examples of simple call setup procedures have also been included, as they are required for a holistic understanding of the entire ecosystem – beginning from a call setup, to its charging, rating and billing.   Background Telecom operations from the perspective of a carrier are divided into two broad categories: OSS and BSS. Operations Support Subsystems (OSS) is an aggregation of functions that enable the operator to provision and manage their inventory, customers, services and network elements. Provisioning implies “making an entry or a record” of some resource. This resource may be a customer, a value added service, inventory such as dongles or even a network element such as a call control switch. For example if a new customer is acquired, his entire record has to be saved in the network such as his name, address, phone number allotted etc. This is customer provisioning. An example of managing an inventory could be as follows. Dongles procured by the carrier need to be recorded in the inventory management system. Each dongle would undergo a lifecycle. Eg: When the dongle is first procured, it is recorded to be in stock. When the dongle is dispatched to a retail store, its status is changed in the inventory management system to be “for sale”. When a customer purchases the dongle, its status changes to “sold” and it is linked to the customer’s identity who bought it. On the other hand, Business Support Subsystems is an aggregation of functions which are used for managing an operator’s day-to-day business characteristics and providing an operator with complete clarity on the performance and management of his diverse lines of businesses. BSS systems include real-time functions such as charging (post-paid/pre-paid), rating using tariff plans and applying discounts. BSS also includes some non-real-time components such as launching new marketing campaigns, analyzing the success of individual marketing campaigns through business intelligence reports, partner management (for dealing with 3rd party content providers, other operators, franchisees etc), creating invoices, payment collection, revenue sharing with partners and billing. This note is more focussed on the BSS components of an operator’s ecosystem. Before going to the specifics of BSS, it is useful to review a typical call setup procedure for post paid calls as well as for pre paid calls, so that some basic grounding is available regarding the call flows in a typical telecom network:?? The above scenario depicts the flow of messages for a pre-paid subscriber. When the subscriber dials out a number, the local office checks whether the subscriber has enough balance in his/her account. Only if the balance is available, does the call proceed further.   Case Study:NOTE-1: There may be one or more transit exchanges between the originating local office and the terminating local office. NOTE-2: The messages that flow between the local exchange and the pre-paid platform are in INAP protocol. INAP stands for Intelligent Network Application Part and is the top most layer of the SS7 protocol stack. Both the local office and the service control point (SCP) understand the INAP messages. NOTE-3: The other levels of the SS7 stack are: MTP1, MTP2, MTP3, SCCP and TCAP. MTP stands for Message Transfer Part. SCCP stands for Signalling Connection Control Part. TCAP stands for Transaction Capability Application Part. Main steps in charging a pre-paid call in a fixed line network:When a pre-paid subscriber makes a call, it arrives at a local exchange (also called local office in the US). The local exchange looks at the calling number (A-party), and finds that it is a call made by a pre-paid customer. So, it cannot connect the call straight away as it would have done for a post –paid customer. The local exchange has to now talk to a pre-paid platform. The local exchange sends a message to the pre-paid platform informing it of the A-party number and the B-party number. This message is received by a component of the pre-paid platform called the Service Control Point (SCP). SCP is nothing but a software module that understands and receives messages sent by the local exchange. On receiving the message from the local exchange, the SCP communicates with a Rating Engine, which is the second component of the pre-paid platform. The rating engine is the heart of the pre-paid platform. As tariff plans get more complex and more services need to be charged, the rating engine must have flexibility and functionality to meet the business requirements. It contains in its database, the current balance of all the pre-paid customers. Since there are hundreds and thousands of customers served by a single pre-paid platform, the rating engine will contain a large number of customer balance records. The rating engine looks at the A-party number and then queries its database to find out the current balance of the A-party. If the A-party has zero or insufficient balance, then it must inform the SCP that the calling customer has insufficient balance. The SCP in turn will send a message to the local exchange informing it that the calling party has insufficient balance. The local exchange rejects the call. If however, the rating engine finds that the calling party has sufficient balance, then via the SCP, the local exchange is informed that the call can proceed. If the balance of the calling party is zero, then no further analysis is required by the rating engine to reject the call. However it has to check by looking at the B-party number if the balance is sufficient to support the minimum rate of the call pulse. For example: If a local call costs 10 cents per minute and the balance of the A-party is currently 5 cents, then this balance is not sufficient. If it was an international call, then even 50 cents balance may not be sufficient. Assuming that the balance was sufficient and the pre-paid platform has allowed the call to proceed, then the local exchange will allow the call to proceed and the call will be eventually connected via one or more exchanges to the called subscriber whose phone will ring. If the call is abandoned for any reason like the calling party abandoning the call before it is answered by the B-party, no answer by B-party etc, then the local exchange must again send a message to the SCP informing it that the call has not matured. The rating engine had at the time of allowing the call reserved a certain amount from the subscriber’s balance. For example the amount that would have been reserved depends upon the tariff plan of the customer. To avoid frequent requests for the reservation of amount, the rating engine may be asked to grant a larger chunk of units from the balance. As an example, assume that the calling party had a balance of USD 100 when the call was made. When the message reaches the pre-paid platform, the rating engine on finding that there is enough balance will not only respond to the local exchange about the sufficiency of the balance as described above, but also reserve an amount, say USD 5. Therefore, temporarily the balance reduces to USD 95. It is important to understand that reserving an amount is not equivalent to actually reducing the balance of the calling party. At this time, the reservation only signifies the “intent”, and actual deduction would happen only if the call is successful and usage takes place. When the call is answered, the deduction will happen periodically from the reserved amount of USD 5. How much amount must be deducted and at what periodicity? This has to ultimately depend upon the tariff plan which the calling party has subscribed to. Assuming that the call charges are 50 cents per minute, this reserved amount of USD 5 would be enough for 10 minutes of conversation. This is the validity time. At the time of informing the local exchange that the calling party has sufficient balance and the call can be connected, the validity time is also conveyed by the SCP. Thus, the local exchange knows that if this call matures, it can let the call continue for the first 10 minutes without again contacting the pre-paid platform. If the conversation ends is less than 10 minutes, for example the call duration is just 3 minutes, the local exchange sends a message to the charging platform and only USD 1.5 will be deducted by the pre-paid platform at the end of the call. The remaining USD 3.5 is returned to the account balance of the calling party. His closing balance will now be USD 8.5. Assuming that the conversation exceeded 10 minutes, the local exchange sends another request to reserve more units. This has to be done prior to the expiry of the validity time. The pre-paid platform will reserve another chunk of USD 5 and inform the local exchange accordingly. It may happen that the unit reservation request fails. This will happen when the subscriber has no units left in his balance. In such a scenario, the pre-paid platform will respond with a message that informs the local exchange that the credit limit is reached. The local exchange will then immediately terminate the call.The architecture for charging a pre-paid call in a mobile network (GSM) is shown below. Note that it is very similar. Only the INAP protocol layer has been replaced by CAMEL:   NOTE-1: CAMEL stands for Customised Applications for Mobile Enhanced Logic. NOTE-2: There will be a base station and the base station controller (BSC) before the call reaches the originating MSC. Likewise, there will be a BSC and a BTS on the terminating side. NOTE-3: There may be one or more transit exchanges between the originating and terminating MSCs. These are called GMSCs (Gateway MSCs) in mobile networks. NOTE-4: The architecture in a CDMA mobile network will be identical except that CAMEL will be replaced by IS-826 protocol layer.   Details of the SMP and the VMS: Service Management Point (SMP): The SMP performs the management functions for the IN platform. For example, all the pre-paid customers must be recorded in the IN platform along with their initial balance. This is typically called the provisioning of customers. This task is performed by the SMP. The SMP will typically have local terminals where operators can do this task through a set of commands called the command line interface (CLI). There may also be a need to periodically delete (de-provision) customers who have left the pre-paid service provided by the operator. It will also be connected to a voucher management system from where commands will flow to the SMP for bulk provisioning of customers so that a large number of customers can be simultaneously provisioned in the pre-paid system. Voucher Management System: The Voucher Management System (VMS) allows prepaid subscribers recharge their accounts using pre-paid vouchers, allowing operators to run widely distributed dealer networks efficiently and easily. VMS allows subscribers to add funds to their accounts within the home network or in roaming networks regardless of location, time of day or model of phone. VMS cuts down administrative, bookkeeping and servicing expenses and makes it easy for subscribes to keep track of their account balance without service centres. Features of a VMS are as follows: Account recharging via voice services The Voucher Management System allows subscribers to recharge their accounts from their own telephone as well as from any tone-dial phone. Account recharging via non-voice services Subscribers can top up their own accounts, as well as the account of any other subscriber of the same network, by sending an SMS request with a PIN code to a service number. Accounts can also be recharged from a special websites of the carrier. Account recharging via operator Subscribers can recharge their accounts via call centre operators. Distributed system The prepaid card system allows distributed systems with card roaming to be built, giving your subscribers the ability to recharge their accounts while in roaming. Blocking The Voucher Management System can bar a specific telephone number from accessing the service after several consecutive unsuccessful attempts to enter a PIN code from that number, protecting you from fraud. Advertising Additional profit can be generated from placing adverts on cards, and can also be a useful marketing tool for the operator, for example to advertise new services. Card Status Information about card usage (transactions made) is recorded automatically in the operator’s billing system. Revisiting Post paid Charging: Charging for Post-paid subscribers: For post-paid subscribers, there is no need to check for the balance. The calls are allowed to proceed further, without any messaging to a pre-paid platform, and the call details are captured in the form of a Call Detail Record (CDR) at the end of the call. These CDRs are used later for billing purposes. A typical CDR file captures the following information: – Calling party number – Called party number – Call start time – Call end time – Call duration (End Time – Start Time) – Call Identifier This information is saved in the form of a file in the local exchange or the MSC and is pushed to the Charging System for post processing.The charging system has a rating engine and a mediation engine. The rating engine analyzes the CDR files and determines the rate to be applied for each CDR file. Once this rating is completed, the CDR is known as a rated-CDR. This rated CDR is pushed to the mediation engine. The mediation engine is required because it may receive CDRs from more than one source and the format of the CDRs may be different from each source. The mediation engine post-processes the rated CDRs from multiple sources and reformats them into a common file. This common file is then pushed to the billing system which generates the itemized bills for each customer based on the CDRs. A simple procedure for a post-paid subscriber is given below in terms of CDR creation and storage:We have so far described charging of voice calls. The figure below shows the post-paid charging of a data call in CDMA:The data call path bifurcates from the PCF – Packet Control Function, which is a logical function of the Base Station Controller as shown above. The Rating, Mediation engine and billing system are common for data calls as well as voice calls. The figure below shows the corresponding architecture for charging a Mobile WiMAX data call. The only difference here is, that we have an ASN gateway as the controller for data calls. The AAA is sending charging events on RADIUS protocol.The AAA (Authorization, Authentication and Accounting) server receives the messages on RADIUS protocol. There may be a dedicated RADIUS to DIAMETER converter in the AAA server, which converts these messages to the DIAMETER protocol and sends them to the charging platform. DIAMETER protocol’s Rf interface is used for post-paid charging while the DIAMETER protocol’s Ro interface is used for pre-paid charging. Introduction to the Realtime Components of BSS: We can now discuss the BSS system in more detail. There may be some overlap of information with what is described above. The figure below shows a “modern” BSS system which works on DIAMETER interfaces and integrates with IMS, Mobile WiMAX and LTE on DIAMETER protocols:LEGEND:This is only an architectural representation meant to show some of the major components of a BSS system and to provide a glimpse in to the complexity of a modern BSS platform. However, we will be discussing conceptually some of the basic components of the BSS system without going into the technical details. Some of the most important real-time network facing components of a BSS system are:Charging Rating Mediation Billing, and Reconciliation1. Charging Charging of customers can be done in two ways: pre-paid (online) charging and post-paid (offline) charging. The first step of revenue generation starts with the charging process, where the network elements identify the events which need to be charged. Some examples include – sending a SMS, MMS, dialling a number, downloading content etc. Network elements such as the local exchange (office), SMSC (Short Message Service Centre) etc generate these “chargeable” events towards the charging platform. Chargeable events can be conceptualized as “intents” to charge the customer based on his/her actions. At this stage, it is not guaranteed that these events will actually lead to revenue realization. For example, a chargeable event will be generated for a missed call, or a failed call setup. However, as the call never got connected successfully, the customer will not be liable for paying for this event. However, the operator may change his policy and may charge the customer even for missed calls if needed. Interfaces for Charging: Charging platforms support multiple interfaces to receive events. Some of the most common charging interfaces are on RADIUS and DIAMETER protocols. Legacy systems used to charge calls on the IN (Intelligent Network) pre-paid platform. There were also Web Service interfaces used for charging in some legacy systems. Web services work on the HTTP protocol and are described by SOAP protocol (Simple Object Access Protocol).   2. Rate Determination and Rating. The next logical step after receiving charging events is Rate Determination and Rating. Once the charging system has received an event, rate calculation takes place for the customer depending upon the current tariff plan in use. The charging system needs to decide the amount of units which need to be consumed for the particular charging event. For post-paid subscribers, these units are determined and attached to a Call Detail Record (CDR). Then this “rated-CDR” is dumped as a file. For pre-paid subscribers, the units are determined and compared with the subscriber’s account balance. If the account balance is sufficient, the charging system borrows the required amount of units from the subscriber’s account and prepares to deduct them from the available balance. Units may have different manifestations. For example, we can represent Units as “money”. We may also represent units in the form of the allowable limit for data download in Megabytes (units measured by volume). Another possibility is to measure units in terms of talk time. In some special cases, the operator may also represent units in terms of the number of free calls offered and the number of free SMS messages allowed. Before the charging system rates a particular charging event and actually deducts units, there is an intermediary step. This step is to apply discounts if applicable. For example, if the charging event cost the subscriber $1 and the tariff plan of the subscriber offered him with a discount of 50% on all calls, then the rate applicable would only be 50 cents. At this stage, it is important to understand the relationship between a tariff plan, the components of a tariff plan, offers attached with the tariff plan, rate to be applied based on the offers, and the validity semantics of the offer. These are important inputs to the rating engine. The final rate to be applied is calculated based on the tariff plan, offers and discounts applicable on the base plan and the validity of the offers. The illustration below explains the relationship between these concepts:The figure above represents a tariff plan. This plan has four components:Voice Messaging Data Content.The voice component applies to the usual calls that the subscriber initiates. The messaging component of this example refers to the SMS messages. The Data component refers to the internet browsing and downloads from websites. Finally, the content component refers to the purchase of premium content such as movies, ringtones and ebooks from an operator’s application store for example. Each component of this tariff plan has an offer attached to it. In this example, the customer has the first 100 voice calls free, the first 50 SMS messages free, first 10 MB of downloads free and 2 ringtones free. These offers have validity applicable to them. The first 100 voice calls are free and must be consumed within a month. Similarly, the free SMS offer must be consumed within a month. The same is applicable to the data download offer. However, the 2 ringtones are free as a onetime offer to the customer. The customer may download at the most 2 ringtones for free. For the 3rd ringtone download, normal charges shall apply. The last concept is that of usage counters. At the top of each month, the free voice call quota is re-initialized to a value of 100. With each call made by the customer, this value of 100 is decremented until it reaches zero. Once the free call quota is zero, normal charges apply to all subsequent calls. Similarly, for messaging the usage quota is set to 50 every month. Validity can be represented in several forms:  The validity semantics of an offer has to be very flexible. This flexibility determines the business agility of an operator. In the figure above, certain validity options are shown. For example:A particular offer may be valid at the top of each month. This is similar to the voice call free usage counters discussed in the example earlier.In some other scenarios, a certain offer may be valid only on certain days of the week (Monday, Wednesday, and Friday). For example all voice calls are discounted 30% on certain days of the week (weekends for example).Another option is for an offer to be applicable within a time band of a day. For example, the operator may offer discounted call rates at night time (between 11 pm and 5 am) to encourage usage during these lean hours.The fourth option shown above is for an offer to be applicable only on some special days such as Diwali, New Year, and Christmas etc.The validity of an offer can also be confined to a closed user group. For example, all calls made between users belonging to a closed user group will be cheaper as compared to the calls made outside the group.Some offers are valid based on the destination of the called party. For example, there may be a special rate applied when calls are made between New York and Chicago.Usage based validity is a very interesting case. For example, when the usage of a particular customer crosses $30 in a given month, all subsequent calls for that month are offered at a discounted rate of 25%. This is a kind of a “reward” given to high ARPU customers.The last option for validity is “unconditional”. This means that the offer in question will be valid at all times and for all calls.3. Mediation: The mediation process is the next logical step after rating. Once the rate is determined as explained in the previous section, it is applied to the CDR file. Now, this rated CDR file is pulled by the mediation engine. The mediation engine can handle CDRs from various sources – The MSC, the local office, the SMSC etc. It converts all these CDR files into a common file format. The file format is chosen so that it is understandable by the billing system. This file is then provided to the billing system for post-processing.   4. Billing: In the billing process, CDRs are provided by the Mediation engine to the billing system. The billing system processes these CDRs and reads them. The output of this process is the generation of an itemized bill in human readable form by the billing system. As the CDRs which reach the billing system are already rated, the billing system can calculate the final bill for a subscriber and the rate applied for each charging event. The details of the CDR are presented in the form of a bill which can then be e-mailed or dispatched to the subscriber’s billing address. These days, some advanced billing systems also provide a feature of billing level discounts. For example, if there is a high paying customer who generates a bill of over USD 100 every month for 6 months consecutively, the billing system can provide this customer with a discount of 10% on his next bill. The billing system may even ask the rating engine to provide this high ARPU customer with a quota for free calls from next month onwards. Wholesale/Interconnect Billing: Wholesale billing applications include a variety of capabilities. Traditionally, this area included inter-carrier settlements capabilities and this was later extended to interconnect billing applications. In today’s competitive markets and complex value chains, it has expanded further to include among others Roaming, wholesale operators, and resellers, Mobile Virtual Network Operators, Content Providers and E-Commerce. There is now an array of applications in the area providing charging, billing, and settlement capabilities on a raw data basis, individual transaction basis and bulk basis across a variety of services and platforms. These applications work across a variety of platforms and support a wide range of services, preferably in one single system.   5. Reconciliation: Reconciliation is an offline process of sorting out the CDRs of the calls made in a certain period to determine the following information:How many calls were made to another carrier’s network? How many calls made to another carrier’s network were local/regional/international? How many calls from other networks terminated in my network? How many calls made from a 3rd party network used my network as a transit and they actually terminated in some other network? Similar details for SMS messages and MMS messages are also post processed.Based on such information, the operator makes a record of the amount of revenue which has to be shared with other network operators. This revenue sharing takes place because all these operators share a point of interconnect (POI) to each other’s networks and they allow calls to be made from one network to another. Due to this, the operator arrives at an amount which other operators need to pay him, and an amount which he needs to pay to other operators for providing interconnect services to each other. This process is known as reconciliation.   Business Perspective and an introduction to the non-realtime components of BSS: Business teams of a carrier are responsible for designing and planning tariff plans which are later launched in the form of marketing campaigns. Each tariff plan needs to be designed carefully, keeping in mind the inputs obtained from competitive analysis of the tariff plans of other carriers. The BSS component which manages the marketing campaigns for the carrier is called “Campaign Management”. The success or failure of a particular marketing campaign is gauged by another component known as “Business Intelligence” or BI. The Business Intelligence engine provides the business teams with clarity on the performance of a particular marketing campaign based on various parameters. An example is provided below:Popularity of a particular marketing campaign (tariff plan) based on market segments such as children, young executives etc.Popularity of a campaign based on the region – Florida, California, New Jersey etc.Popularity of a campaign based on age.Popularity of a campaign based on market segment – High ARPU customers, Customers with traditionally low ARPU, mid ARPU customers, enterprise customers etc.Some of the other major functions of a BSS system which is critical from the viewpoint of a network carrier are: Revenue Assurance: The main revenue assurance application areas are:Detection of data discrepancies – Detection of data discrepancies between systems and data repositories that might affect the ability to generate revenues or increase costs, including,Configuration data – e.g. between Customer Relationship Management systems, inventory and network) Events data – e.g., between the call Switch, Mediation, and billing) Interconnect/partners billingDetection of data integrity and correctness problems, e.g., a switch that rounds incorrectly the durations of the call. Rating and Billing Verification – Verification of the correctness of the application of the rating and billing rules – e.g., is the customer billed according to the correct plan, and is he billed correctly according to the correspondent plan. Investigation of revenue leakages, finding and correcting their root cause to prevent the recurrence of similar leakages Grouping and classification of leakages Equipment and system testing – Proactively test equipment and systems and processes to verify that they provide accurate information- e.g., using test call generation Trouble Reports and Alarms – Generation and tracking of Revenue Assurance Trouble Reports and Alarms Automation of revenue assurance controls and data collection Automation of leakages correction Generation of revenue leakage reports and documentation both for internal needs as well as a support to regulatory compliance activities.Fraud Management: Investigating, preventing and responding to activities that indicate fraudulent use of networks or systems. This is achieved by effective Fraud Management systems coupled with the instrumentation and monitoring that enables potential fraudulent activities to be identified. There are close linkages between fraud identification and anomaly detection. Eg:Someone fraudulently tops up the pre paid account from the system backend without using a voucher. Someone reduces the value of the pos paid bill just before it is processed by the billing system. Someone deletes the CDRs or reduces the rate on each CDR.Conclusion: The domain of BSS is huge and it is not possible to cover all aspects in a single post. However, this post provided an introduction to some of the important concepts of this area and how the BSS connects to network elements of a telecom system over standard interfaces. We also discussed some of the most critical concepts of a BSS platform which directly influence the operator’s revenue realization capabilities – such as rating, charging, billing, reconciliation, revenue assurance and fraud management. Reference: Telecommunications for Dummies – Telecom Basics and Introduction to BSS. from our JCG partner Aayush at the Aayush: weblog blog....

FreeChart with Groovy and Apache POI

The point of this article is to show you how to parse data from an Excel spreadsheet that looks like this:and turn it into a series of graphs that look like this:Recently I was looking for an opportunity to get some practice with JFreeChart and ended up looking at a dataset released by the Canadian government as part of their ‘Open Data’ initiative. The particular set of data is entitled ‘Number of Seedlings Planted by Ownership, Species’ and is delivered as an Excel spreadsheet, hence the need for the Apache POI library in order to read the data in. As is fairly usual, at least in my experience, the Excel spreadsheet is designed primarily for human consumption which adds a degree of complexity to the parsing. Fortunately the spreadsheet does follow a repetitive pattern that can be accounted for fairly easily, so this is not insurmountable. Still, we want to get the data out of Excel to make it more approachable for machine consumption so the first step is to convert it to a JSON representation. Once it is in this much more transportable form we can readily convert the data into graph visualizations using JFreeChart.The spreadsheet format Excel as a workplace tool is very well established, can increase individual productivity and is definitely a boon to your average office worker. The problem is that once the data is there it’s often trapped there. Data tends to be laid out based on human aesthetics and not on parsability, meaning that unless you want to use Excel itself to do further analysis, there’s not a lot of options. Exports to more neutral formats like csv suffer from the same problems- namely that there’s no way to read in the data coherently without designing a custom parser. In this particular case, parsing the spreadsheet has to take into account the following:Merged cells where one column is meant to represent a fixed value for a number of sequential rows. Column headers that do not represent all of the actual columns. Here we have a ‘notes’ column for each province that immediately follows its’ data column. As the header cells are merged across both of these columns, they cannot be used directly to parse the data. Data is broken down into several domains that lead to repetitions in the format. The data contains a mix of numbers where results are available and text where they are not. The meanings of the text entries are described in a table at the end of the spreadsheet. Section titles and headers are repeated throughout the document, apparently trying to match some print layout, or perhaps just trying to provide some assistance to those scrolling through the long document.Data in the spreadsheet is first divided into reporting by Provincial crown land, private land, Federal land, and finally a total for all of them. Within each of these sections, data is reported for each tree species on a yearly basis across all Provinces and Territories along with aggregate totals of these figures across Canada. Each of these species data-tables has an identical row/column structure which allows us to create a single parsing structure sufficient for reading in data from each of them separately.Converting the spreadsheet to JSON For parsing the Excel document, I’m using the Apache POI library and a Groovy wrapper class to assist in processing. The wrapper class is very simple but allows us to abstract most of the mechanics of dealing with the Excel document away. The full source is available on this blog post from author Goran Ehrsson. The key benefit is the ability to specify a window of the file to process based on ‘offset’ and ‘max’ parameters provided in a simple map. Here’s an example for reading data for the text symbols table at the end of the spreadsheet. We define a Map which states which sheet to read from, which line to start on(offset) and how many lines to process. The ExcelBuilder class(which isn’t really a builder at all) takes in the path to a File object and under the hood reads that into a POI HSSFWorkbook which is then referenced by the call to the eachLine method. public static final Map SYMBOLS = [sheet: SHEET1, offset: 910, max: 8] ... final ExcelBuilder excelReader = new ExcelBuilder(data.absolutePath) Map<String, String> symbolTable = [:] excelReader.eachLine(SYMBOLS) { HSSFRow row -> symbolTable[row.getCell(0).stringCellValue] = row.getCell(1).stringCellValue } Eventually when we turn this into JSON, it will look like this: 'Symbols': { '...': 'Figures not appropriate or not applicable', '..': 'Figures not available', '--': 'Amount too small to be expressed', '-': 'Nil or zero', 'p': 'Preliminary figures', 'r': 'Revised figures', 'e': 'Estimated by provincial or territorial forestry agency', 'E': 'Estimated by the Canadian Forest Service or by Statistics Canada' } Now processing the other data blocks gets a little bit trickier. The first column consists of 2 merged cells, and all but one of the other headers actually represents two columns of information: a count and an optional notation. The merged column is handled by a simple EMPTY placeholder and the extra columns by processing the list of headers;. public static final List<String> HEADERS = ['Species', 'EMPTY', 'Year', 'NL', 'PE', 'NS', 'NB', 'QC', 'ON', 'MB', 'SK', 'AB', 'BC', 'YT', 'NT *a', 'NU', 'CA'] /** * For each header add a second following header for a 'notes' column * @param strings * @return expanded list of headers */ private List<String> expandHeaders(List<String> strings) { strings.collect {[it, '${it}_notes']}.flatten() } Each data block corresponds to a particular species of tree, broken down by year and Province or Territory. Each species is represented by a map which defines where in the document that information is contained so we can iterate over a collection of these maps and aggregate data quite easily. This set of constants and code is sufficient for parsing all of the data in the document. public static final int HEADER_OFFSET = 3 public static final int YEARS = 21 public static final Map PINE = [sheet: SHEET1, offset: 6, max: YEARS, species: 'Pine'] public static final Map SPRUCE = [sheet: SHEET1, offset: 29, max: YEARS, species: 'Spruce'] public static final Map FIR = [sheet: SHEET1, offset: 61, max: YEARS, species: 'Fir'] public static final Map DOUGLAS_FIR = [sheet: SHEET1, offset: 84, max: YEARS, species: 'Douglas-fir'] public static final Map MISCELLANEOUS_SOFTWOODS = [sheet: SHEET1, offset: 116, max: YEARS, species: 'Miscellaneous softwoods'] public static final Map MISCELLANEOUS_HARDWOODS = [sheet: SHEET1, offset: 139, max: YEARS, species: 'Miscellaneous hardwoods'] public static final Map UNSPECIFIED = [sheet: SHEET1, offset: 171, max: YEARS, species: 'Unspecified'] public static final Map TOTAL_PLANTING = [sheet: SHEET1, offset: 194, max: YEARS, species: 'Total planting'] public static final List<Map> PROVINCIAL = [PINE, SPRUCE, FIR, DOUGLAS_FIR, MISCELLANEOUS_SOFTWOODS, MISCELLANEOUS_HARDWOODS, UNSPECIFIED, TOTAL_PLANTING] public static final List<String> AREAS = HEADERS[HEADER_OFFSET..-1] Closure collector = { Map species -> Map speciesMap = [name: species.species] excelReader.eachLine(species) {HSSFRow row -> //ensure that we are reading from the correct place in the file if (row.rowNum == species.offset) { assert row.getCell(0).stringCellValue == species.species } //process rows if (row.rowNum > species.offset) { final int year = row.getCell(HEADERS.indexOf('Year')).stringCellValue as int Map yearMap = [:] expandHeaders(AREAS).eachWithIndex {String header, int index -> final HSSFCell cell = row.getCell(index + HEADER_OFFSET) yearMap[header] = cell.cellType == HSSFCell.CELL_TYPE_STRING ? cell.stringCellValue : cell.numericCellValue } speciesMap[year] = yearMap.asImmutable() } } speciesMap.asImmutable() } The defined collector Closure returns a map of all species data for one of the four groupings(Provincial, private land, Federal and totals). The only thing that differentiates these groups is their offset in the file so we can define maps for the structure of each simply by updating the offsets of the first. public static final List<Map> PROVINCIAL = [PINE, SPRUCE, FIR, DOUGLAS_FIR, MISCELLANEOUS_SOFTWOODS, MISCELLANEOUS_HARDWOODS, UNSPECIFIED, TOTAL_PLANTING] public static final List<Map> PRIVATE_LAND = offset(PROVINCIAL, 220) public static final List<Map> FEDERAL = offset(PROVINCIAL, 441) public static final List<Map> TOTAL = offset(PROVINCIAL, 662)private static List<Map> offset(List<Map> maps, int offset) { maps.collect { Map map -> Map offsetMap = new LinkedHashMap(map) offsetMap.offset = offsetMap.offset + offset offsetMap } } Finally, we can iterate over these simple map structures applying the collector Closure and we end up with a single map representing all of the data. def parsedSpreadsheet = [PROVINCIAL, PRIVATE_LAND, FEDERAL, TOTAL].collect { it.collect(collector) } Map resultsMap = [:] GROUPINGS.eachWithIndex {String groupName, int index -> resultsMap[groupName] = parsedSpreadsheet[index] } resultsMap['Symbols'] = symbolTable And the JsonBuilder class provides an easy way to convert any map to a JSON document ready to write out the results. Map map = new NaturalResourcesCanadaExcelParser().convertToMap(data) new File('src/test/resources/NaturalResourcesCanadaNewSeedlings.json').withWriter {Writer writer -> writer << new JsonBuilder(map).toPrettyString() }Parsing JSON into JFreeChart line charts All right, so now that we’ve turned the data into a slightly more consumable format, it’s time to visualize it. For this case I’m using a combination of the JFreeChart library and the GroovyChart project which provides a nice DSL syntax for working with the JFreeChart API. It doesn’t look to be under development presently, but aside from the fact that the jar isn’t published to an available repository it was totally up to this task. We’re going to create four charts for each of the fourteen areas represented for a total of 56 graphs overall. All of these graphs contain plotlines for each of the eight tree species tracked. This means that overall we need to create 448 distinct time series. I didn’t do any formal timings of how long this takes, but in general it came in somewhere under ten seconds to generate all of these. Just for fun, I added GPars to the mix to parallelize creation of the charts, but since writing the images to disk is going to be the most expensive part of this process, I don’t imagine it’s speeding things up terribly much. First, reading in the JSON data from a file is simple with JsonSlurper. def data new File(jsonFilename).withReader {Reader reader -> data = new JsonSlurper().parse(reader) } assert data Here’s a sample of what the JSON data looks like for one species over a single year, broken down first by one of the four major groups, then by tree species, then by year and finally by Province or Territory. { 'Provincial': [ { 'name': 'Pine', '1990': { 'NL': 583.0, 'NL_notes': '', 'PE': 52.0, 'PE_notes': '', 'NS': 4.0, 'NS_notes': '', 'NB': 4715.0, 'NB_notes': '', 'QC': 33422.0, 'QC_notes': '', 'ON': 51062.0, 'ON_notes': '', 'MB': 2985.0, 'MB_notes': '', 'SK': 4671.0, 'SK_notes': '', 'AB': 8130.0, 'AB_notes': '', 'BC': 89167.0, 'BC_notes': 'e', 'YT': '-', 'YT_notes': '', 'NT *a': 15.0, 'NT *a_notes': '', 'NU': '..', 'NU_notes': '', 'CA': 194806.0, 'CA_notes': 'e' }, ... Building the charts is a simple matter of iterating over the resulting map of parsed data. In this case we’re ignoring the ‘notes’ data but have included it in the dataset in case we want to use it later. We’re also just ignoring any non-numeric values. GROUPINGS.each { group -> withPool { AREAS.eachParallel { area -> ChartBuilder builder = new ChartBuilder(); String title = sanitizeName('$group-$area') TimeseriesChart chart = builder.timeserieschart(title: group, timeAxisLabel: 'Year', valueAxisLabel: 'Number of Seedlings(1000s)', legend: true, tooltips: false, urls: false ) { timeSeriesCollection { data.'$group'.each { species -> Set years = (species.keySet() - 'name').collect {it as int} timeSeries(name:, timePeriodClass: '') { years.sort().each { year -> final value = species.'$year'.'$area' //check that it's a numeric value if (!(value instanceof String)) { add(period: new Year(year), value: value) } } } } } } ... } Then we apply some additional formatting to the JFreeChart to enhance the output styling, insert an image into the background, and fix the plot color schemes. JFreeChart innerChart = chart.chart String longName = PROVINCE_SHORT_FORM_MAPPINGS.find {it.value == area}.key innerChart.addSubtitle(new TextTitle(longName)) innerChart.setBackgroundPaint(Color.white) innerChart.plot.setBackgroundPaint(Color.lightGray.brighter()) innerChart.plot.setBackgroundImageAlignment(Align.TOP_RIGHT) innerChart.plot.setBackgroundImage(logo) [Color.BLUE, Color.GREEN, Color.ORANGE, Color.CYAN, Color.MAGENTA, Color.BLACK, Color.PINK, Color.RED].eachWithIndex { color, int index -> innerChart.XYPlot.renderer.setSeriesPaint(index, color) }And we write out each of the charts to a formulaically named png file. def fileTitle = '$FILE_PREFIX-${title}.png' File outputDir = new File(outputDirectory) if (!outputDir.exists()) { outputDir.mkdirs() } File file = new File(outputDir, fileTitle) if (file.exists()) { file.delete() } ChartUtilities.saveChartAsPNG(file, innerChart, 550, 300) To tie it all together, an html page is created using MarkupBuilder to showcase all of the results, organized by Province or Territory. def buildHtml(inputDirectory) { File inputDir = new File(inputDirectory) assert inputDir.exists() Writer writer = new StringWriter() MarkupBuilder builder = new MarkupBuilder(writer) builder.html { head { title('Number of Seedlings Planted by Ownership, Species') style(type: 'text/css') { mkp.yield(CSS) } } body { ul { AREAS.each { area -> String areaName = sanitizeName(area) div(class: 'area rounded-corners', id: areaName) { h2(PROVINCE_SHORT_FORM_MAPPINGS.find {it.value == area}.key) inputDir.eachFileMatch(~/.*$areaName\.png/) { img(src: } } } } script(type: 'text/javascript', src: '', '') script(type: 'text/javascript') { mkp.yield(JQUERY_FUNCTION) } } } writer.toString() } The generated html page assumes that all images are co-located in the same folder, presents four images per Province/Territory and, just for fun, uses JQuery to attach a click handler to each of the headers. Click on a header and the images in that div will animate into the background. I’m sure the actual JQuery being used could be improved upon, but it serves its purpose. Here’s a sample of the html output: <ul> <div class='area rounded-corners' id='NL'> <h2>Newfoundland and Labrador</h2> <img src='naturalResourcesCanadaNewSeedlings-Federal-NL.png' /> <img src='naturalResourcesCanadaNewSeedlings-PrivateLand-NL.png' /> <img src='naturalResourcesCanadaNewSeedlings-Provincial-NL.png' /> <img src='naturalResourcesCanadaNewSeedlings-Total-NL.png' /> </div> ... The resulting page looks like this in Firefox.  Source code and Links The source code is available on GitHub. So is the final resulting html page. The entire source required to go from Excel to charts embedded in an html page comes in at slightly under 300 lines of code and I don’t think the results are too bad for the couple of hours effort involved. Finally, the JSON results are also hosted on the GitHub pages for the project for anyone else who might want to delve into the data. Some reading related to this topic:Groovy loves POI and POI loves Groovy Writing batch import scripts with Grails, GSQL and GPars GSheets – A Groovy Builder based on Apache POI Groovy for the OfficeRelated links:Groovy inspect()/Eval for Externalizing Data Groovy reverse map sort done easy Five Cool Things You Can Do With Groovy ScriptsReference: JFreeChart with Groovy and Apache POI from our JCG partner Kelly Robinson at the The Kaptain on … stuff blog....

Spring @Configuration and FactoryBean

Consider a FactoryBean for defining a cache using a Spring configuration file: <cache:annotation-driven /> <context:component-scan base-package='org.bk.samples.cachexml'></context:component-scan> <bean id='cacheManager' class=''> <property name='caches'> <set> <ref bean='defaultCache'/> </set> </property> </bean> <bean name='defaultCache' class='org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean'> <property name='name' value='default'/> </bean> The factory bean ConcurrentMapCacheFactoryBean is a bean which is in turn responsible for creating a Cache bean. My first attempt at translating this setup to a @Configuration style was the following: @Bean public SimpleCacheManager cacheManager(){ SimpleCacheManager cacheManager = new SimpleCacheManager(); List<Cache> caches = new ArrayList<Cache>(); ConcurrentMapCacheFactoryBean cacheFactoryBean = new ConcurrentMapCacheFactoryBean(); cacheFactoryBean.setName('default'); caches.add(cacheFactoryBean.getObject()); cacheManager.setCaches(caches ); return cacheManager; } This did not work however, the reason is that here I have bypassed some Spring bean lifecycle mechanisms altogether. It turns out that ConcurrentMapCacheFactoryBean also implements the InitializingBean interface and does a eager initialization of the cache in the ‘afterPropertiesSet’ method of InitializingBean. Now by directly calling factoryBean.getObject() , I was completely bypassing the afterPropertiesSet method. There are two possible solutions: 1. Define the FactoryBean the same way it is defined in the XML: @Bean public SimpleCacheManager cacheManager(){ SimpleCacheManager cacheManager = new SimpleCacheManager(); List<Cache> caches = new ArrayList<Cache>(); caches.add(cacheBean().getObject()); cacheManager.setCaches(caches ); return cacheManager; }@Bean public ConcurrentMapCacheFactoryBean cacheBean(){ ConcurrentMapCacheFactoryBean cacheFactoryBean = new ConcurrentMapCacheFactoryBean(); cacheFactoryBean.setName('default'); return cacheFactoryBean; }In this case, there is an explicit FactoryBean being returned from a @Bean method, and Spring will take care of calling the lifecycle methods on this bean. 2. Replicate the behavior in the relevant lifecycle methods, in this specific instance I know that the FactoryBean instantiates the ConcurrentMapCache in the afterPropertiesSet method, I can replicate this behavior directly this way: @Bean public SimpleCacheManager cacheManager(){ SimpleCacheManager cacheManager = new SimpleCacheManager(); List<Cache> caches = new ArrayList<Cache>(); caches.add(cacheBean()); cacheManager.setCaches(caches ); return cacheManager; }@Bean public Cache cacheBean(){ Cache cache = new ConcurrentMapCache('default'); return cache; } Something to keep in mind when translating a FactoryBean from xml to @Configuration. Note: A working one page test as a gist is available here: package org.bk.samples.cache; import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.equalTo; import java.util.ArrayList; import java.util.List; import java.util.Random; import org.junit.Test; import org.junit.runner.RunWith; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.cache.Cache; import org.springframework.cache.annotation.Cacheable; import org.springframework.cache.annotation.EnableCaching; import org.springframework.cache.concurrent.ConcurrentMapCacheFactoryBean; import; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.ComponentScan; import org.springframework.context.annotation.Configuration; import org.springframework.stereotype.Component; import org.springframework.test.context.ContextConfiguration; import org.springframework.test.context.junit4.SpringJUnit4ClassRunner; @RunWith(SpringJUnit4ClassRunner.class) @ContextConfiguration(classes={TestSpringCache.TestConfiguration.class}) public class TestSpringCache { @Autowired TestService testService; @Test public void testCache() { String response1 = testService.cachedMethod('param1', 'param2'); String response2 = testService.cachedMethod('param1', 'param2'); assertThat(response2, equalTo(response1)); } @Configuration @EnableCaching @ComponentScan('org.bk.samples.cache') public static class TestConfiguration{ @Bean public SimpleCacheManager cacheManager(){ SimpleCacheManager cacheManager = new SimpleCacheManager(); List<Cache> caches = new ArrayList<Cache>(); caches.add(cacheBean().getObject()); cacheManager.setCaches(caches ); return cacheManager; } @Bean public ConcurrentMapCacheFactoryBean cacheBean(){ ConcurrentMapCacheFactoryBean cacheFactoryBean = new ConcurrentMapCacheFactoryBean(); cacheFactoryBean.setName('default'); return cacheFactoryBean; } } } interface TestService{ String cachedMethod(String param1,String param2); } @Component class TestServiceImpl implements TestService{ @Cacheable(value='default', key='#p0.concat('-').concat(#p1)') public String cachedMethod(String param1, String param2){ return 'response ' + new Random().nextInt(); } }Reference: Spring @Configuration and FactoryBean from our JCG partner Biju Kunjummen at the all and sundry blog....

Fixing Bugs that can’t be Reproduced

There are bugs that can’t be reproduced, or at least not easily: intermittent and transient errors; bugs that disappear when you try to look for them; bugs that occur as the result of a long chain of independent operations or cross-request timing. Some of these bugs are only found in high-scale production systems that have been running for a long time under heavy load. Capers Jones calls these bugs “abeyant defects” and estimates that in big systems, as much as 10% of bugs cannot be reproduced, or are too expensive to try reproducing. These bugs can cost 100x more to fix than a simple defect – an “average” bug like this can take more than a week for somebody to find (if they can be found at all) by walking through the design and code, and up to another week or two to fix. Heisenbugs One class of bugs that can’t be reproduced are Heisenbugs: bugs that disappear when you attempt to trace or isolate them. When you add tracing code, or step through the problem in a debugger, the problem goes away. In Debug It!, Paul Butcher offers some hope for dealing with these bugs. He says Heisenbugs are caused by non-deterministic behavior which in turn can only be caused by:Unpredictable initial state – a common problem in C/C++ code Interaction with external systems – which can be isolated and stubbed out, although this is not always easy Deliberate randomness – random factors can also be stubbed out in testing Concurrency – the most common cause of Heisenbugs today, at least in Java.Knowing this (or at least making yourself believe it while you are trying to find a problem like this) can help you decide where to start looking for a cause, and how to go forward. But unfortunately, it doesn’t mean that you will find the bug, at least not soon. Race Conditions Race conditions used to be problems only for systems programmers and people writing communications handlers. But almost everybody runs into race conditions today – is anybody writing code that isn’t multi-threaded anymore? Races are synchronization errors that occur when two or more threads or processes access the same data or resource without a consistent locking approach. Races can result in corrupted data – memory getting stomped on (especially in C/C++), changes applied more than once (balances going negative) or changes being lost (credits without debits) – inconsistent UI behaviour, random crashes due to null pointer problems (a thread references an object that has already been freed by another thread), intermittent timeouts, and in actions executed out of sequence, including time-of-check time-of-use security violations. The results will depend on which thread or process wins the race a particular time. Because races are the result of unlucky timing, and because they problem may not be visible right away (e.g., something gets stepped on but you don’t know until much later when something else tries to use it), they’re usually hard to understand and hard to make happen. You can’t fix (or probably even find) a race condition without understanding concurrency. And the fact that you have a race condition is a good sign that whoever wrote this code didn’t understand concurrency, so you’re probably not dealing with only one mistake. You’ll have to be careful in diagnosing and especially in fixing the bug, to make sure that you don’t change the problem from a race into a stall, thread starvation or livelock, or deadlock instead, by getting the synchronization approach wrong. Fixing bugs that can’t be reproduced If you’re think you’re dealing with a concurrency problem, a race condition or a timing-related bug, try introducing log pauses between different threads – this will expand the window for races and timing-related problems to occur, which should make the problem more obvious. This is what IBM Research’s ConTest tool does (or did – unfortunately, ConTest seems to have disappeared off of the IBM alphaWorks site), messing with thread scheduling to make deadlocks and races occur more often. If you can’t reproduce the bug, that doesn’t mean that you give up. There are still some things to look at and try. It’s often faster to find concurrency bugs, timing problems and other hard problems by working back from the error and stepping through the code – in a debugger or by hand – to build up your own model of how the code is supposed to work. “Even if you can’t easily reproduce the bug in the lab, use the debugger to understand the affected code. Judiciously stepping into or over functions based on your level of “need to know” about that code path. Examine live data and stack traces to augment your knowledge of the code paths.” Jeff Vroom, Debugging Hard Problems I’ve worked with brilliant programmers who can work back through even the nastiest code, tracing through what’s going on until they see the error. For the rest of us, debugging is a perfect time to pair up. While I’m not sure that pair programming makes a lot of sense as an everyday practice for everyday work, I haven’t seen too many ugly bugs solved without two smart people stepping through the code and logs together. It’s also important to check compiler and static analysis warnings – these tools can help point to easily-overlooked coding mistakes that could be the cause of your bug, and maybe even other bugs. Findbugs has statistical concurrency bug pattern checkers that can point out common concurrency bug patterns as well as lots of other coding mistakes that you can miss finding on your own. There are also dynamic analysis tools that are supposed to help find race conditions and other kinds of concurrency bugs at run-time, but I haven’t seen any of them actually work. If anybody has had success with tools like this in real-world applications I would like to hear about it. Shotgun Debugging, Offensive Programming, Brute-Force Debugging and other ideas Debug It! recommends that if you are desperate, just take a copy of the code, and try changing something, anything, and see what happens. Sometimes the new information that results from your change may point you in a new direction. This kind of undirected “Shotgun Debugging” is basically wishful thinking, relying on luck and accidental success, what Andy Hunt and Dave Thomas call “Programming by Coincidence”. It isn’t something that you want to rely on, or be proud of. But there are some changes that do make a lot of sense to try. In Code Complete, Steve McConnell recommends making “offensive coding” changes: adding asserts and other run-time debugging checks that will cause the code to fail if something “impossible” happens, because something “impossible” apparently is happening. Jeff Vroom suggests writing your own debugging code: “For certain types of complex code, I will write debugging code, which I put in temporarily just to isolate a specific code path where a simple breakpoint won’t do. I’ve found using the debugger’s conditional breakpoints is usually too slow when the code path you are testing is complicated. You may hit a specific method 1000?s of times before the one that causes the failure. The only way to stop in the right iteration is to add specific code to test for values of input parameters… Once you stop at the interesting point, you can examine all of the relevant state and use that to understand more about the program.” Paul Butcher suggests that before you give up trying to reproduce and fix a problem, see if there are any other bugs reported in the same area and try to fix them – even if they aren’t serious bugs. The rationale: fixing other bugs may clear up the situation (your bug was being masked by another one); and by working on something else in the same area, you may learn more about the code and get some new ideas about your original problem. Refactoring or even quickly rewriting some of the code where you think the problem might be can sometimes help you to see the problem more clearly, especially if the code is difficult to follow. Or it could possibly move the problem somewhere else, making it easier to find. If you can’t find it and fix it, at least add defensive code: tighten up error handling and input checking, and add some logging – if something is happening that can’t happen or won’t happen for you, you need more information on where to look when it happens again. Question Everything Finally, The Pragmatic Programmer tells us to remember that nothing is impossible. If you aren’t getting closer to the answer, question your assumptions. Code that has always worked might not work in this particular case. Unless you have a complete trace of the problem in the original report, it’s possible that the report is incomplete or misleading – that the problem you need to solve is not the problem that you are looking at. If you’re trapped in a real debugging nightmare, look outside of your own code. The bug may not be in your code, but in underlying third party code in your service stack or the OS. In big systems under heavy load, I’ve run into problems in the operating system kernel, web servers, messaging middleware, virtual machines, and the DBMS. Your job when debugging a problem like this is to make sure that you know where the bug isn’t (in your code), and try to come up with a simple a test case as possible that shows this – when you’re working with a vendor, they’re not going to be able to setup and run a full-scale enterprise app in order to reproduce your problem. Hard bugs that can’t be reproduced can blow schedules and service levels to hell, and wear out even your best people. Taking all of this into account, let’s get back to the original problem of how or whether to estimate bug fixes, in the next post. Reference: Fixing Bugs that can’t be Reproduced from our JCG partner Jim Bird at the Building Real Software blog....

Enterprise SOAP Services in a Web 2.0 REST/JSON World

With the popularlity of JSON and other NoSQL data standard formats, the complexity and in some cases the plain verbosity of XML formats are being shunned. However, XML and the abilities and standards that have formed around it have become key to the enterprise and their business processes. However, the needs of their customers require that they start to support multiple formats. To this end, tooling frameworks like Axis2 have started to add support for taking WSDL based services and generate JSON responses.Enterprises need to live in this post SOAP world, but leverage the expertise and SOA infrastructures they have developed over the years. Axis2 is one way to do it, but it doesn’t provide monitoring and policy support of the box. Another alternative is the Turmeric SOA project from ebay. Natively out of the box one can take a WSDL like, the one provided by the STAR standard organization, and add support not only from SOAP 1.1/1.2, but also for REST style services serving XML, JSON, and other any other data format you would need to support.There is a catch though, Turmeric SOA was not designed with full SOAP and the W3C web service stack in mind. It uses WSDL to only describe the operations and the data formats supported by the service. So advanced features like WS-Security, WS-Reliable Messaging, and XML Encryption are not natively built into Turmeric. Depending on your needs you will need to work with pipeline Handlers and enhance the protocol processors to support some of the advance features. However, these are items that can be worked around, and it can interoperate with existing web services to act as a proxy.As an example, the STAR organisation provides a web service specification that has been implemented in the automative industry to provide transports for their specifications. Using a framework like Turmeric SOA existing applications can be made available to trading partners and consumer of the service in multiple formats. As an example, one could provide data in RESTful xml: <?xml version='1.0' encoding='UTF-8'?> <ns2:PullMessageResponse xmlns:ms="" xmlns:ns2=""> <ns2:payload> <ns2:content id="1"/> </ns2:payload> </ns2:PullMessageResponse>Or one can provide the same XML represented in a JSON format: { "jsonns.ns2": "", "jsonns.ns3": "", "": "", "jsonns.xs": "", "jsonns.xsi": "", "ns2.PullMessageResponse": { "ns2.payload": { "ns2.content": [ { "@id": "1" } ] } } }The above is generated from the same web service, but with just a header changed to indicate the data format that should be returned. No actual changes to business logic or the web service implementation code itself has to change. In Turmeric this is handled with the Data Binding Framework and its corresponding pipeline handlers. With Axis2 this is a message request configuration entry. Regardless of how it is done, it is important to be able to leverage existing services, but provide the data in a format that your consumers require.For those that are interested, I’ve created a sample STAR Web Service that can be used with Turmeric SOA to see how this works. Code is available at github.While Turmeric handles the basics of the SOAP protocol, advance items like using and storing information in the soap:header are not as easily supported. You can get to the information, but because the use of WSDL in Turmeric Services are there just to describe the data formats and messages, the underlying soap transport support is not necessarily leveraged to the full specification. Depending on your requirements, Axis2 may be better, but Turmeric SOA provides additional items like Service Metrics, Monitoring Console, Policy Administration, Rate Limiting through XACML 2.0 based policies, and much more. If you already have existing web services written the W3C way, but need to provide data in other formats, Turmeric can be used along side. It isn’t a one or the other proposition. Leverage the tools that provide you the greatest flexibility to provide the data to your consumers, with the least amount of effort.Reference: Enterprise SOAP Services in a Web 2.0 REST/JSON World from our JCG partner David Carver at the Intellectual Cramps blog....

Using EasyMock or Mockito

I have been using EasyMock for most of time but recently I worked with a few people who were pretty much inclined to use Mockito. Not intending to use two frameworks for the same purpose in the same project I adopted Mockito. So for the last couple of months I have been using Mockito and here is my comparative analysis of the two.The people with whom I have work cite reasons of test readability for using Mockitio but I have a different opinion on the same. Suppose we have the following code that we intend to test : public class MyApp { MyService service; OtherService otherService;void operationOne() { service.operationOne(); }void operationTwo(String args) { String operationTwo = otherService.operationTwo(args); otherService.operationThree(operationTwo); }void operationThree() { service.operationOne(); otherService.operationThree("success"); } }class MyService { void operationOne() {} }class OtherService { public String operationTwo(String args) { return args; }public void operationThree(String operationTwo) {} } Now let me write a simple test case for this class using EasyMock and the using Mockito. public class MyAppEasyMockTest { MyApp app; MyService service; OtherService otherService;@Before public void initialize() { service = EasyMock.createMock(MyService.class); otherService = EasyMock.createMock(OtherService.class); app = new MyApp(); app.service = service; app.otherService = otherService; }@Test public void verifySimpleCall() { service.operationOne(); EasyMock.replay(service); app.operationOne(); EasyMock.verify(service); } } public class MyAppMockitoTest { MyApp app; MyService service; OtherService otherService;@Before public void initialize() { service = Mockito.mock(MyService.class); otherService = Mockito.mock(OtherService.class); app = new MyApp(); app.service = service; app.otherService = otherService; }@Test public void verifySimpleCall() { app.operationOne(); Mockito.verify(service).operationOne(); } } This is a really simple test and I must say the Mockito one is more readable . But according to the classic testing methodology the Mockito test is not complete. We have verified the call that we are looking for but if tomorrow I change the source code by adding one more call to service the test would not break. void operationOne() { service.operationOne(); service.someOtherOp(); } Now this makes me feel that the tests are not good enough. But thankfully Mockito gives the verifyNoMoreInteractions that can be used to complete the test. Now let me write a few more test for the MyApp class. public class MyAppEasyMockTest { @Test public void verifyMultipleCalls() { String args = "one"; EasyMock.expect(otherService.operationTwo(args)).andReturn(args); otherService.operationThree(args); EasyMock.replay(otherService); app.operationTwo(args); EasyMock.verify(otherService); }@Test(expected = RuntimeException.class) public void verifyException() { service.operationOne(); EasyMock.expectLastCall().andThrow(new RuntimeException()); EasyMock.replay(service); app.operationOne(); }@Test public void captureArguments() { Capture<String> captured = new Capture<String>(); service.operationOne(); otherService.operationThree(EasyMock.capture(captured)); EasyMock.replay(service, otherService); app.operationThree(); EasyMock.verify(service, otherService); assertTrue(captured.getValue().contains("success")); }}public class MyAppMockitoTest { @Test public void verifyMultipleCalls() { String args = "one"; Mockito.when(otherService.operationTwo(args)).thenReturn(args); app.operationTwo(args); Mockito.verify(otherService).operationTwo(args); Mockito.verify(otherService).operationThree(args); Mockito.verifyNoMoreInteractions(otherService); Mockito.verifyZeroInteractions(service); }@Test(expected = RuntimeException.class) public void verifyException() { Mockito.doThrow(new RuntimeException()).when(service).operationOne(); app.operationOne(); }@Test public void captureArguments() { app.operationThree(); ArgumentCaptor capturedArgs = ArgumentCaptor .forClass(String.class); Mockito.verify(service).operationOne(); Mockito.verify(otherService).operationThree(capturedArgs.capture()); assertTrue(capturedArgs.getValue().contains("success")); Mockito.verifyNoMoreInteractions(service, otherService); } } These are some practical scenarios of testing where we would like to assert arguments, Exceptions etc. If I look and compare the ones written using EasyMock with the ones using Mockito I tend to feel that both the tests are equal in readability, none of them do a better task.The large number of expect and return calls in EasyMock make the tests not readable and the verify statements of Mockito often compromise over test readility. As per the documentation of Mockito the verifyZeroInteractions, verifyNoMoreInteractions should not be used in every test that you write but if I leave them out of my tests then my tests are not good enough.Moreover in tests everything thing should be under the control of the developer i.e. how the interaction are happening and what interactions are happening. In EasyMock this aspect is more visible as the devloper must put down all of these interaction in his code but in Mockito, the framework takes care of all interactions and the developer is just concerned with their verification ( if any). But this can lead to testing scenarios where the developer is not under control of all interactions.There are few nice things that Mockito has like the JunitRunner that can be used to create Mocks of all the required dependencies. It is a nice way of removing some of the infrastructure code and EasyMock should also have one. @RunWith(MockitoJUnitRunner.class) public class MyAppMockitoTest { MyApp app; @Mock MyService service; @Mock OtherService otherService;@Before public void initialize() { app = new MyApp(); app.service = service; app.otherService = otherService; } } Conclusion:Since I have used both frameworks, I feel that except for simple test cases both EasyMock and Mockito lead to test cases that equal in readability. But EasyMock is better for the unit testing as it forces the developer to take control of things. Mockito due to its assumptions and considerations hides this control under the carpet and thus is not a good choice. But Mockito offers certaing things that are quite useful(eg. junitRunner, call chaining) and EasyMock should have one in its next release. Reference: using EasyMock or Mockito from our JCG partner Rahul Sharma at the The road so far… blog blog. ...

ADF : Dynamic View Object

Today I want to write about dynamic view object which allow me to change its data source (SQL query) and attributes at run time. I will use oracle.jbo.ApplicationModule :: createViewObjectFromQueryStmt method to do this issue. I will present how to do this step by step Create View Object and Application Module   1- Right click on Model project and choose New2- Choose from left pane “ADF Business Component” , then from list choose “View Object” and click “OK” button3- Enter “DynamicVO” in “Name” and choose “Sql Query” radio button and click “Next” button.4- Write in Select field “select * from dual” and click “Next” button until reach Window “Step 8 of 9″  5- Check “Add to Application Module” check box and click “Finish” button.Implement Changes in Application Module 1- Open application module “AppModule”, then open Java tab and check “Generate Application Module Class AppModuleImpl” check box2- Open Class and Add the below method for dynamic view object public void changeDynamicVoQuery(String sqlStatement) { ViewObject dynamicVO = this.findViewObject("DynamicVO1"); dynamicVO.remove(); dynamicVO = this.createViewObjectFromQueryStmt("DynamicVO1", sqlStatement); dynamicVO.executeQuery(); }3- Open “AppModule” then open Java tab and Add changeDynamicVoQuery method to Client InterfaceTest Business Component   1- Right click on AppModue in Application navigator and choose Run from drop down list.2- Right click on AppModule in left pane and choose Show from drop down lsit Write “Select * from Emp” in sqlStatement parameter Click on Execute button, The result will be Success .3- Click double click on DynamicVO1 in left pane, it will display the data of DynamicVO and display data which I entered “Select * from Emp” before not “Select * from dual” that was used in design time of view object.  To use dynamic view objects in ADF Faces, you should use ADF Dynamic Table or ADF Dynamic Form. You can download sample application from here Reference: ADF : Dynamic View Object from our JCG partner Mahmoud A. ElSayed at the Dive in Oracle blog....
Java Code Geeks and all content copyright © 2010-2015, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: