Featured FREE Whitepapers

What's New Here?


Good & Bad Patterns in Development and Operations

As part of my role at a new company I’ve been asked to provide feedback about structuring Dev & Ops as well as what sorts of things work and don’t. I certainly don’t claim to have all the answers, but I’ve seen some very functional and some very dysfunctional organizations. I’ve spent a fair amount of time thinking about what works & why. Below is a cleaned up version of a message I sent to our CEO who asked for my thoughts on what does and doesn’t work. This was intended as scaffolding for further discussion so I didn’t go into deep details. If you want more details on any particular area just throw some comments out there.   I realize not all these issues are black & white to many folks – there are gray areas. My goal with this message was to drive conversation. I figure this is probably review to many folks, but maybe it’ll help someone. First, there are some very simple goals that all these bullets drive toward & they’re somewhat exclusive to SaaS companies:Customers should continuously receive value from Developers as code is incrementally pushed out Developers should get early feedback from customers on changes by enabling features for customers to test We can address problems for customers very quickly – often in a matter of hours We can inspect and understand customer behavior very deeply, gathering exceptional detail about how they use the service. We can swap out components & substantially change the underlying software without the customer knowing (if we do it right) We can measure how happy customers are with changes as we make them based on behavior & feedbackThe lists below are what I feel make that possible (Good) and what inhibit it (Bad) : Culture & Communication Good:Stand ups. Retrospectives Small, self-formed teams (Let folks work on their area of passion) Use Information Radiators whenever possible (Kanban boards, stats on big monitors, etc) Decisions by teams, Leaders facilitate consensus Discovering what doesn’t work is part of finding the right solution, not something to fear. Hackathons allow Developers to do things they are passionate about Hire for personality & team fit first, technical ability second Data driven decisions, strive to have facts to back up decisions. Make the right behavior the easiest thing to do – build a low resistance path to doing the right thing.Bad:Top down decision making Strict role assignments & Silos Fear of not getting it right the first time Hiring for technical ability thinking team fit will come later Creating process out of fear that makes it difficult to do the right thing.Eliminate Manual Processes Good:Continuous Deployment / Delivery Fully automated testing Test Driven Development Fully automated system monitoring, configuration & provisioning Separate Deploy & Release (Feature toggles) Deploy from master, do not branch (Forces particular behaviors)Bad:Manual testing by a QA Team – sometimes it’s necessary, but should be avoided Deploying off a branch, slows things down & allows for other bad behaviors Writing tests after writing code, code isn’t written with testing in mind Developers relying on other teams to perform tasks that could be automated. Processes that are the result of fear rather than necessary business process.If it moves, measure it Good:Collect high resolution metrics about everything you possibly can Developers can add new metrics by pushing new code, do not rely on additional configuration by other teams. Graphs & metrics can be seen by anyone – Developers should rely on these. There should be individuals or teams who are passionate about data visualization & analysis. Dev teams rely on these metrics to make decisions, help identify what metrics are important Developers watch metrics after pushing new code, watch for trend changes (Devs take responsibility for availability)Bad:Operations has to configure new metrics after developers have added support for them (Manual) Operations monitors metrics & asks Dev teams when they think there’s a problem Developers don’t look at metrics unless something is brought to their attention Code doesn’t expose metrics until someone else asks for itAnd here is the long version of all of that… #1 Culture & Communication Above all else I consider these most important. I think most problems in other areas of the business can be overcome if you do well in these areas. Rally has been, by far, the best example of a very successful model that I’ve seen in this area. They aren’t unique – there are other companies with similar models & similar successes. Main pointsStand ups. By far the most effective tool for keeping everyone in touch. As teams grow you have to break them apart, so you have a 2nd standup where teams can bring cross-team items to share. Projects are tackled by relatively small, typically self-formed teams. Get individuals who are interested in working in an area together & they feed on each others passion. Perform retrospectives. This gives individuals & small groups the ability to voice concerns in a way that fosters resolution. There’s an art to facilitating this but it works well when done right. It also allows recognition of things that are done well. Use open information radiators – it should be easy to see what’s going on by looking at status somewhere vs. having to ask for status, go to meetings, etc. Kanban boards are great for this. Leaders exist to facilitate and help drive consensus but decisions are largely made by teams, not leaders. This makes being a leader harder, but it makes the teams more empowered. Accept that things may not work & the team and company will adjust when things do not work. This makes it easy to try new things & easy for people to vocalize when they think it isn’t working. If it’s hard to change process then people are more resistant to try new things. This goes back to retrospectives for keeping things in check. Also important in this are “spikes” or time boxed efforts explicitly designed to explore possibilities. Give developers time to pursue their own projects for the company. Many awesome features have come out of Hackathons where developers spent their own time to build something they were passionate about. Hire for personality fit first. I have seen many awesome people find a special niche in a company because they grew into a role that you couldn’t hire for – but what made that possible was that they worked well with the team as an individual. Hiring for technical skill also means you lose that skill when that person leaves, I would prefer to have cross-functional teams. Data driven decisions. This helps keep emotion and “I think xyz” out of the discussion & focuses on the data we do and do not have. If we don’t have data we either get more or acknowledge we may not be making the right decision but we’re going to move forward. Make the right thing the easiest thing. I’ve seen too many companies put process out there that makes the “right thing” really difficult, so it gets bypassed. The right thing should be an express train to done – very little resistance and very easy to do. It’s when you start wanting to do things differently that it should become harder, more painful.Also, everyone owns the quality of the service. This includes availability, performance, user experience, cost to deliver, etc. At my last company, there was exceptional collaboration between Operations, Engineering and Product (and across engineering teams) on all aspects of the service and there was a strong culture of shared ownership & very little finger pointing. If you want more details on this specific to Rally I wrote a blog post with some more info: Blog Post #2 Obsessively eliminate manual process – let computers do what they are good at. This is so much easier to do up front. There should be as little manual process as possible standing between a developer adding value for customers (writing code) and that code getting into production. There may be business process that controls when that feature is enabled for customers – but the act of deploying & testing that code should not be blocked by manual process. I refer to this as separating “Deploy” from “Release” – those are two very different things. Testing should only be manual to invalidate assumptions, validating assumptions should be automatic When we assume that if x is true then y will occur, there should be a test to validate that this is true. Testers should not manually validate these sorts of things unless there is just no way to automate them (rare). Testers are valuable to invalidate assumptions. Testers should be looking at the assumptions made by Developers and helping identify those assumptions that may not always be correct. Too many organizations rely on manual testing because it’s “easier”, but it has some serious drawbacks:You can only change your system as fast as your team can manually test it – which is very slow. Your testing is done by humans who make mistakes and don’t behave predictably so you get inaccurate results. The # of tests will only grow over time, requiring either more humans or more time, or both. It doesn’t scale.Over time the software quality gets lower, takes longer to test, and the test results become less reliable. This is a death spiral for many companies who eventually find it very hard to make changes due to fear & low confidence in testing. Avoiding this requires developers spend more time up front writing automated tests. This means developers might spend 60-70% of their time developing tests vs. writing code – this is the cost of doing business if you want to produce high quality software. That may seem excessive, but the tradeoffs are significant:Much higher code quality which stays high (those tests are always run, so re-introduced bugs (regressions) get caught) Faster developer on boarding, the tests describe how the code should behave and act as documentation. Refactoring code becomes easier because you know the tests describe what it should do. Each commit to the codebase is fully tested, allowing nearly immediate deployment to production if done right. Problems that make it into product feed back into more tests & continually improve code quality.Much of the time developing tests is spent thinking about how to solve the problem, but you are also writing code with the intent of making it testable. Code is often written differently when the developer knows tests need to pass vs. someone manually testing it. It’s much harder to come along later and write tests for existing code. You will hear me talk about Continuous Deployment & Continuous Integration – I feel these practices are extremely important to driving the above “good” behaviors. If you strive for Continuous Deployment then everything else falls into place without much disagreement because it has to be that way. This has a lot of benefits beyond what’s listed above:Value can be delivered to customers in days or hours instead of weeks or months Developers can get immediate feedback on their change in production New features can be tuned & tweaked while they are fresh in a developers mind You can focus on making it fast to resolve defects, no matter how predictable they are, rather than trying to predict all the ways things might go wrong. Most of the tools and behaviors that enable Continuous Deployment scale to very large teams & very frequent deployments. Amazon is a prime example of this, deploying something, somewhere, about every 11 seconds. Many companies that are in the 30-100 engineer size talk about deploying tens of times per day. This also impacts how you hire QA/Testers. This is a longer discussion, but you want to hire folks who can help during the test planning phase & can help Developers write better tests. Ideally your testers are also developers & work in a way that’s similar to Operations, helping your Developers to be better at their jobs.#3 If it moves, measure it I mentioned above, two big advantages a SaaS organization has are the amount it can learn about how customers use the product & the ability to change things rapidly. Both of these require obsessive measurement of everything that is going on so that you know if things are better or worse. Some of these metrics are about user behavior & experience to understand how the service is being used. Other metrics are about system performance & behavior. The ability to expose some % of your customer base to a new feature & measure their feedback to that is huge. Plenty of companies have perfected the art of A/B testing but at the heart of it is the ability to measure behavior. Similar to testing, the software has to be built in a way which allows this behavior to be measured. System performance similarly requires a lot of instrumentation to identify changes in trends, to identify problems areas in the application & to verify when changes actually improve the situation. I’ve been at too many companies where they simply had no idea how the system was performing today compared to last week to understand if things were better or worse. At my last company I saw a much more mature approach to this measurement which worked pretty well, but it required investment. They had two people fully dedicated to performance analysis & customer behavior analysis.   Reference: Good & Bad Patterns in Development and Operations from our JCG partner Aaron Nichols at the Operation Bootstrap blog. ...

Running unit tests and integration tests separately with Maven Failsafe and TestNG

Recently for my new pet project I decided that I would like to have some tests executed during standard mvn test and some other ones only during different phase, let’s call it integration phase. I googled and googled and nothing seemed to work, so after struggling with making my setup work I’ve decided to write down my findings how I was able to configure TestNG with Maven to run integration and unit tests separately. Basic (not working) setup For integration testing there is a Maven Failsafe plugin that is supposed to do what we want out of the box. Unfortunately, things are not as easy and straightforward as we might expect. Let’s look at basic setup that should work: We have two test classes, one with IT postfix that will indicate that this is our integration test: import org.testng.annotations.Test;import static org.fest.assertions.Assertions.assertThat;public class ExampleUnitTest {@Test public void shouldPass() { assertThat(false).isFalse(); } } and import org.testng.annotations.Test;import static org.fest.assertions.Assertions.assertThat;public class StatusTestIT {@Test public void shouldFail() { assertThat('aaa').isEqualTo(''); } } and now let’s declare plugin mentioned above in our pom.xml: <project><!-- ... --> <build> <plugins> <!-- ... --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>2.9</version> </plugin><plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.13</version> <configuration> <includes> <include>**/*IT.java</include> </includes> </configuration> </plugin> </plugins> </build> </project> After that we should see two things. mvn test should pass as we have a one green test in ExampleUnitTest class, but mvn failsafe:integration-test should fail with StatusTestIT red. But as I said before, this does not work. mvn test looks as expected, but second Maven execution is passing as well showing that no test were run. Plugin seems to omit our completely valid test… Small fix doing its job After testing different approaches I found out that all we need to make this setup work is to add plugin execution to an integration-test phase of maven life cycle. So tiny change, but now our integration tests are executed only when we call mvn integration-test. <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-failsafe-plugin</artifactId> <version>2.13</version> <configuration> <includes> <include>**/*IT.java</include> </includes> </configuration> <executions> <execution> <id>failsafe-integration-tests</id> <phase>integration-test</phase> <goals> <goal>integration-test</goal> </goals> </execution> </executions> </plugin> And that’s all, we could execute integration tests only when we really want to. Of course there are some things to remember when adopting setup described above:our unit tests must have Test postfix, if you use something else, we have to configure surefire plugin using include during integration test unit tests are executed too. This is a small flaw in this setup, but exclude **/*Test.java seem not to work  Reference: Running unit tests and integration tests separately with Maven Failsafe and TestNG from our JCG partner Tomasz Dziurko at the Code Hard Go Pro blog. ...

Design Doesn’t Emerge from Code

I know a lot of people who are transitioning to Agile or already following Agile development methods. Almost all of them are using something based on Scrum at the core, mixed with common XP practices like Continuous Integration and refactoring and automated unit testing – pretty much how Mike Cohn says things should be done in his book Succeeding with Agile. Emergent Design in Scrum and XP But none of them are doing emergent design as Cohn describes it, or as Kent Beck explains how design is done in Extreme Programming: trying to get away without any upfront design and architecture work, coding features right away and relying on   test-first development, refactoring and technical spikes to work out a design on the fly, one week or two weeks at a time.“For the first iteration, pick a set of simple, basic stories that you expect will force you to create the whole architecture. Then narrow your horizon and implement the stories in the simplest way that can possibly work. At the end of this exercise you will have your architecture. It may not be the architecture you expected, but then you will have learned something.” Kent Beck You don’t need upfront architecture and design? Maybe it’s because everyone I know is working at scale – building big enterprise systems and online systems used by lots of customers, systems that have a lot of constraints and dependencies. Many of them are working on brownfield projects where you need to understand the existing system’s design and implementation first, before you can come up with a new design and before you can make any changes. Performance-critical, mission-critical systems in highly-regulated environments. Emergent, incremental design doesn’t work for these cases. And it doesn’t scale to large projects or any project that has to be delivered along with other projects and that has specified integration points and dependencies – which is pretty much every project that I’ve ever worked on. Bob Martin, another one of the people who helped define how Agile development should be done, thinks that this incremental approach to design is, well…“One of the more insidious and persistent myths of agile development is that up-front architecture and design are bad; that you should never spend time up front making architectural decisions. That instead you should evolve your architecture and design from nothing, one test-case at a time. Pardon me, but that’s Horse Shit.”Martin goes on to say that: “there are architectural issues that need to be resolved up front. There are design decisions that must be made early. It is possible to code yourself into a very nasty cul-de-sac that you might avoid with a little forethought.”Architecture and Design in Disciplined Agile Delivery The way that most people that I know approach Agile development is better described by Scott Ambler in Disciplined Agile Delivery, a model for scaling Agile to larger systems, projects and organizations. As Ambler’s research shows, almost all teams (86%) spend at least some time (on average a month or more) on upfront on planning, scoping and architecture envisioning – what he calls the “Inception Phase” (borrowing from Rational’s Unified Process) or what most others call “Sprint 0” or “Iteration 0”. This is time spent to understand the scope of the system at a high-level at least, and the constraints and dependencies that the project needs to work within. Time to model the main chunks of the system and their interfaces, and to choose a technical direction to start with. Upfront architectural and design work doesn’t have to take a lot of time. As Ambler points out, for many teams (except for some startups), a lot of architectural decisions have already been made for you: “In practice, it’s likely you won’t need to do much initial architectural modeling: a large majority of project teams work with technical architecture decisions that were made years earlier. Your organization will most likely already have chosen a network infrastructure, an application-server platform, a database platform, and so on. In these situations your team will need to invest the time to understand those decisions and to consider whether the existing infrastructure build-out is sufficient (and if not, identify potential improvements).”It’s when you have a real greenfield development project, when you don’t have anything to leverage and you’re doing something completely new, that you should spend more time on upfront thinking about design – not less. Can you “be Agile” without Emergent Design? Of course you can. Bob Martin points out that there’s nothing in “Agile Development” that says that you shouldn’t do design upfront – as much design as you need to for the size of the system that you are building and the environment that you are working in. You can and should do iterative, incremental design and development starting with a plan of where you are going and how you think that you are going to get there. As you go along and prove out your design and respond to feedback and deal with changes in requirements, this is where incremental design actually does come into play – handling changes in direction, filling in gaps, correcting misunderstandings. The design will change and maybe become something that you didn’t expect. But you need a place to start from – designs don’t just emerge from code.   Reference: Design Doesn’t Emerge from Code from our JCG partner Jim Bird at the Building Real Software blog. ...

Custom error pages for expired conversations involving CDI and JSF

It’s been a while since I last blogged. I keep thinking of blogging something technical but end up getting busy with other things. This last week there was a very interesting discussion at the coderanch forums. It was even more interesting because it involved JBoss. Developers familiar with Java EE web applications would know that the web application deployment descriptor (web.xml) allows you to specify ‘error pages’ which the container will display when a specific exception (class) or a error code is thrown by the server for a web request. Here’s a quick example of how it looks like:     <web-app> ... <!-- A custom error page for error code == 500 --> <error-page> <error-code>500</error-code> <location>/my-foo-bar-500-page.html</location> </error-page> <!-- A custom error page for exception type org.myapp.foo.bar.MyException --> <error-page> <exception-type>org.myapp.foo.bar.MyException</exception-type> <location>/my-foo-bar-exception-page.html</location> </error-page> ... </web-app> Simple enough – a couple of custom error pages defined for a specific error code and an exception type respectively. All of this works fine. In current days, more and more programming models and frameworks come into picture while developing web applications. CDI and JSF are some of those. CDI has this concept of scopes (ex: request scope, session scope, application scope, conversation scope). We won’t go into the details of what those are and when those are used, but let’s consider conversation scope in this blog since that’s what the discussion was about in the forum thread that prompted this blog. So CDI allows multiple requests to be part of a ‘conversation scope’. A conversation has a ‘start’ and an ‘end’, both of which can be managed by the application. When the application involves JSF, any conversation (id) gets auto-propagated to the JSF request(s). Apart from an explicit start/end demarcation of conversations, a conversation can also timeout. A request which refers to a conversation which has ended or timed out will run into an exception. So we know have a bit of background on CDI conversation scope. So let’s consider a case where the application wants to present a good looking page when the ‘conversation no longer present’ exception is thrown (maybe because of a timeout). We have seen how to write a web.xml for error-page configurations – it would be as simple as: <web-app> ... <!-- A custom error page for exception type org.jboss.weld.context.NonexistentConversationException --> <error-page> <exception-type>org.jboss.weld.context.NonexistentConversationException</exception-type> <location>/my-foo-bar-exception-page.html</location> </error-page> ... </web-app> Simple enough. The org.jboss.weld.context.NonexistentConversationException is the exception class type which gets thrown when the conversation has timed out (note that we are assuming that the web application is relying on Weld as the CDI spec implementation library). The above configuration works fine. The my-foo-bar-exception-page.html gets displayed when the org.jboss.weld.context.NonexistentConversationException is thrown. BUT, let’s now consider that we want to involve JSF in the error page just like the other parts of our application. So let’s point the error-page to a URL pattern which maps to the JSF servlet: ...

On The Illusion of Controlling People’s Time

Software Engineering is going through an identity crisis. The old school has failed because of too much control, moving people away from what they really have to deliver. The new school, or agile fever, is more focused on humanities, directing all the attention to human beings. This approach practically decimated the old school, showing that writing software is not an exact science. It is actually unpredictable and cannot be done without creativity, inspiration and transpiration. Maybe the old school, which insists to exist, got confused by the fact that software execution, data processing, and system and hardware design are engineering stuff, but it doesn’t mean that developing those stuff is also engineering. On the contrary… it’s human and social.   When I see managers trying to measure the time spent on people’s daily activities, I understand they are taking into consideration only the transpiration of their teams and completely ignoring the creativity and inspiration. These two last aspects are so subjective that it’s hard to compare their manifestation on different people. How to compare artists? We may find artists’ comparisons out there, but they heavily rely on who is producing the comparison. Let’s ask ourselves: why on earth would someone use quantitative criteria, such as time and money, to measure and compare subjective matters?Measurements are only useful when we’re capable of making comparisons with them in an objective way. We compare the date of our birth with the calendar to know our age. In times of crisis, we pay attention to our water, gaz and electricity consumption and make comparisons, finding ways to save money. Climate is measured because we need to decide which clothes to wear or which precautions to take by comparing with previous experiences or common sense. Notice we put measured subjects into perspective to understand the real dimension, the scale, the direction, and then we’re able to reason about trends, behaviours, issues, patterns, etc. When it comes to subjective matters, we might be able to make comparisons, but measurements are just useless in this case. Actually, we can come closer with a lot of statistical methods, multi-variable analysis, multi-criteria craziness, but do we really have time and resources to produce all these analyses in a daily basis, and even take the risk of being probabilistically far from what it’s expected? It sounds complicated, isn’t it? That’s because it is. In software development, every activity is different from one another. It’s impracticable to precisely compare two activities even if they look the same and are done by the same person. It’s all subjective, otherwise it would be easy to put the project on schedule by adding more people in the team. We know that since 1975, when Frederick Brooks published ‘The Mythical Man-Month. So, when a manager asks their developers to inform the time spent in every activity, he/she is in fact collecting lies (or non-truths). If he/she prepares a report with those collected times, this is going to be the spread of an even bigger lie (or useless information).But why lies? There are two good explanations for that:most people don’t track exactly how much time they spent on their activities, thus they actually inform safer figures, which are usually higher than reality; time control gives a feeling of surveillance, making developers believe their manager wants to know whether they are really working or not, thus they smartly add some extra time to mask non-productive time.Managers should also be aware that time control also kills creativity. When developers have new ideas, potentially useful for the project, they cannot simply turn them into activities because of scope constraints or just because those ideas are not clear enough to be described. Without activities, developers cannot report their time. Therefore, if they spend too much time working on their ideas, it will look like huge time gaps for their managers. Eventually, managers are moved to implement time control because of visible procrastination. I remember once when a new project was starting, a big room was allocated to a team of 12 people, and the project manager asked the team to choose their places in the room. There were 15 places available, 12 with monitors facing the wall and 3 with monitors facing the door. Guess which places were left available? That’s right! The three ones facing the door. I remember the symphony of mouse clicks and sudden movements when the manager walked through the room. That was so funny! But time control wouldn’t solve that problem because inputing time on activities is also a distraction. Anything that is not building deliverables is a distraction. We may never eliminate them, but we can always reduce them. The solution for procrastination, in a single word, is: Motivation! But, is time control totally out of sense? It has some sense when the work is fully based on time, which is the case of consultants payed hourly. This is about their business model, not software engineering. That’s a good practice for consultants to inform to customers how many hours they had spent because it’s directly related to their payments. Notice that this kind of service is for a short period of time and for highly specialised skills. However, we are talking about software projects that last longer. Finding ways to motivate your team is the best strategy to put them on track and fully productive. Every form of control that threats their feelings will put them down. Motivation doesn’t need to be monetary or entertaining. Sometimes, it’s enough to show their wins, how important is their work for society, how it changed people’s lives, and why it’s important to keep doing what they’re doing.   Reference: http://www.hildeberto.com/2013/01/on-illusion-of-controlling-peoples-time.html from our JCG partner Hildeberto Mendonca at the Hildeberto’s Blog blog. ...

Iterator Pattern and Java

Hello all, in this post we’ll be checking on the Iterator Pattern. A design pattern that I know many of you have already used, but maybe you didn’t realize it was pattern or didn’t know its great value. According to the book Head First Design:The Iterator Pattern provides a way to access the elements of an aggregate object sequentially without exposing its underlaying representation.Whaaaaat? Well, it says that no matter what data structure (arrays, lists, hashtables,etc.) you are using, you can always traverse it in the same way if you implement this pattern. It gives you a uniform way of accessing the elements of your data structures (aggregates), but you don’t have to know what kind of data structure   you are traversing… nice! Also, it sets the responsibility of iteration on the Iterator object not on your data structure, which simplifies the coding in your data structure. Let’s check the classic class diagram for the Iterator pattern:The actual class diagram for the Iterator Pattern has a few changes, specially in the Iterator class (interface), where we now have different methods as we’ll see in a minute, but first, lets review each of the previous classes (or interfaces):Aggregate: This is the base class (or interface) of our data structures, you can think of it as the java.util.Collection interface, which defines a lot of methods for the collection classes. ConcreteAggregate: This is the concrete data structure that we’ll be iterating, for example, java.util.ArrayList, java.util.Vector, etc. Iterator: Base class (or interface) for iterators. You can find one in Java at java.util.Iterator. You can notice that the Java version has different methods that we’ll be discussing later in this post. Here you define de methods you need in order to traverse the data structures. ConcreteIterator: As you want to traverse different data structures you need different iterators. So a concreteIterator is an Iterator for the data structure you want to traverse.Now, let’s take a look at the Java implementation of the Iterator Pattern. The following diagram was generated using Architexa’s Free Tool for Understanding Code and it shows the relations between some classes of the Java Collections Framework where we can see a structure similar to the classic class diagram:The above diagram shows only one implementation of the pattern in Java, there are a lot more, but they always use the java.util.Iterator interface; this is the interface you should use in your implementations of the Iterator Pattern when coding in Java. Let’s compare both diagrams:Classic DiagramJava exampleAggregatejava.util.AbstractListConcreteAggregatejava.util.ArrayList, java.util.VectorIteratorjava.util.IteratorConcreteIteratorPrivate internal class at java.util.AbstractList You can’t see this class in  the JavaDocs, but it is there in the source code: java.util.ItrNotice that the methods of the Iterator object in the Java example are different from the methods in the classic class diagram:There is no +First() method. If you need to go to the first element, you have to instance a new iterator. The +IsDone() method has been renamed to +hasNext(). The +Next() and +CurrentItem() has been merged into +next(). The +remove() method has been added.So, if you ever have to work with different data structures and need a uniform way to traverse them and/or access their items, think about the Iterator Pattern: //... in a class /** * Traverse a list, hashtable, vector, etc. what ever that implements * the Iterator Pattern */ public void traverse(Iterator iter) { while(iter.hasNext()) { System.out.println(iter.next()); } } Of course, you will always have to create the ConcreteIterator class for your data structure, but if you are using classes from the Java Collections Framework, it’s already done. One last thing, remember the most important OO Principle of all: Always use the simplest solution that meets your needs, even if it doesn’t include a pattern. Resources: Freeman Eric and Freeman Elisabeth and Sierra Kathy and Bates Bert (2004). Head First Design Patterns. United States of America: O’Reilly Media, Inc.   Reference: Iterator Pattern and Java from our JCG partner Alexis Lopez at the Java and ME blog. ...

10 XML Interview questions and answers for Java Programmer

XML Interview questions are very popular in various programming job interviews, including Java interviews for web developer. XML is a matured technology and often used as standard for transporting data from one platform other. XML Interview questions contains questions from various XML technologies like XSLT which is used to transform XML files, XPATH, XQuery and fundamentals of XML e.g. DTD or Schema. In this article we will see 10 frequently asked XML Interview questions and answers from above topics. These questions are mostly asked in various Java interviews but they are equally useful in other programming interviews like C, C++, Scala or any other programming language. Since XML is not tied with any programming language   and like SQL its one of the desired skill in programmer, it make sense to practice some XML questions before appearing in any technical job interview. XML Interview Questions and Answers Here is my list of some common and frequently asked Interview questions on XML technologies. Questions on this list is not very tough but touches some important areas of XML technologies e.g. DTD, XML Schema, XSLT transformations, XPATH evaluation, XML binding, XML parsers and fundamentals of XML e.g. namespace, validation, attribute, elements etc. Question 1: What is XML ? Answer : XML stands for Extensible Markup language which means you can extend XML based upon your needs. You can define custom tags like <books>, <orders> etc in XML easily as opposed to other mark-up language like HTML where you need to work with predefined tags e.g. <p> and you can not use user defined tag. Though structure of XML can be standardize by making use of DTD and XML Schema. XML is mostly used to transfer data from one system to another e.g. between client and server in enterprise applications. Question 2: Difference between DTD and XML Schema? Answer : There are couple of differences between DTD and XML Schema e.g. DTD is not written using XML while XML schema are xml documents in itself, which means existing XML tools like XML parsers can be used to work with XML schema. Also XML schema is designed after DTD and it offer more types to map different types of data in XML documents. On the other hand DTD stands for Document Type definition and was a legacy way to define structure of XML documents. Question 3: What is XPath ? Answer : XPath is an XML technology which is used to retrieve element from XML documents. Since XML documents are structured, XPath expression can be used to locate and retrieve elements, attributes or value from XML files. XPath is similar to SQL in terms of retrieving data from XML but it has it’s own syntax and rules. See here to know more about How to use XPath to retrieve data from XML documents. Question 4: What is XSLT? Answer : XSLT is another popular XML technology to transform one XML file to other XML, HTML or any other format. XSLT is like a language which specifies its own syntax, functions and operator to transform XML documents. Usually transformation is done by XSLT Engine which reads instruction written using XSLT syntax in XML style sheets or XSL files. XSLT also makes extensive use of recursion to perform transformation. One of the popular example of using XSLT is for displaying data present in XML files as HTML pages. XSLT is also very handy to transforming one XML file into another XML document. Question 5: What is element and attribute in XML? Answer : This can be best explained by an example. let’s see a simple XML snippet <Orders> <Order id="123"> <Symbol> 6758.T</Symbol> <Price> 2300</Price> <Order> <Orders> In this sample XML id is an attribute of element. Here , and are also other elements but they don’t have any attribute. Question 6: What is meaning of well formed XML ? Answer : Another interesting XML interview question which most appeared in telephonic interviews. A well formed XML means an XML document which is syntactically correct e.g. it has a root element, all open tags are closed properly, attributes are in quotes etc. If an XML is not well formed, it may not be processed and parsed correctly by various XML parsers. Question 7: What is XML namespace? Why it’s important? Answer : XML namespace are similar to package in Java and used to provide a way to avoid conflict between two xml tags of same name but different sources. XML namespace is defined using xmlns attribute at top of the XML document and has following syntax xmlns:prefix=’URI’. later that prefix is used along with actual tag in XML documents. Here is an example of using XML namespace : <root xmlns:inst="http://instruments.com/inst" <inst:phone> <inst:number>837363223</inst:number> </inst:phone> </root>Question 8: Difference between DOM and SAX parser ? Answer : This is another very popular XML interview question, not just in XML world but also on Java world. Main difference between DOM and SAX parser is the way they parse XML documents. DOM creates an in memory tree representation of XML documents during parsing while SAX is a event driven parser. See Difference between DOM and SAX parser for more detailed answer of this question. Question 9: What is a CDATA section in XML? Answer : I like this XML Interview questions for its simplicity and importance, yet many programmer doesn’t know much about it. CDATA stands for character data and has special instruction for XML parsers. Since XML parser parse all text in XML document e.g. <name>This is name of person</name> here even though value of tag <name> will be parsed because it may contain XML tags e.g. <name><firstname>First Name</firstname></name>. CDATA section is not parsed by XML parser. CDATA section starts with “<![CDATA[” and finishes with “]]>”. Question 10: What is XML data Binding in Java? Answer : XML binding in Java refers to creating Java classes and object from XML documents and then modifying XML documents using Java programming language. JAXB , Java API for XML binding provides convenient way to bind XML documents with Java objects. Other alternatives for XML binding is using open source library e.g. XML Beans. One of the biggest advantage of XML binding in Java is to leverage Java programming capability to create and modify XML documents. This list of XML Interview questions and answers are collected from programmers but useful to anyone who is working in XML technologies. Important of XML technologies like XPath, XSLT, XQuery is only going to increase because of platform independent nature of XML and it’s popularity of transmitting data over cross platform. Though XML has disadvantage like verbosity and size but its highly useful in web services and transmitting data from one system to other where bandwidth and speed is of secondary concern. Other Interview questions articles from Javarevisited Top 30 UNIX and Linux command Interview questions – Answered20 design pattern and software design Interview questions with answers 10 Oracle Interview questions with answers 15 Java multi-threading Interview questions with answers asked in Investment banks Top 10 Java String Interview questions – Answered  Reference: 10 XML Interview questions and answers for Java Programmer from our JCG partner Javin Paul at the Javarevisited blog. ...

When git ignores your… .gitignore?

I feel like I should start this post saying that I absolutely love git. If you’ve never heard of it, is a source control system like CVS or Subversion but, unlike those two, is a distributed version control system. I’m not going to get into much details about the history and capabilities of git but if you’re curious about it you can go to http://git-scm.com/book which is an amazing resource with everything from intro to advanced concepts. I imagine that by now most professional software developers use some form of version control at their daily jobs but you shouldn’t stop there. I use git for personal, one-man projects as well. While some people might think that this could be an overkill, I completely disagree. There is nothing like the comfort in knowing that all the history   of all your files is safe and ready to be brought back if you ever need them. We all make mistakes sometimes after all. With git this is as easy as writing 3 simple commands: mkdir myNewProject cd myNewProject git init That’s it! Every file you create or modify on “myNewProject” will be now tracked by git. A pretty useful feature that you get with every source control tool is the possibility to ignore certain files from the tool’s tracking. Generally speaking, you don’t want to get into your code repository any file that can be computed as a result of another file. In a typical Java Maven project this would be for example the “target” directory, or if you are using Eclipse the “.metadata”, “.project” or “.settings” files. In git the easiest way to do this is to have a special file named “.gitignore” at the root of your project with all the exclusion rules you want to set. The syntax of this file is fairly straightforward. You can also have a “.gitignore” file for each subdirectory in your project, but this is less common. A tricky thing with git and ignore rules is that if the file you want to ignore is already being tracked by git then adding it to “.gitignore” won’t make git to automatically forget about the file. To illustrate this point, consider the following example. First we create a repository with two initial files and commit them: mkdir gitExample cd gitExample touch file1 file2 git init git add . git commit -m 'Initial commit' Let’s create now the .gitignore file to try to ignore “file2? and commit that: echo file2 > .gitignore git add . git commit -m 'Added gitignore for file2' Now, let’s modify “file2? and see what happens: echo 'Hello World' >> file2 git status We get:# On branch master # Changes not staged for commit: # (use 'git add ...' to update what will be committed) # (use 'git checkout -- ...' to discard changes in working directory) # # modified: file2 # no changes added to commit (use 'git add' and/or 'git commit -a')Git is effectively still tracking file2 even though is already on our .gitignore. Like I said before, this happens because git was already tracking the file when we added it to our ignores. So let’s see what happens when we add the “.gitignore” before adding the file to git: mkdir gitExample cd gitExample touch file1 file2 git init echo 'file2' > .gitignore git status And now we get:# On branch master # # Initial commit # # Untracked files: # (use 'git add ...' to include in what will be committed) # # .gitignore # file1 nothing added to commit but untracked files present (use 'git add' to track)Cool! No mention of file2 anywhere! But if, as our first example, we forgot to add the files to our .gitignore initially? How do we stop git from tracking them? A nice command we can use for this cases is git rm --cached file. In our first example: git rm --cached file2 git commit -m 'Removed file2' If we now modify the file again and do a git status we get: echo 'Hello World Again' >> file2 git status# On branch master nothing to commit (working directory clean)Exactly what we wanted! Note that this little command will remove the file from git index but it won’t do anything with your working copy. That means that you will still have the file on your directory but the file won’t be a part of the git repository anymore. This also implies that the next time you do a push to a remote repository the file won’t be pushed and, if it already existed on the remote repository, it will be deleted. This is fine for a typical use case when you added a file that you never intended to have on the repository. But consider this other use case. In a lot of projects the developers upload into the remote central repository a set of config files for their IDE with default values for things like code formatting and style checking. But at the same time, when developers clone the repository they customize those config files to suit their personal preferences. However, those changes that apply to each particular developer should not be commited and pushed back to the repository. The problem with using git rm --cached here is that, while each developer will still have their own copy, the next time they push to the server they’ll remove the default config from there. In cases like this, there is another pretty useful command that will do the trick: git update-index --assume-unchanged <file>. Let’s see that with an example: mkdir gitExample cd gitExample touch file1 file2 git init git add . git commit -m 'Initial commit' There we have our default “file2?. Now let’s use the git update-index command and make some changes to the file: git update-index --assume-unchanged file2 echo 'Hello World' >> file2 git status The result:# On branch master nothing to commit (working directory clean)Magic! Changes to our file are no longer seen by git and the original file is still on the repository to be cloned by other users with its default value.   Reference: When git ignores your… .gitignore? from our JCG partner Jose Luis at the Development the way it should be blog. ...

Java Memory Model and optimisation

Overview Many developers of multi-threaded code are familiar with the idea that different threads can have a different view of a value they are holding, this not the only reason a thread might not see a change if it is not made thread safe. The JIT itself can play a part. Why do different threads see different values? When you have multiple threads, they will attempt to minimise how much they will interact e.g. by trying to access the same memory. To do this they have a separate   local copy e.g. in Level 1 cache. This cache is usually eventually consistent. I have seen short periods of between one micro-second and up to 10 milli-seconds where two threads see different values. Eventually the thread is context switched, the cache cleared or updated. There is no guarantee as to when this will happen but it is almost always much less than a second. How can the JIT play a part? The Java Memory Model says there is no guarantee that a field which is not thread safe will ever see an update. This allows the JIT to make an optimisation where a value only read and not written to is effectively inlined into the code. This means that even if the cache is updated, the change might not be reflected in the code. An example This code will run until a boolean is set to false. >static class MyTask implements Runnable { private final int loopTimes; private boolean running = true; boolean stopped = false;public MyTask(int loopTimes) { this.loopTimes = loopTimes; }@Override public void run() { try { while (running) { longCalculation(); } } finally { stopped = true; } }private void longCalculation() { for (int i = 1; i < loopTimes; i++) if (Math.log10(i) < 0) throw new AssertionError(); } }public static void main(String... args) throws InterruptedException { int loopTimes = Integer.parseInt(args[0]); MyTask task = new MyTask(loopTimes); Thread thread = new Thread(task); thread.setDaemon(true); thread.start(); TimeUnit.MILLISECONDS.sleep(100); task.running = false; for (int i = 0; i < 200; i++) { TimeUnit.MILLISECONDS.sleep(500); System.out.println("stopped = " + task.stopped); if (task.stopped) break; } } This code repeatedly performs some work which has no impact on memory. The only difference it makes is how long it takes. By taking longer, it determines whether the code in run() will be optimised before or after running is set to false. If I run this with 10 or 100 and -XX:+PrintCompilation I see 111 1 java.lang.String::hashCode (55 bytes) 112 2 java.lang.String::charAt (29 bytes) 135 3 vanilla.java.perfeg.threads.OptimisationMain$MyTask :longCalculation (35 bytes) 204 1 % ! vanilla.java.perfeg.threads.OptimisationMain$MyTask :run @ 0 (31 bytes) stopped = false stopped = false stopped = false stopped = false ... many deleted ... stopped = false stopped = false stopped = false stopped = false stopped = false If I run this with 1000 you can see that the run() hasn’t been compiled and the thread stops 112 1 java.lang.String::hashCode (55 bytes) 112 2 java.lang.String::charAt (29 bytes) 133 3 vanilla.java.perfeg.threads.OptimisationMain $MyTask::longCalculation (35 bytes) 135 1 % vanilla.java.perfeg.threads.OptimisationMain $MyTask::longCalculation @ 2 (35 bytes) stopped = true Once the thread has been compiled, the change is never seen even though the thread will have context switched etc. many times. How to fix this The simple solution is to make the field volatile. This will guarantee the field’s value consistent, not just eventually consistent which is what the cache might do for you. Conclusion While there are many examples of question like; Why doesn’t my thread stop? The answer has more to do with Java Memory Model which allows the JIT “inline” the fields that it does the hardware and having multiple copies of the data in different caches.   Reference: Java Memory Model and optimisation from our JCG partner Peter Lawrey at the Vanilla Java blog. ...

Scala: Collections 1

This post contains some info on Scala’s collections. Problem? We want a function that will take an List of Rugby players as input and return those players names that play for Leinster and can run the 100 meters from the fastest to the slowest. Step 1: Have a representation for a Rugby player. Ok so it’s obvious we want something like a POJO to represent a Rugby player. This   representation should have a player’s name, their team and the time they can the 100 meters in. Let’s use Scala case class construct which removes the need for boiler plate code. case class RugbyPlayerCaseClass(team: String, sprintTime100M: BigDecimal, name: String) Step 2: Create some rugby players val lukeFitzGerald = RugbyPlayerCaseClass('Leinster', 10.2, 'Luke Fitzgerald'); val fergusMcFadden = RugbyPlayerCaseClass('Leinster', 10.1, 'Fergus McFadden'); val rog = RugbyPlayerCaseClass('Leinster', 12, 'Ronan O'Gara'); val tommyBowe = RugbyPlayerCaseClass('Ulster', 10.3, 'Tommy Bowe'); val leoCullen = RugbyPlayerCaseClass('Leinster', 15, 'Leo Cullen'); The code above should be self explanatory. The various rugby players are instantiated. Note the inferred typing. There is no need to declare any of the rugby players as RugbyPlayers types. Instead, it is inferred. Another thing that is interesting is the keyword val is used. This means the reference is immutable It is the equivalent to final in Java. Step 3: Write the function def validByAge(in: List[RugbyPlayerCaseClass]) = in.filter(_.team == 'Leinster').sortWith(_.sprintTime100M < _.sprintTime100M).map(_.name); Key points regarding this function:The function begins with def keyword signifying a function declartion. A List of RugbyPlayerCaseClass instances are taken in as input. The List type is a Scala type. The return type is optional. In this case it is not explictly specified as it is inferred. The part to the left of the = is what the function does. In this case the function invokes three difference collection operators..filter(_.team =='Leinster) – this iterates over every element in the List. In each iteration the _ is filled in with the current value in the List. If the team property of the current Rugby player is Leinster the element is included in the resulting collection. .sortWith(_.sprintTime100M < _.sprintTime100M) – sortWith is a special method which we can use to sort collections. In this case, we our sorting the output fromthe previous collection operator and we are sorting based on the sprintTime for 100M. .map(_.name) – this maps every element from the output of the sort operator to just ther name property.The function body does not need to be surrounded by {} because it is only one line code. There is no return statement needed. In Scala, whatever the last line evaluates to will be returned. In this example since there only is one line, the last line is the first line.Finally – put it all together. object RugbyPlayerCollectionDemos { def main(args: Array[String]){ println('Scala collections stuff!'); showSomeFilterTricks(); }// Case class remove need for boiler plater code. case class RugbyPlayerCaseClass(team: String, sprintTime100M: BigDecimal, name: String)def showSomeFilterTricks() {// team: String, sprintTime100M: Int, name: String val lukeFitzGerald = RugbyPlayerCaseClass('Leinster', 10.2, 'Luke Fitzgerald'); val fergusMcFadden = RugbyPlayerCaseClass('Leinster', 10.1, 'Fergus McFadden'); val rog = RugbyPlayerCaseClass('Munster', 12, 'Ronan O'Gara'); val tommyBowe = RugbyPlayerCaseClass('Ulster', 10.3, 'Tommy Bowe'); val leoCullen = RugbyPlayerCaseClass('Leinster', 15, 'Leo Cullen');println(validByAge(List(lukeFitzGerald, fergusMcFadden, rog, tommyBowe, leoCullen)));}def validByAge(in: List[RugbyPlayerCaseClass]) = in.filter(_.team == 'Leinster').sortWith(_.sprintTime100M < _.sprintTime100M).map(_.name);} The above program will output: Scala collections stuff! List(Luke Fitzgerald, Fergus McFadden, Leo Cullen) Something similar in Java Pre Java 8, to implement the same functionality in Java would be a lot more code. public class RugbyPLayerCollectionDemos { public static void main(String args[]) { RugbyPLayerCollectionDemos collectionDemos = new RugbyPLayerCollectionDemos(); collectionDemos.showSomeFilterTricks(); }public void showSomeFilterTricks() { // team: String, sprintTime100M: Int, name: String final RugbyPlayerPOJO lukeFitzGerald = new RugbyPlayerPOJO('Leinster', new BigDecimal(10.2), 'Luke Fitzgerald'); final RugbyPlayerPOJO fergusMcFadden = new RugbyPlayerPOJO('Leinster', new BigDecimal(10.1), 'Fergus McFadden'); final RugbyPlayerPOJO rog = new RugbyPlayerPOJO('Munster', new BigDecimal(12), 'Ronan O'Gara'); final RugbyPlayerPOJO tommyBowe = new RugbyPlayerPOJO('Ulster', new BigDecimal(10.3), 'Tommy Bowe'); final RugbyPlayerPOJO leoCullen = new RugbyPlayerPOJO('Leinster', new BigDecimal(15), 'Leo Cullen');ListrugbyPlayers = Arrays.asList(lukeFitzGerald, fergusMcFadden, rog, tommyBowe, leoCullen);System.out.println(filterRugbyPlayers(rugbyPlayers)); }/** * Return the names of Leinster Rugby players in the order of their sprint times. */ public ListfilterRugbyPlayers(Listpojos) { ArrayListleinsterRugbyPlayers = new ArrayList();for (RugbyPlayerPOJO pojo: pojos) { if (pojo.getTeam().equals('Leinster')) { leinsterRugbyPlayers.add(pojo); } }RugbyPlayerPOJO [] rugbyPlayersAsArray = leinsterRugbyPlayers.toArray(new RugbyPlayerPOJO[0]);Arrays.sort(rugbyPlayersAsArray, new Comparator() { public int compare(RugbyPlayerPOJO rugbyPlayer1, RugbyPlayerPOJO rugbyPlayer2) { return rugbyPlayer1.getSprintTime100M().compareTo(rugbyPlayer2.getSprintTime100M()); } });ListrugbyPlayersNamesToReturn = new ArrayList();for (RugbyPlayerPOJO rugbyPlayerPOJO: rugbyPlayersAsArray) { rugbyPlayersNamesToReturn.add(rugbyPlayerPOJO.getName()); }return rugbyPlayersNamesToReturn; }class RugbyPlayerPOJO { private BigDecimal sprintTime100M; private String team; private String name;public RugbyPlayerPOJO(String team, java.math.BigDecimal sprintTime100M, String name) { this.name = name; this.sprintTime100M = sprintTime100M; this.team = team; }public BigDecimal getSprintTime100M() { return sprintTime100M; }public String getTeam() { return team; }public String getName() { return name; } } } Does Java 8 help out? Yes. According to the Project Lambda specsJava 8 will have similar looking filter,map and sort functions. The functionality in this post in Java 8 would look something like: ListrugbyPlayers = Arrays.asList(lukeFitzGerald, fergusMcFadden, rog, tommyBowe, leoCullen); //... //... ListfilteredPLayersNames = rugbyPlayers.filter(e -> e.getTeam.equals('Leinster')). sorted((a, b) -> a.getSprintTime100M() - b.getSprintTime100M()).mapped(e -> {return e.getName();}).into(new List<>()); So Java 8 is definetly catching up a great deal in this regard. But will it be enough?   Reference: Scala: Collections 1 from our JCG partner Alex Staveley at the Dublin’s Tech Blog blog. ...
Java Code Geeks and all content copyright © 2010-2015, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: