Featured FREE Whitepapers

What's New Here?

neo4j-logo

Neo4j: Cypher – Avoiding the Eager

  Although I love how easy Cypher’s LOAD CSV command makes it to get data into Neo4j, it currently breaks the rule of least surprise in the way it eagerly loads in all rows for some queries even those using periodic commit. This is something that my colleague Michael noted in the second of his blog posts explaining how to use LOAD CSV successfully:  The biggest issue that people ran into, even when following the advice I gave earlier, was that for large imports of more than one million rows, Cypher ran into an out-of-memory situation. That was not related to commit sizes, so it happened even with PERIODIC COMMIT of small batches.I recently spent a few days importing data into Neo4j on a Windows machine with 4GB RAM so I was seeing this problem even earlier than Michael suggested. Michael explains how to work out whether your query is suffering from unexpected eager evaluation:If you profile that query you see that there is an “Eager” step in the query plan. That is where the “pull in all data” happens.You can profile queries by prefixing the word ‘PROFILE’. You’ll need to run your query in the console of /webadmin in your web browser or with the Neo4j shell. I did this for my queries and was able to identify query patterns which get evaluated eagerly and in some cases we can work around it. We’ll use the Northwind data set to demonstrate how the Eager pipe can sneak into our queries but keep in mind that this data set is sufficiently small to not cause issues. This is what a row in the file looks like: $ head -n 2 data/customerDb.csv OrderID,CustomerID,EmployeeID,OrderDate,RequiredDate,ShippedDate,ShipVia,Freight,ShipName,ShipAddress,ShipCity,ShipRegion,ShipPostalCode,ShipCountry,CustomerID,CustomerCompanyName,ContactName,ContactTitle,Address,City,Region,PostalCode,Country,Phone,Fax,EmployeeID,LastName,FirstName,Title,TitleOfCourtesy,BirthDate,HireDate,Address,City,Region,PostalCode,Country,HomePhone,Extension,Photo,Notes,ReportsTo,PhotoPath,OrderID,ProductID,UnitPrice,Quantity,Discount,ProductID,ProductName,SupplierID,CategoryID,QuantityPerUnit,UnitPrice,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued,SupplierID,SupplierCompanyName,ContactName,ContactTitle,Address,City,Region,PostalCode,Country,Phone,Fax,HomePage,CategoryID,CategoryName,Description,Picture 10248,VINET,5,1996-07-04,1996-08-01,1996-07-16,3,32.38,Vins et alcools Chevalier,59 rue de l'Abbaye,Reims,,51100,France,VINET,Vins et alcools Chevalier,Paul Henriot,Accounting Manager,59 rue de l'Abbaye,Reims,,51100,France,26.47.15.10,26.47.15.11,5,Buchanan,Steven,Sales Manager,Mr.,1955-03-04,1993-10-17,14 Garrett Hill,London,,SW1 8JR,UK,(71) 555-4848,3453,\x,"Steven Buchanan graduated from St. Andrews University, Scotland, with a BSC degree in 1976. Upon joining the company as a sales representative in 1992, he spent 6 months in an orientation program at the Seattle office and then returned to his permanent post in London. He was promoted to sales manager in March 1993. Mr. Buchanan has completed the courses ""Successful Telemarketing"" and ""International Sales Management."" He is fluent in French.",2,http://accweb/emmployees/buchanan.bmp,10248,11,14,12,0,11,Queso Cabrales,5,4,1 kg pkg.,21,22,30,30,0,5,Cooperativa de Quesos 'Las Cabras',Antonio del Valle Saavedra,Export Administrator,Calle del Rosal 4,Oviedo,Asturias,33007,Spain,(98) 598 76 54,,,4,Dairy Products,Cheeses,\x MERGE, MERGE, MERGE The first thing we want to do is create a node for each employee and each order and then create a relationship between them. We might start with the following query: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) This does the job but if we profile the query like so… PROFILE LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row WITH row LIMIT 0 MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) …we’ll notice an ‘Eager’ lurking on the third line: ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | employee, order, UNNAMED216 | MergePattern | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | employee, employee, order, order | MergeNode; :Employee; MergeNode; :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ You’ll notice that when we profile each query we’re stripping off the periodic commit section and adding a ‘WITH row LIMIT 0′. This allows us to generate enough of the query plan to identify the ‘Eager’ operator without actually importing any data. We want to split that query into two so it can be processed in a non eager manner: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row WITH row LIMIT 0 MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, employee, order, order | MergeNode; :Employee; MergeNode; :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ Now that we’ve created the employees and orders we can join them together: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, order, UNNAMED216 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(0) | 0 | 0 | order, order | :Order | ==> | Filter(1) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(1) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ Not an Eager in sight! MATCH, MATCH, MATCH, MERGE, MERGE If we fast forward a few steps we may now have refactored our import script to the point where we create our nodes in one query and the relationships in another query. Our create query works as expected: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (product:Product {productId: row.ProductID}) ==> +-------------+------+--------+----------------------------------------------------+--------------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+----------------------------------------------------+--------------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, employee, order, order, product, product | MergeNode; :Employee; MergeNode; :Order; MergeNode; :Product | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+----------------------------------------------------+------------------------------------------------------------ We’ve now got employees, products and orders in the graph. Now let’s create relationships between the trio: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MATCH (product:Product {productId: row.ProductID}) MERGE (employee)-[:SOLD]->(order) MERGE (order)-[:PRODUCT]->(product) If we profile that we’ll notice Eager has sneaked in again! ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | order, product, UNNAMED318 | MergePattern | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | employee, order, UNNAMED287 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(product,productId) == Property(row,ProductID) | ==> | NodeByLabel(0) | 0 | 0 | product, product | :Product | ==> | Filter(1) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(1) | 0 | 0 | order, order | :Order | ==> | Filter(2) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(2) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ In this case the Eager happens on our second call to MERGE as Michael identified in his post:The issue is that within a single Cypher statement you have to isolate changes that affect matches further on, e.g. when you CREATE nodes with a label that are suddenly matched by a later MATCH or MERGE operation.We can work around the problem in this case by having separate queries to create the relationships: LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, order, UNNAMED236 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(0) | 0 | 0 | order, order | :Order | ==> | Filter(1) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(1) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (order:Order {orderId: row.OrderID}) MATCH (product:Product {productId: row.ProductID}) MERGE (order)-[:PRODUCT]->(product) ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | order, product, UNNAMED229 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(product,productId) == Property(row,ProductID) | ==> | NodeByLabel(0) | 0 | 0 | product, product | :Product | ==> | Filter(1) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(1) | 0 | 0 | order, order | :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ MERGE, SET I try to make LOAD CSV scripts as idempotent as possible so that if we add more rows or columns of data to our CSV we can rerun the query without having to recreate everything. This can lead you towards the following pattern where we’re creating suppliers: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (supplier:Supplier {supplierId: row.SupplierID}) SET supplier.companyName = row.SupplierCompanyName We want to ensure that there’s only one Supplier with that SupplierID but we might be incrementally adding new properties and decide to just replace everything by using the ‘SET’ command. If we profile that query, the Eager lurks: ==> +----------------+------+--------+--------------------+----------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+--------------------+----------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | | PropertySet | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | supplier, supplier | MergeNode; :Supplier | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+--------------------+----------------------+ We can work around this at the cost of a bit of duplication using ‘ON CREATE SET’ and ‘ON MATCH SET': USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (supplier:Supplier {supplierId: row.SupplierID}) ON CREATE SET supplier.companyName = row.SupplierCompanyName ON MATCH SET supplier.companyName = row.SupplierCompanyName ==> +-------------+------+--------+--------------------+----------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+--------------------+----------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | supplier, supplier | MergeNode; :Supplier | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+--------------------+----------------------+ With the data set I’ve been working with I was able to avoid OutOfMemory exceptions in some cases and reduce the amount of time it took to run the query by a factor of 3 in others. As time goes on I expect all of these scenarios will be addressed but as of Neo4j 2.1.5 these are the patterns that I’ve identified as being overly eager. If you know of any others do let me know and I can add them to the post or write a second part.Reference: Neo4j: Cypher – Avoiding the Eager from our JCG partner Mark Needham at the Mark Needham Blog blog....
akka-logo

Akka Notes – Child Actors and ActorPath – 6

Actors are completely hierarchical. Whatever Actors that you create HAS to be a child of some other Actor. Let’s analyze that a bit : Path Say, we create an ActorRef using ActorSystem.actorOf and try to print it’s path.       val actorSystem=ActorSystem("SupervisionActorSystem") val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor]) println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/$a As you see, a path looks very similar to a file path in a file system.akka here is fixed because all these are addresses of Akka Actors – more like file:// or http:// prefix (nothing to do with protocol though). The SupervisionActorSystem is just the name of your ActorSystem that you created. We’ll talk about the user in the next section. The $a is the name of your Actor that the system generated for you. How would you like if your operating system generated random file names for your files? You’d obviously hate it because you would want to refer to that name in future. So, let’s give it a proper meaningful name to it :val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor], "teacherActor") println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/teacherActor That’s it. Now, the path actually makes sense.Child Actors Similar to top level actors which we create out of ActorSystem, we could also create child actors out of the ActorContext. In fact, the power of Actors’s fault tolerance primarily lie within leveraging the Actor hierarchy and the ability of a parent to manage the life of child actors. Assume you have a TeacherSupervisor and you would like to create a TeacherActor to be a child of the Supervisor, you do a ActorContext.actorOf instead of a ActorSystem.actorOf: class TeacherSupervisor extends Actor with ActorLogging { val teacherActor=context.actorOf(Props[TeacherActor], "teacherActor") ... ... Frankly, in any application, you will be creating a whole lot of child actors than top level actors – which means that you’ll be calling a lot more actorContext.actorOf than actorSystem.actorOf.You’ll notice that the path of the child actor is akka://SupervisionActorSystem/user/teacherSupervisor/teacherActor, which is very similar to the path you get when you create a child folder within a parent folder. When do you create child Actors? You generally create child actors when a particular task is composed of a subtask or multiple subtasks. You also create a child actor when a particular task to be executed by the parent is error-prone and you would want to isolate it (so that if it fails, you could recover). When there is no parent child relationship between tasks, then you DON’T create child actors. Also, there’s nothing which is stopping a child actor from creating children to delegate its subtasks. Actors and their creation are really cheap but the power that comes with it is amazing (we’ll see about this while we talk about supervision). Now what is that user in the path? For lack of a creativity, let me compare the ActorSystem to a Unix file system – with a / root folder and all those /etc, /usr, /bin and various other folders. ActorSystem are much like that. It creates a few top level Actors – the most important being the root Actor with the path /, the user Actor with the path /user and a system Actor with the path /system. (there’s also a /deadLetters that represent the DeadLetterActorRef. We saw this in our previous post) Codewise, the ActorSystem composes three Actors inside it (via ActorRefProvider). Those are the root for ALL the Actors that gets created under the ActorSystem.systemGuardian actor – root of all actors under /system guardian actor – root of all actors under /user and rootGuardian Actor – root of both the systemGuardian and the userGuardian actors./** * Reference to the supervisor of guardian and systemGuardian; .... */ def rootGuardian: InternalActorRef/** * Reference to the supervisor used for all top-level user actors. */ def guardian: LocalActorRef/** * Reference to the supervisor used for all top-level system actors. */ def systemGuardian: LocalActorRef/user (aka) user guardian Any Actor that you create in your program like the StudentActor or the TeacherActor using the ActorSystem‘s actorOf method would directly fall under /user. That’s the reason your teacherActor in the first part of this write-up had the path slug /user/teacherActor. /system (aka) system guardian The system guardian shuts itself down when it notices that the userGuardian is dead. Makes sense considering if the userGuardian is down, then all the business actors under it is also down and therefore all administrative actors needs to be gone too. We could see two distinct places where System Actors are being created – I mean actors under the /system hierarchy.Like we saw earlier, any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor. The DeadLetter Actor wraps each message as a DeadLetter and publishes it to the EventStream. One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Now, the DeadLetterListener is a system Actor with path /system/deadLetterListener. Remember the TestEventListener that we created in our previous write-up to subscribe to log messages in the EventStream? They are System actors too. In fact, all akka.loggers are created as System actors.class TeacherTest extends TestKit(ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]"""))) ... ... The documentation here says that any Actor that is configured in the configuration files and created and deployed into the ActorSystem while it starts, falls under the /system umbrella. Let me update this post when I find something interesting around this / (aka) root guardian As we saw earlier, the / Actor is the parent of the user and the system guardians. TRIVIA Technically, there’s a superfluous parent for the root actor too. This Actor’s only job is to shut down the entire ActorSystem if the root actor fails. Since it’s strictly not considered within the Actor hierarchy, the Akka team calls it : private[akka] val theOneWhoWalksTheBubblesOfSpaceTime: InternalActorRef = new MinimalActorRef { ...Reference: Akka Notes – Child Actors and ActorPath – 6 from our JCG partner Arun Manivannan at the Rerun.me blog....
java-logo

Why to use String

Recently I was tutoring juniors during a training session. One of the task was to write a class that can dwarwle maps based on some string key. The result one of the juniors created contained the following method:                 void dwarwle(HashMap<String,Dwarwable> mapToDwarwle, String dwarwleKey){ for( final Entry<String, Dwarwable> entry : mapToDwarwle.entrySet()){ dwarwle(entry.getKey(),entry.getValue(),dwarwleKey); } } The code was generally ok. The method to dwarwle an individual dwarwable entry using the actual key it is assigned to in the hash map and the dwarwle key is factored to a separate method. It is so simple that I do not list here. The variable names are also meaningful so long as long you know what actually dwarwling is. The method is short and readable, but the argument list expects a HashMap instead of a Map. Why do we want to restrict the caller to use a HashMap? What if the caller has a TreeMap and for good reason. Do we want to have separate method that can dwarwle TreeMap? Certainly not.Expect the interface, pass the implementation.The junior changed the code replacing HashMap to Map but after five minutes or so this clever lady raised her hand and had the following question: “If we changed HashMap to Map, why did not we change String to CharSequence?” It is not so easy to answer a question like that when it comes out of the blue. The first thing that came up in my mind is that the reason is that we usually do it that way and that is why. But that is not a real argument, at least I would not accept anything like that and I also except my students not to accept such answer. It would be very dictator style anyway. The real answer is that the parameter is used as a key in a map and the key of a map should be immutable (at least mutation should be resilient to equals and hashcode calculation). CharSequence is an interface and an interface in Java (unfortunately) can not guarantee immutability. Only implementation can. String is a good, widely known and well tested implementation of this interface and therefore can be a good choice. There is a good discussion about it on stackoverflow. In this special case we expect the implementation because we need something immutable and we “can not” trust the caller to pass a character sequence implementation that is immutable. Or: we can, but it has a price. If a StringBuilder is passed and modified afterwards our dwarwling library may not work and a blame war may start. When we design an API and a library we should also think about not only the possible but also about the actual, average use.A library is as good as it is used and not as it could be used.This can also be applied to other products, not only libraries but this may lead too far (physics and weapon).Reference: Why to use String from our JCG partner Peter Verhas at the Java Deep blog....
java-logo

Java Extension Mechanism Loads All JARs

The Java Extension Mechanism is described in the Java Tutorial as a “standard, scalable way to make custom APIs available to all applications running on the Java platform.” As described in Understanding Extension Class Loading, “the extension framework makes use of the class-loading delegation mechanism” with extension classes loaded after the bootstrap classes in rt.jar (and related JARs) but before the classes loaded from the typical classpath. The extension directory works a bit like the classpath in that its part of the class loading mechanism and classes available within JARs in the extension directory are made available to Java applications. There are some key, differences, however, and some of these are highlighted next.  Characteristic Classpath Extension Mechanism (Optional Packages)Scope Typically Application-Specific-classpath/-cp java.class.path Executable JAR Manifest’s Class-PathPotentially All JREs on HostCLASSPATH environment variableAll JVMs Running in Specific JREjava.ext.dirsAll Host’s JREsSolaris: /usr/jdk/packages/lib/ext Linux: /usr/java/packages/lib/ext Windows: %SystemRoot%\Sun\Java\lib\extHow Specified .jar FilesExplicitly specified by name (including .jar) Wildcard (*) matching all all JAR files with .jar extensions.class FilesDirectory containing .class files specifiedAll JAR files (even if extension other than .jar or no extension at all) in designated directories are loaded.Class Loading Order After bootstrap and extensions loading. After bootstrap but before classpath.One of the most significant observations worth some more emphasis is that the extension mechanism will pick up all JAR format files in the extension directory even if the file does not have a .jar extension. The implication of this is that while one can change the name of a JAR located in a classpath directory to have an extension other than .jar so that the wildcard does not pick it up, this technique will not work with the extension directory. I’m going to use some simple examples in this post to demonstrate some of these differences. The next two code listings are for a very simple HelloWorld class and a main application class called Main that uses the HelloWorld class. HelloWorld.java public class HelloWorld { @Override public String toString() { return "Hello, World!"; } } Main.java import static java.lang.System.out;public class Main { public static void main(final String[] arguments) { out.println(new HelloWorld()); } } To demonstrate a primary difference between classpath and the extension mechanism (optional packages), I will archive the compiled HelloWorld.class file into a JAR called HelloWorld.jar and place it in a different directory than the compiled Main.class file. To demonstrate the use of the traditional classpath, I place the HelloWorld.jar file in a directory called C:\hello and will access that JAR via wildcard (*) for Main to use. This is demonstrated in the next two screen snapshots.The two previous images demonstrate that the Java Main application can still load the HelloWorld.class file even though I had deleted it from the current directory because the Java launcher was explicitly told (via the -classpath option) to look for it in C:\hello. Using the extensions mechanism (optional packages), it is possible to have the class loaded without it being in the same directory and without explicit classpath specification. This is shown in the next screen snapshot.The previous screen snapshot demonstrates that the Java launcher doesn’t even need the HelloWorld.class in the same directory or specified on its classpath when that class is inside a JAR that is in the extensions (optional packages) directory. This is often cited as a benefit of using the extensions mechanism because all applications using that JRE (or potentially all applications on the host) can see the same classes without need to explicitly specify them on the classpath. With the traditional classpath approach of instructing an application to load classes from JARs, the JAR file containing the .class file needs to end with the .jar extension. The next screen snapshot demonstrates what happens when the HelloWorld.jar is renamed HelloWorld.backup in the same classpath-referenced directory.The last image demonstrates that a NoClassDefFoundError is encountered when the JAR file in the classpath-referenced directory does not have a .jar extension. Perhaps a bit surprisingly, the extensions (optional packages) mechanism does not work the same way. Instead, all JAR files in the extensions specified directory are loaded regardless of their extension and regardless of even if they have a file extension. This is demonstrated in the next screen image.The previous image demonstrates that renaming the JAR file that resides within the extensions directory to not have any file extension whatsoever does not prevent the classloader from loading the classes of that JAR. In other words, the classloading mechanism loads all JAR files in the specified extensions directory based on file type rather than on file name or extension. As the Optional Packages Overview summarizes, “There is nothing special about any particular JAR file itself or the classes it contains that makes it an installed optional package. It is an installed optional package by virtue of its location in jre/lib/ext.” There are some risks and downsides associated with placing too many class definitions in JARs inside the extensions directory. It can be maddening to wonder why NoSuchMethodErrors, for example, are occurring when one can see that a class specified explicitly on the classpath has the method in question. I have previously written about one of the many potential causes of NoSuchMethodError, but forgotten outdated and obsolete class definitions residing inside of JAR files in the extensions directory are another potential cause. This is demonstrated next. The next two code listings show revised versions of Main.java and HelloWorld.java. In particular, HelloWorld has an all-new method that the new version of Main invokes. In this case, I’m going to leave the newly compiled HelloWorld.class file in the same directory when I run the Main to demonstrate that the old, busted version of HelloWorld.class in the JAR in the extensions directory takes precedence over the new hotness HelloWorld.class in the current directory. Revised Hello World.java (New Method) public class HelloWorld { @Override public String toString() { return "Hello, World!"; }public String directedHello(final String name) { return "Hello, " + name; } } Revised Main.java import static java.lang.System.out;public class Main { public static void main(final String[] arguments) { final HelloWorld helloWorld = new HelloWorld(); out.println(helloWorld); out.println(helloWorld.directedHello("Dustin")); } }The last image demonstrates that the now obsolete class definition of HelloWorld in the extensions directory takes precedence over the new class definition of HelloWorld in the same directory. Even when I specify the current directory on the classpath, the old version in the extensions directory takes precedence. This is shown in the next screen snapshot, which also shows that the JAR in the extensions directory that is “hiding” the newer JAR and its class’s newer method is still not even named with a .jar extension.The example just demonstrated is not even the most difficult situation a forgotten JAR in the specified extensions directory (or directories) can cause. In that example, at least I had a NoSuchMethodError to alert me to a problem. A potentially even more difficult situation to debug can exist when the old class definition has the same method signature but has an outdated method implementation. In such cases, there may be no error, exception, or throwable of any kind, but the application logic will simply not work correctly or as expected. The old functionality could exist in the code base for some time before it’s even recognized as an issue, especially if unit tests and other testing is lacking. Use of the extensions directory can make things easier on developers because classes in JAR files residing in the extensions directory (or directories) are available to all applications in the JRE associated with the extensions directory (or with all JREs on the host if the operating system-based host-wide extensions directory is used). However, there are definite risks associated with too liberal use of the directory. It can be easy to forget that outdated class definitions residing in JARs in that directory are preventing classloaders from loading the newer and seemingly obvious versions of the class definitions. When this happens, the very extensions (optional packages) mechanism that made developers’ lives easier now make it more difficult. Elliotte Rusty Harold provides a warning about use of the extensions (optional packages) mechanism, “While this seems convenient, it is also a long-term mistake… Sooner or later (probably sooner), you’ll load the wrong version of a class from a place you aren’t even thinking about and waste hours debugging.” The Java Tutorial also recommends caution (I added the emphasis), “Since this mechanism extends the platform’s core API, its use should be judiciously applied. Most commonly it is used for well standardized interfaces such as those defined by the Java Community Process, although it may also be appropriate for site wide interfaces.” Although the extensions (optional packages) mechanism is similar to the classpath mechanism and both are used as part of class loading, the differences between the two are important to note. In particular, it is important to remember that all JAR files (even if they don’t have .jar file extensions) that reside in the directory referenced as an extension directory will be loaded. Renaming these JARs and even changing their file extension will not be sufficient to have the classloading ignore them. With classpath, on the other hand, renaming the JAR is sufficient to prevent loading when the classpath specifies individual JAR files explicitly and changing the file extension is typically sufficient to prevent loading even when the classpath uses the wildcard (*) to specify all JARs in a directory. There are situations when the extensions (optional packages) mechanism is the appropriate choice, but these seem fairly rare. It is also important to keep in mind the extensions (optional packages) mechanism in mind when dealing with unexplained NoSuchMethodErrors so that one can check it out to see if the offender lives in that directory or directories.Reference: Java Extension Mechanism Loads All JARs from our JCG partner Dustin Marx at the Inspired by Actual Events blog....
docker-logo

Docker RabbitMQ cluster

I have been trying to create a Docker based RabbitMQ cluster on and off for sometime and got it working today – fairly basic and flaky but could be a good starting point for others to improve on. This is how the sample cluster looks on my machine, this is a typical cluster described in the RabbitMQ clustering guide available here -https://www.rabbitmq.com/clustering.html. As recommended at the site, there are 2 disk based nodes and 1 RAM based node here.        To quickly replicate this, you only need to have fig in your machine, just create a fig.yml file with the following entry: rabbit1: image: bijukunjummen/rabbitmq-server hostname: rabbit1 ports: - "5672:5672" - "15672:15672"rabbit2: image: bijukunjummen/rabbitmq-server hostname: rabbit2 links: - rabbit1 environment: - CLUSTERED=true - CLUSTER_WITH=rabbit1 - RAM_NODE=truerabbit3: image: bijukunjummen/rabbitmq-server hostname: rabbit3 links: - rabbit1 - rabbit2 environment: - CLUSTERED=true - CLUSTER_WITH=rabbit1 and in the folder holding this file, run: fig up That is it!, the entire cluster should come up. If you need more nodes, just modify the fig.yml file. The docker files for creating the dockerized rabbitmq-server is available at my github repo here: https://github.com/bijukunjummen/docker-rabbitmq-cluster and the “rabbitmq-server” image itself is here at the docker hub. ReferencesThe base rabbitmq image is somewhat based on cthulhuology’s docker rabbitmq image: https://github.com/cthulhuology/docker-rabbitmq Docker file: https://github.com/bijukunjummen/docker-rabbitmq-cluster Docker image: https://registry.hub.docker.com/u/bijukunjummen/rabbitmq-server/Reference: Docker RabbitMQ cluster from our JCG partner Biju Kunjummen at the all and sundry blog....
scala-logo

Akka Notes – Actor Lifecycle – Basic – 5

(Please note that this lifecycle write-up does not cover the preRestart or the postRestart methods. We’ll talk about them when we discuss supervision) The basic Actor lifecycle is very much intuitive. You could actually compare the basic Actor lifecycle with a Java servlet lifecycle with one special difference.            Just like any other regular class, we have a Constructor The preStart method gets called back next. Here, you could initialize resources that you would like to clean-up in postStop The “servicing” or the message handling by the receive method occupies the major chunk of time and that happens in between.Let’s look at a simple actor which prints the lifecycle. Dumb Lifecycle Actor package me.rerun.akkanotes.lifecycleimport akka.actor.{ActorLogging, Actor} import akka.event.LoggingReceiveclass BasicLifecycleLoggingActor extends Actor with ActorLogging{log.info ("Inside BasicLifecycleLoggingActor Constructor") log.info (context.self.toString()) override def preStart() ={ log.info("Inside the preStart method of BasicLifecycleLoggingActor") }def receive = LoggingReceive{ case "hello" => log.info ("hello") }override def postStop()={ log.info ("Inside postStop method of BasicLifecycleLoggingActor") }} App The LifecycleApp just initiates, sends a message to the Actor and shuts down the ActorSystem. import akka.actor.{ActorSystem, Props}object LifecycleApp extends App{val actorSystem=ActorSystem("LifecycleActorSystem") val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")lifecycleActor!"hello"//wait for a couple of seconds before shutdown Thread.sleep(2000) actorSystem.shutdown()} Output Inside BasicLifecycleLoggingActor ConstructorActor[akka://LifecycleActorSystem/user/lifecycleActor#-2018741361]Inside the preStart method of BasicLifecycleLoggingActorhelloInside postStop method of BasicLifecycleLoggingActor What’s that special difference between Servlets and the basic Actor lifecycle? That there is no difference between constructor and preStart in Actor lifecycle – more or less. The reason why I printed the context.self in the constructor is this – unlike Servlets, Actors have access to the ActorContext even inside the constructor. The difference between the preStart and the constructor then becomes very subtle. We’ll revisit the difference while we talk about supervision but if you are curious – calling the preStart when the Actor restarts (in case of failure) could be controlled. With constructor, that isn’t possible. When is postStop called? As we saw from the program, the postStop gets called when the ActorSystem shuts down. There are a couple of other times when the callback gets invoked too. 1. ActorSystem.stop() We could stop an Actor using the stop method of the ActorSystem and the ActorContext object LifecycleApp extends App{val actorSystem=ActorSystem("LifecycleActorSystem") val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")actorSystem.stop(lifecycleActor);... ...} 2. ActorContext.stop 1) Either by way of a message (externally or passing a message to itself) class BasicLifecycleLoggingActor extends Actor with ActorLogging{ ... ... def receive = LoggingReceive{ case "hello" => log.info ("hello") case "stop" => context.stop(self) } and object LifecycleApp extends App{val actorSystem=ActorSystem("LifecycleActorSystem") val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")lifecycleActor!"stop" ... ... 2) OR kill itself for no reason (this is just for fun. No Actor with an ambition would do this) class BasicLifecycleLoggingActor extends Actor with ActorLogging{log.info ("Inside BasicLifecycleLoggingActor Constructor") log.info (context.self.toString()) context.stop(self) ... ... 3. PoisonPill In the previous example, we passed a message called stop from the LifecycleApp to the Actor. The Actor received that message and killed itself using a context.stop. We could achieve the same thing by passing a PoisonPill message to the target actor. Please note that the PoisonPill message, just like the previous stop message gets enqueued in the regular mailbox and will be processed when it turn comes up. object LifecycleApp extends App{val actorSystem=ActorSystem("LifecycleActorSystem") val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")lifecycleActor!PoisonPill ... ... TRIVIA What do I mean regular mailbox? Is there a “special” mailbox too? Yup. There is. And we’ll talk about it when we talk about it when we discuss supervision and system messages. Termination Once the Actor is stopped, it is said to enter into a Terminated state. The immediate question that would come up to your mind is what would happen to the messages that is sent to an Actor which is already terminated? Let’s see that : App object LifecycleApp extends App{val actorSystem=ActorSystem("LifecycleActorSystem") val lifecycleActor=actorSystem.actorOf(Props[BasicLifecycleLoggingActor],"lifecycleActor")lifecycleActor!"hello" lifecycleActor!"stop" lifecycleActor!"hello" //Sending message to an Actor which is already stopped} Actor – Just as before class BasicLifecycleLoggingActor extends Actor with ActorLogging{def receive = LoggingReceive{ case "hello" => log.info ("hello") case "stop" => context.stop(self)} } Output BasicLifecycleLoggingActor - helloakka.actor.RepointableActorRef - Message from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-569230546] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. From the logs, you see that there are some references of deadletters. Any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor.CURIOUS AS TO WHAT HAPPENS NEXT? The DeadLetter Actor processes the messages in its mailbox, wraps each message as a DeadLetter and publishes it to theEventStream. One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Check this out. Remember, when we talked about logging, we saw that all log messages gets published to the EventStream and that we are free to subscribe to that EventStream – just that the subscriber also needs to be an Actor. Let’s try that now. For our example, we’ll subscribe to the EventStream and watch out for all DeadLetter messages and will print to the console (so much for creativity??!!). Frankly, we are free to do anything from generating an alert, storing it into a database or even feeding into analytics. SUBSCRIBING TO DEADLETTERS IN EVENTSTREAM import akka.actor.ActorSystem import akka.actor.Props import akka.actor.PoisonPill import akka.actor.DeadLetter import akka.actor.Actorobject LifecycleApp extends App {val actorSystem = ActorSystem("LifecycleActorSystem") val lifecycleActor = actorSystem.actorOf(Props[BasicLifecycleLoggingActor], "lifecycleActor")val deadLetterListener = actorSystem.actorOf(Props[MyCustomDeadLetterListener]) actorSystem.eventStream.subscribe(deadLetterListener, classOf[DeadLetter])lifecycleActor ! "hello" lifecycleActor ! "stop" lifecycleActor ! "hello"}class MyCustomDeadLetterListener extends Actor { def receive = { case deadLetter: DeadLetter => println(s"FROM CUSTOM LISTENER $deadLetter") } } Output 164 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO BasicLifecycleLoggingActor - hello167 [LifecycleActorSystem-akka.actor.default-dispatcher-4] INFO akka.actor.RepointableActorRef - Message from Actor[akka://LifecycleActorSystem/deadLetters] to Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.FROM CUSTOM LISTENER DeadLetter(hello,Actor[akka://LifecycleActorSystem/deadLetters],Actor[akka://LifecycleActorSystem/user/lifecycleActor#-782937925])Reference: Akka Notes – Actor Lifecycle – Basic – 5 from our JCG partner Arun Manivannan at the Rerun.me blog....
agile-logo

Agile tester part 2, questions and answers

Warning: The opinions expressed in this post are mine only, please do not use them against any other group of people, but only against me, that is Augusto Evangelisti a.k.a. Gus. After writing my most recent blog post “The Agile Tester, a Curious and Empathetic Animal” I received quite a lot of feedback for which I am very grateful. Feedback in the form of a conversation is the true fuel for learning and improvement. To all of you that have mailed me, tweeted about my post, responded using the comments in my blog and talked to me face to face about it, thank you so much for helping me learn! When i looked at the feedback I received, I saw 2 quite different trends. Looking closer I saw that the different feedback types came from different schools of thought. Let me give you 2 examples for some context. This one from twitter: Well there are too many great quotes in @augeva's post to tweet 'em all, just go read it! :-> — lisacrispin (@lisacrispin) October 13, 2014This one from one comment to my blog post:Michael Bolton: Something about this post troubles me, Gus. In order to test what was troubling me, I took the content and replaced each instance of “tester” with “programmer”. The result made perfect sense, of course, but it left me wondering: if I can replace “tester” with “programmer”, what distinguishes the tester from anyone else on the project? What is the special role, the particular set of activities, that the tester performs? Is there a difference between the programmer’s mindset and the tester’s mindset? What is the mission, the distinctive reason for having a tester on the team—whether that person has the title “tester” or something else? What is it that distinguishes testing work from all the other work? What testing-specific skills do testers bring to the table? What is testing? I have answers of my own, of course. But I’m wondering what your answers might be.I am using Lisa’s and Michael’s as examples of the feedback received because the former is a recognisable exponent of the agile testing community and the latter is a recognisable exponent of the context driven testing community. The other feedback that i received from people close to each of the 2 communities is extremely similar to theirs. If a group of people finds troubling what the other group finds great, I smell something interesting and an opportunity for learning. I’ll try answering Michael questions and look forward to his very own answers to learn something: Question#1:  Something about this post troubles me, Gus. In order to test what was troubling me, I took the content and replaced each instance of “tester” with “programmer”. The result made perfect sense, of course, but it left me wondering: if I can replace “tester” with “programmer”, what distinguishes the tester from anyone else on the project? What is the special role, the particular set of activities, that the tester performs? Michael, the distinction between different roles in agile teams is becoming more and more blurry. Agile teams value competencies more than roles. As an agile tester in my team I have core competencies that i use to support my team. These include but are not limited  to the ability to evaluate a product by learning about it through experimentation. I also use such competencies to coach and lead other members of my team that are not that strong in that area and help them grow towards an ideal form of generalising specialist. Finally I also perform tasks outside my main competency to support the team. In some cases I will need guidance from another member of the team whose core competency includes the ability of performing such task. I believe that this blurring of the roles increases agile team members accountability, in fact nobody in an agile team (developer, tester, business analyst, operations specialist, UX expert, et cetera) should ever say, “I’m X, doing Y is not my job” but they should instead ask their colleagues “how can I help you?”. Shared activities are key to learning and gaining competencies. The main goal for any agile team member, regardless of his role is delivering customer value. Question#2: Is there a difference between the programmer’s mindset and the tester’s mindset? Yes, there is a difference, I don’t think it constitutes an insurmountable obstacle. One of the teams I have worked with, knowing that I wouldn’t be available for a period of time, suggested they needed the tester’s mindset for certain activities and decided that to make sure they were focusing on the right things, they would wear a big red hat. This simple change helped them keep their focus and their mindset in the right place. It might not be prefect but it worked. It works also because I speak to them about the testers mindset when we work together, I give them examples of what I am thinking at specific times and why it is important to think about such things. Developers are very smart people, once they understand a practice has value and receive sufficient amount of coaching, they can learn to do almost anything. Just to avoid misunderstandings, our developers wear the testers hat when performing testing activities with other developers, i.e. they don’t use the hat for their own code as maybe this could be asking too much.  Question#3: What is the mission, the distinctive reason for having a tester on the team—whether that person has the title “tester” or something else? The mission is to provide the team with the testing competencies it needs so that customer value can be delivered. The secondary purpose is to train and coach the team so that they can gain some of the competencies to support test activities. Coaching and training can be also formal but it is mainly delivered by working in pairs.  Question#4: What is it that distinguishes testing work from all the other work? Every activity the team engages in, is performed to deliver customer value, including test activities, but I am not sure I understood your question completely, could you please rephrase? Question#5: What testing-specific skills do testers bring to the table? See answer to question#1 re. competencies. My main competencies are exploration and learning. Question#6: What is testing? Testing to me is exploration and learning. I test ideas to prevent defects, software to detect them and product owners assumptions to reduce waste. I hope I have clarified some of your doubts and I am curious to hear your answers to your own questions.Reference: Agile tester part 2, questions and answers from our JCG partner Augusto Evangelisti at the mysoftwarequality blog....
java-logo

Supercharged jstack: How to Debug Your Servers at 100mph

A guide for using jstack to debug live Java production servers jstack is like U2 – it’s been with us since the dawn of time, and we can’t seem to be rid of it. Jokes aside, jstack is by far one of the handiest tools in your arsenal to debug a live production server. Even so, I still feel it’s deeply underused in terms of its ability to pull you out of the fire when things go bad, so I wanted to share a few ways in which you can supercharge into an even stronger weapon in your war against production bugs. At its core jstack is a super easy tool to show you the stack traces of all Java threads running within a target JVM. Just point it to a JVM process via a pid and get a printout of all the thread stack traces at that point in time. This enables you to answer the age old question of “what is this server doing?”, and bringing you one step closer to understand why it’s actually doing it. The biggest pro about jstack is that it’s lightweight – it doesn’t add any performance overhead to the JVM or changes its execution state (unlike a debugger or profiler). Since nothing is perfect, jstack has two significant disadvantages. The first is that jstack doesn’t provide you with any variable state other than a call stack, which means that while you might be looking at a stack, you will have no idea as to what is the state that got it there. A good example would be looking at a JVM that’s hanging, where jstack would show you that a large number of threads are executing DB queries or waiting to obtain a connection. This probably means that some queries are taking too long to execute, causing other threads to either wait for a connection or be denied one. This is one spot where you’d really like to know which query is being executed (or what are its parameters) that’s causing the slowdown, and when it began. This is of course just one example, out of a plethora of scenarios in which some threads are blocked and are reducing the throughput of your application. But unfortunately with jstack, as you’re not getting any variable state – you can’t really tell which thread is to blame. Or Can you? The second disadvantage to jstack is that it’s not an always-on tool. This means that you have to be there when the problem occurs – which in production can be a rare event. This is even more true in elastic environments where VMs are constantly restarted. Here comes the good part – let’s take a look at two techniques that can help us overcome these two shortcomings, and make a good tool really great. Creating stateful thread data The first question is how can you add state to your jstack print out? The answer is simple and powerful – thread names. While many mistakenly consider a thread name to be an immutable, or an OS determined property, it’s in fact a mutable and incredibly important trait that each thread has. It’s also the one that gets picked up into your jstack stream, and therein lies the key. The practical application is that much like logging you should control the thread name once it enters your code through an entry point such as servlet, actor or scheduler. At that point you’ll want to set its name into a meaningful value that can help you understand the execution context and relevant parameters that can help you isolate the transaction and its contents. This would most likely include –The purpose of the thread (e.g. processing a message, responding to user request, etc..). The transaction ID which would enable you to identify this specific data flow across different machines and parts of the application. Parameter values such as servlet parameters or the ID of a message being dequeued. The time in which you obtained control of the thread. This last item is critically important for you to know exactly which threads in your code are stuck when you use jstack to observe them.Thread.currentThread().setName(Context + TID + Params + current Time,..); This data will mean the difference between looking at a printout such as the one below which doesn’t actually tell us anything about what a thread is doing or why and one which is informative: “pool-1-thread-1″ #17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait() [0x000000013ebcc000] Compare this with the following thread printout: ”Queue Processing Thread, MessageID: AB5CAD, type: AnalyzeGraph,queue: ACTIVE_PROD, Transaction_ID: 5678956, Start Time: 10/8/2014 18:34″ #17 prio=5 os_prio=31 tid=0x00007f9d620c9800 nid=0x6d03 in Object.wait() [0x000000013ebcc000] What you’re seeing here is a much fuller explanation of what this thread is actually doing. You can easily see its dequeuing messages from an AWS queue, which message it’s analyzing, its type, ID and the transaction ID. And last, but far from least – when did the thread start working on it. This can help you focus very quickly on those threads that are stuck, and see the state which they’re in. From there on out, optimizing and reproducing locally becomes a much easier job. The alternative here would be to either hope that there’s data in the log files, and be able to correlate data in the logs to this exact thread. Another option would be to attach a debugger in production either locally or remotely. Both not very pleasant and time consuming. Writing this information in the thread name also helps with traditional logging. Even though most logging frameworks provide thread-based context that can be added to the log, you have to make sure you configure it correctly. Using thread name can also ensure you will have all the data you need in the log. Note: Some folks may say that thread names are not to be tempered with or changed. I’m a very small believer in this, from both my personal experience in doing so for years and that of many colleagues. Making jstack always-on The second challenge we’re faced with when using jstack is that just like a debugger, it’s a tool that you have to manually operate at the moment where the problem is happening to capture the corrupt state. However, there’s a more active way of using jstack to automatically generate printouts when a server hangs or falls below or above a certain threshold. The key is to invoke jstack programmatically just as you would any logging feature from within the JVM whenever specific application conditions are met. The two key challenges here are when and how do you do it. How to activate jstack programmatically? As jstack is a plain OS process, invoking it is fairly straightforward. All you have to do is activate the jstack process and point it at yourself. The kicker here is how to get the pid for your process from within the JVM. There’s actually no standard Java API to do it (At least not until Java 9). Here’s a little snippet that gets the job done (albeit not part of a documented api): String mxName = ManagementFactory.getRuntimeMXBean().getName();int index = mxName.indexOf(PID_SEPERATOR);String result;if (index != -1) { result = mxName.substring(0, index); } else { throw new IllegalStateException("Could not acquire pid using " + mxName); } Another minor challenge is directing jstack output into your log. That’s also fairly easy to setup using output stream gobblers. Look here for an example as to how to direct output data printed out by a process which you invoke into your log file or output stream. While it’s possible to capture the stack trace of running threads internally using getAllStackTraces, I prefer to do it by running jstack for a number of reasons. The first is that this is something that I would usually want to happen externally to the running application (even if the JVM is participating in providing the information) to make sure I don’t impact the stability of the application by making introspective calls. Another reason is that jstack is more powerful in terms of its capabilities such as showing you native frames and lock state, something that’s not available from within the JVM. When do you activate jstack? The second decision you need to make is what are the conditions under which you’ll want the JVM to log a jstack. This would probably be done after a warm-up period, when the server falls below or above a specific processing (i.e. request or message processing) threshold. You may also want to make sure you take enough time between each activation; just to make sure you don’t flood your logs under low or high load. The pattern you would use here is load up a watchdog thread from within the JVM which can periodically look at the throughput state of the application (e.g the number of messages processed in the last two minutes) and decide whether or not a “screenshot” of thread state would be helpful in which case it would activate jstack and log it to file. Set the name of this thread to contain the target and actual throughput state, so when you do take an automatic jstack snapshot you can see exactly why the watchdog thread decided to do so. As this would only happen every few minutes, there’s no real performance overhead to the process – especially compared to the quality of data provided. Below is snippet showing this pattern in action. The startScheduleTask loads up a watchdog thread to periodically check a throughput value which is incremented using a Java 8 concurrent adder whenever a message is being processed. public void startScheduleTask() {scheduler.scheduleAtFixedRate(new Runnable() { public void run() {checkThroughput();} }, APP_WARMUP, POLLING_CYCLE, TimeUnit.SECONDS); }private void checkThroughput() { int throughput = adder.intValue(); //the adder in inc’d when a message is processedif (throughput < MIN_THROUGHPUT) { Thread.currentThread().setName("Throughput jstack thread: " + throughput); System.err.println("Minimal throughput failed: exexuting jstack"); executeJstack(); //see the code on github to see how this is done }adder.reset(); }The full source code for preemptively invoking jstack from within your code can be found here.Reference: Supercharged jstack: How to Debug Your Servers at 100mph from our JCG partner Tal Weiss at the Takipi blog....
agile-logo

Is Your Culture Working the Way You Think it Is?

Long ago, I was a project manager and senior engineer for a company undergoing a Change Transformation. You know the kind, where the culture changes, along with the process. The senior managers had bought into the changes. The middle managers were muddling through, implementing the changes as best they could. Us project managers and the technical staff, we were the ones doing the bulk of the changes. The changes weren’t as significant as an agile transformation, but they were big. One day, the Big Bosses, the CEO and the VP Engineering spoke at an all-hands meeting. “You are empowered,” they said. No, they didn’t say it as a duet. They each said it separately. They had choreographed speeches, with great slide shows, eight by ten color glossies, and pictures. They had a vision. They just knew what the future would hold. I managed to keep my big mouth shut. The company was not doing well. We had too many managers for not enough engineers or contracts. If you could count, you could see that. I was traveling back and forth to a client in the midwest. At one point, the company owed me four weeks of travel expenses. I quietly explained that no, I was not going to book any more airline travel or hotel nights until I was paid in full for my previous travel. “I’m empowered. I can refuse to get on a plane.” That did not go over well with anyone except my boss, who was in hysterics. He thought it was quite funny. My boss agreed I should be reimbursed before I racked up more charges. Somehow, they did manage to reimburse me. I explained that from now on, I was not going to float the company more than a week’s worth of expenses. If they wanted me to travel, I expected to be reimbursed within a week of travel. I got my expenses in the following Monday. They could reimburse me four days later, on Friday. “But that’s too fast for us,” explained one of the people in Accounting. “Then I don’t have to travel every other week,” I explained. “You see, I’m empowered. I’ll travel after I get the money for the previous trip. I won’t make a new reservation until I receive all the money I spent for all my previous trips. It’s fine with me. You’ll just have to decide how important this project is. It’s okay.” The VP came to me and tried to talk me out of it. I didn’t budge. (Imagine that!) I told him that I didn’t need to float the company money. I was empowered. “Do you like that word?” “Sure I do.” “Do you feel empowered?” “Not at all. I have no power at all, except over my actions. I have plenty of power over what I choose to do. I am exercising that power. I realized that during your dog and pony show. “You’re not changing our culture. You’re making it more difficult for me to do my job. That’s fine. I’m explaining how I will work.” The company didn’t get a contract it had expected. It had a layoff. Guess who got laid off? Yes, I did. It was a good thing. I got a better job for more money. And, I didn’t have to travel every other week. Change can be great for an organization. But telling people the culture is one thing and then living up to that thing can be difficult. That’s why this month’s management myth is Myth 34: You’re Empowered Because I Say You Are. I picked on empowerment. I could have chosen “open door.” Or “employees are our greatest asset.” (Just read that sentence. Asset???) How you talk about culture says a lot about what the culture is. Remember, culture is how you treat people, what you reward, and what is okay to talk about. Go read Myth 34: You’re Empowered Because I Say You Are.Reference: Is Your Culture Working the Way You Think it Is? from our JCG partner Johanna Rothman at the Managing Product Development blog....
software-development-2-logo

4 Biggest Reasons Why Software Developers Suck at Estimation

Estimation is difficult. Most people aren’t good at it–even in mundane situations. For example, when my wife asks me how much longer it will take me to fix some issue I’m working on or to head home, I almost always invariably reply “five minutes.” I almost always honestly believe it will only take five minutes, but it never does. Most of the time my five minutes ends up being half an hour or more. But, in comparison to the world of software development efforts, my five minutes estimation is usually fairly accurate–it’s only off by a factor of six or so. It’s not unheard of to have software development estimations be off by as much as one-hundred-fold. I’ve literally had an hour long estimation turn into two weeks. But, why are software development estimations usually off by so much? Here are the four biggest reasons I have found: Reason 1: The unknown unknowns This phrase was first uttered by former Secretary of Defense of the United States, Donald Rumsfeld. It basically refers to those things that we don’t even know that we don’t know.By far, this is the biggest reason why software developers often flub at giving good estimations. It also happens to be the primary reason why I suck at telling my wife how long it will take me to get home–I don’t know about the distractions that I don’t know about yet. Software development has a lot of unknowns. Some of the unknowns we know about. When we first start a project, we might have a good idea that we need to store the data in a database, but we don’t know how we’ll do it. That is a known unknown. We know that we don’t know something that we’ll eventually need to know. Estimating a known unknown is pretty difficult, but we can do a decent job of it if we can compare it to something similar we’ve already done before. I don’t know how long it will take me to write this particular blog post, but I know about how long it has taken me to write other blog posts of a similar length. What is really scary though, are the things that we don’t even know we don’t know yet.   These unknown unknowns sneak up on us, because we don’t even know they exist–by definition they are unknowable. It’s one thing to know that we have a gap in a bridge somewhere that we have to cross, it is a whole other thing to have to cross a bridge blindfolded and only find out about gaps when you fall through them. Constantly, in software development, we are faced with situations where we have to encounter these unknown unknowns. There is no good way to estimate around them. The best we can do in these cases is give ourselves a lot of padding and a lot of rope so that we can climb out of any gaps in the bridge that we fall into. Reason 2: Lengthy time periods As if unknown unknowns weren’t bad enough, the deck is stacked against us even further. Most software development estimations involve fairly long periods of time. This has changed somewhat with Agile development practices, but we still are often asked to estimate a week or more worth of work at a time. (Plus, let’s not forget those sneaky project managers who try to throw Agile projects into Microsoft Project anyway and say “yeah, I know this is Agile and all, but I’m just trying to get a rough idea of when we’ll be done with all of the features.”) It’s fairly easy to estimate short periods of time–well, I guess unless it’s me telling my wife how long it will take for me to get off the computer. Most of us can pretty accurately predict how long it will take us to brush our teeth, write an email or eat dinner. Long periods of time are much more difficult to estimate accurately. Estimating how long it will take you to clean out the garage, write a book, or even just to go grocery shopping is much more challenging.  The longer the period of time you are trying to estimate, the more that small miscalculations and the effects of known unknowns can cause an initial estimate to be grossly off target. In my experience, I’ve found that estimating anything that will take more than about two hours is where things really start to go off of the rails. As a mental exercise, try and estimate things of varying lengths. How long will it take you to:do 10 pushups? make a cup of coffee? go to the convenience store and buy something? write a one page letter? read a 300 page novel? get the oil changed in your car?Notice how the things that can be done in under half an hour are very easy to estimate with a high level of confidence, but as you go out further in time it gets much more difficult. Most of the time when we do software development estimates, we don’t try and estimate short things, like how long it will take to write a single unit test, instead we tend to estimate longer things, like how long it will take to complete a feature. Reason 3: Overconfidence I’m pretty presumptuous when it comes to estimations. I usually think I’m very accurate at making estimations. My wife would disagree–at least when it comes to making estimations of time things will take me. History would probably tend to vindicate her viewpoint. As software developers, we can often become pretty convinced of our ability to accurately predict how long something will take. Often, if the programming task we are about to embark upon is one we feel confident about, we can be pretty aggressive with our estimates–sometimes to the point of absurdity.  How long will it take you to get that feature done? Oh, that? That’s easy. I can get that done in a few hours. I’ll have it done by tomorrow morning. Are you sure? What about testing? What if something comes up? Don’t worry, it’s easy. Shouldn’t be a problem at all. I just need to throw a few buttons on a page and hook up the backend code. But, what happens when you actually sit down and try to implement the feature? Well, first you find out that it wasn’t quite as easy as you thought. You forgot to consider a few of the edge cases that have to be handled. Pretty soon you find yourself taking the entire night to just get set up in order to actually start working on the problem. Hours turn into days, days into weeks and a month later, you’ve finally got some shippable code. Now, this might be a bit of an exaggeration, but overconfidence can be a big problem in software development–especially when it comes to estimations. Reason 4: Under-confidence Under-confidence isn’t actually a word. I suppose that is because someone wasn’t confident enough to put it in the dictionary. But, just as overconfidence can cause a software developer to under-estimate how long a programming task will take, under-confidence can cause that same software developer to overestimate a completely different task, which may even be much easier. I don’t know about you, but I often find myself in situations where I am very unsure of how long something will take. I can turn a simple task that I don’t feel comfortable doing into a huge mountain that seems almost impassible. We tend to view things that we’ve never done before as harder than they are and things that we have done before as easier than they are–it’s just human nature. Although it may not seem like it, under-confidence can be just as deadly to estimations though. When we are under-confident, we are more likely to add large amounts of padding to our estimates. This padding might not seem all that bad, but work has a way of filling the time allotted for it. (This is known as Parkinson’s law.) So, even though, when we are under-confident, it might appear that we are pretty accurate with our estimations, the truth is we may be wasting time by having work that might have been done in half the time fill the entire time that was allotted for it. (By the way, if you are looking for a good book on Agile estimation, check out Mike Cohn’s book: Agile Estimating and Planning.) What else? Did I leave anything out? What do you think is the biggest reason why software development estimations are so difficult?Reference: 4 Biggest Reasons Why Software Developers Suck at Estimation from our JCG partner John Sonmez at the Making the Complex Simple blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below:
Close