Featured FREE Whitepapers

What's New Here?


10 things you can do to make your app secure: #3 Validate Input

This is part #3 of a series of posts on the OWASP Top 10 Proactive Development Controls. Your first line of defence against attacks should always be to check all data from untrusted sources. Input validation is fundamental to application security, and a basic part of good defensive programming. This is simple, and obvious – and often done wrong.       Don’t Rely on Client-Side Validation One common mistake is relying on client-side validation to catch problems. Client-side validation is useful in providing immediate feedback at the UI. But it won’t protect your system, especially in web apps. If an attacker can find a way to insert themselves in between the browser and your app, which can be done using a proxy, they can play with any of the data, including header fields and other hidden data, after local client-side editing has already been done. Data from a client, especially a client outside of your network, should never be trusted. Another common mistake is relying on negative checking or “black list” validation to try to catch known bad data. This is a weak technique: you can’t be sure that you will catch all of the bad things or combinations, and negative checks can be subverted through (double) encoding and other evasion tricks. White List Validation OWASP recommends that you always use positive restrictions – white list validation – where you define the acceptable type and size, and all allowable values using regular expressions (regex). An example from the Proactive Controls: the regex for a password field could be:^(?=.*[a-z])(?=.*[A-Z]) (?=.*\d) (?=.*[@#$%]).{10,64}$This regular expression ensures that a password is 10 to 64 characters in length and includes a uppercase letter, a lowercase letter, a number and a special character (one or more uses of @, #, $, or %). White list validation is trivial for enumerated data: days of the week, months of the year … anything that you can fit into a defined list. For other common data types (such as dates and time zones, numeric types, currencies, email addresses, IP addresses, URLs and domain names, and credit card numbers), you could use the Apache Commons Validator library. Validation of free-format text fields (names, comments) is more difficult, and especially text fields that can – or need to – contain special characters, especially markup:If your application handles markup — untrusted input that is supposed to contain HTML — it can be very difficult to validate. Encoding is also difficult, since it would break all the tags that are supposed to be in the input. Therefore, you need a library that can parse and clean HTML formatted text such as the OWASP Java HTML Sanitizer. A regular expression is not the right tool to parse and sanitize untrusted HTML. You can test how solid your input validation is with different tools, such as fuzzers or static analysis taint checking (using tools that run through execution paths in the code and identify when you are referencing data that has not been validated) and through pen testing and manual exploratory testing.Reference: 10 things you can do to make your app secure: #3 Validate Input from our JCG partner Jim Bird at the Building Real Software blog....

Stop Trying to Emulate SQL OFFSET Pagination with Your In-House DB Framework!

I’m pretty sure you’ve gotten it wrong in numerous ways, so far. And you probably won’t get it right any time soon. So why waste your precious time on SQL tweaking, when you could be implementing business logic? Let me explain… It hasn’t been until the recent SQL:2008 standard that what MySQL users know as LIMIT .. OFFSET was standardised into the following simple statement:       SELECT * FROM BOOK OFFSET 2 ROWS FETCH NEXT 1 ROWS ONLY Yes. So many keywords.SQL is indeed a very verbose language. Personally, we really like the conciseness of MySQL’s / PostgreSQL’s LIMIT .. OFFSET clause, which is why we chose that for the jOOQ DSL API. In SQL: SELECT * FROM BOOK LIMIT 1 OFFSET 2 In jOOQ: select().from(BOOK).limit(1).offset(2); Now, when you’re a SQL framework vendor, or when you’re rolling your own, in-house SQL abstraction, you might think about standardising this neat little clause. Here’s a couple of flavours from databases that natively support offset pagination: -- MySQL, H2, HSQLDB, Postgres, and SQLite SELECT * FROM BOOK LIMIT 1 OFFSET 2-- CUBRID supports a MySQL variant of the -- LIMIT .. OFFSET clause SELECT * FROM BOOK LIMIT 2, 1-- Derby, SQL Server 2012, Oracle 12, SQL:2008 SELECT * FROM BOOK OFFSET 2 ROWS FETCH NEXT 1 ROWS ONLY-- Ingres. Eek, almost the standard. Almost! SELECT * FROM BOOK OFFSET 2 FETCH FIRST 1 ROWS ONLY-- Firebird SELECT * FROM BOOK ROWS 2 TO 3-- Sybase SQL Anywhere SELECT TOP 1 ROWS START AT 3 * FROM BOOK-- DB2 (without OFFSET) SELECT * FROM BOOK FETCH FIRST 1 ROWS ONLY-- Sybase ASE, SQL Server 2008 (without OFFSET) SELECT TOP 1 * FROM BOOK So far, so good. These can all be handled. Some databases put offsets before limits, others put limits before offsets, and the T-SQL family puts the whole TOP clause before the SELECT list. This is easy to emulate. Now what about:Oracle 11g and less SQL Server 2008 and less DB2 with OFFSET(note that you can enable various alternative syntaxes in DB2) When you google for this, you will find millions of ways to emulate OFFSET .. FETCH in those older databases. The optimal solutions always involve:Using doubly-nested derived tables with ROWNUM filtering in Oracle Using single-nested derived tabels with ROW_NUMBER() filtering in SQL Server and DB2So you’re emulating it.Do you think you will get it right? Let us go through a couple of issues that you may not have thought about. First off, Oracle. Oracle obviously wanted to create a maximum vendor-lockin, which is only exceeded by Apple’s recent introduction of Swift. This is why ROWNUM solutions perform best, even better than SQL:2003 standard window function based solutions. Don’t believe it? Read this very interesting article on Oracle offset pagination performance. So, the optimal solution in Oracle is: -- PostgreSQL syntax: SELECT ID, TITLE FROM BOOK LIMIT 1 OFFSET 2-- Oracle equivalent: SELECT * FROM ( SELECT b.*, ROWNUM rn FROM ( SELECT ID, TITLE FROM BOOK ) b WHERE ROWNUM <= 3 -- (1 + 2) ) WHERE rn > 2 So that’s really the equivalent? Of course not. You’re selecting an additional column, the rn column. You might just not care in most cases, but what if you wanted to make a limited subquery to be used with an IN predicate? -- PostgreSQL syntax: SELECT * FROM BOOK WHERE AUTHOR_ID IN ( SELECT ID FROM AUTHOR LIMIT 1 OFFSET 2 )-- Oracle equivalent: SELECT * FROM BOOK WHERE AUTHOR_ID IN ( SELECT * -- Ouch. These are two columns! FROM ( SELECT b.*, ROWNUM rn FROM ( SELECT ID FROM AUTHOR ) b WHERE ROWNUM <= 3 ) WHERE rn > 2 ) So, as you can see, you’ll have to do some more sophisticated SQL transformation. If you’re manually emulating LIMIT .. OFFSET, then you might just patch the ID column into the subquery: SELECT * FROM BOOK WHERE AUTHOR_ID IN ( SELECT ID -- better FROM ( SELECT b.ID, ROWNUM rn -- better FROM ( SELECT ID FROM AUTHOR ) b WHERE ROWNUM <= 3 ) WHERE rn > 2 ) So, that’s more like it, right? But since you’re not writing this manually every time, you’re about to start creating your own nifty in-house SQL framework covering the 2-3 use cases that you’ve encountered so far, right? You can do it. So you’ll regex-search-replace column names automagically to produce the above. So now, it is correct? Of course not! Because you can have ambiguous column names in top-level SELECTs, but not in nested selects. What if you want to do this: -- PostgreSQL syntax: -- Perfectly valid repetition of two ID columns SELECT BOOK.ID, AUTHOR.ID FROM BOOK JOIN AUTHOR ON BOOK.AUTHOR_ID = AUTHOR.ID LIMIT 1 OFFSET 2-- Oracle equivalent: SELECT * FROM ( SELECT b.*, ROWNUM rn FROM ( -- Ouch! ORA-00918: column ambiguously defined SELECT BOOK.ID, AUTHOR.ID FROM BOOK JOIN AUTHOR ON BOOK.AUTHOR_ID = AUTHOR.ID ) b WHERE ROWNUM <= 3 ) WHERE rn > 2 Nope. And the trick of manually patching ID columns from the previous example doesn’t work, because you have multiple ID instances. And renaming the columns to random values is nasty, because the user of your home-grown in-house database framework wants to receive well-defined column names. I.e. ID and… ID. So, the solution is to rename the columns twice. Once in each derived table: -- Oracle equivalent: -- Rename synthetic column names back to original SELECT c1 ID, c2 ID FROM ( SELECT b.c1, b.c2, ROWNUM rn FROM ( -- synthetic column names here SELECT BOOK.ID c1, AUTHOR.ID c2 FROM BOOK JOIN AUTHOR ON BOOK.AUTHOR_ID = AUTHOR.ID ) b WHERE ROWNUM <= 3 ) WHERE rn > 2 But now, we’re done? Of course not! What if you doubly nest such a query? Will you think about doubly renaming ID columns to synthetic names, and back? … Let’s leave it here and talk about something entirely different: Does the same thing work for SQL Server 2008? Of course not! In SQL Server 2008, the most popular approach is to use window functions. Namely, ROW_NUMBER(). So, let’s consider: -- PostgreSQL syntax: SELECT ID, TITLE FROM BOOK LIMIT 1 OFFSET 2-- SQL Server equivalent: SELECT b.* FROM ( SELECT ID, TITLE, ROW_NUMBER() OVER (ORDER BY ID) rn FROM BOOK ) b WHERE rn > 2 AND rn <= 3 So that’s it, right? Of course not! OK, we’ve already had this issue. We should not select *, because that would generate too many columns in the case that we’re using this as a subquery for an IN predicate. So let’s consider the correct solution with synthetic column names: -- SQL Server equivalent: SELECT b.c1 ID, b.c2 TITLE FROM ( SELECT ID c1, TITLE c2, ROW_NUMBER() OVER (ORDER BY ID) rn FROM BOOK ) b WHERE rn > 2 AND rn <= 3 But now we got it, right? Make an educated guess: Nope!What happens, if you add an ORDER BY clause to the original query? -- PostgreSQL syntax: SELECT ID, TITLE FROM BOOK ORDER BY SOME_COLUMN LIMIT 1 OFFSET 2-- Naive SQL Server equivalent: SELECT b.c1 ID, b.c2 TITLE FROM ( SELECT ID c1, TITLE c2, ROW_NUMBER() OVER (ORDER BY ID) rn FROM BOOK ORDER BY SOME_COLUMN ) b WHERE rn > 2 AND rn <= 3 Now, that doesn’t work in SQL Server. Subqueries are not allowed to have an ORDER BY clause, unless they also have a TOP clause (or an OFFSET .. FETCH clause in SQL Server 2012). OK, we can probably tweak this using TOP 100 PERCENT to make SQL Server happy. -- Better SQL Server equivalent: SELECT b.c1 ID, b.c2 TITLE FROM ( SELECT TOP 100 PERCENT ID c1, TITLE c2, ROW_NUMBER() OVER (ORDER BY ID) rn FROM BOOK ORDER BY SOME_COLUMN ) b WHERE rn > 2 AND rn <= 3 Now, that’s correct SQL according to SQL Server, although you do not have a guarantee that the ordering of the derived table will survive after query execution. It may well be that the ordering is changed again by some influence. If you wanted to order by SOME_COLUMN in the outer query, you’d have to again transform the SQL statement to add another synthetic column: -- Better SQL Server equivalent: SELECT b.c1 ID, b.c2 TITLE FROM ( SELECT TOP 100 PERCENT ID c1, TITLE c2, SOME_COLUMN c99, ROW_NUMBER() OVER (ORDER BY ID) rn FROM BOOK ) b WHERE rn > 2 AND rn <= 3 ORDER BY b.c99 That does start getting a bit nasty. And let’s guess whether: This is the correct solution! Of course not! What if the original query had DISTINCT in it? -- PostgreSQL syntax: SELECT DISTINCT AUTHOR_ID FROM BOOK LIMIT 1 OFFSET 2-- Naive SQL Server equivalent: SELECT b.c1 AUTHOR_ID FROM ( SELECT DISTINCT AUTHOR_ID c1, ROW_NUMBER() OVER (ORDER BY AUTHOR_ID) rn FROM BOOK ) b WHERE rn > 2 AND rn <= 3 Now, what happens if an author has written several books? Yes, the DISTINCT keyword should remove such duplicates, and effectively, the PostgreSQL query will correctly remove duplicates first, and then apply LIMIT and OFFSET. However, the ROW_NUMBER() predicate always generates distinct row numbers before DISTINCT can remove them again. In other words, DISTINCT has no effect. Luckily, we can tweak this SQL again, using this neat little trick: -- Better SQL Server equivalent: SELECT b.c1 AUTHOR_ID FROM ( SELECT DISTINCT AUTHOR_ID c1, DENSE_RANK() OVER (ORDER BY AUTHOR_ID) rn FROM BOOK ) b WHERE rn > 2 AND rn <= 3 Read more about this trick here: ...

Continuous Delivery: Unit Tests

In the previous article we explored static analysis as one of the first steps in Continuous Delivery. Our journey will continue with unit tests. Unit Tests Unit tests are probably the most important part of Continuous Delivery. While unit tests cannot substitute integration and functional tests, they are very easy to write and should be very fast to execute. As any other type of tests, each unit test should be independent of each other. What differentiates unit from other types of tests (integration, functional) is the scope. Each unit test should verify only a single unit of code (method, class, etc). Main benefit of unit tests are that they can find problems early due to their speed of execution. When ease of writing is combined with very short execution time it is no wonder that unit test often represent biggest part of automated tests. We won’t go deeper into unit tests. It is a huge subject that requires its own article. Test-Driven Development (TDD) For the successful implementation of continuous delivery, tests must be pushed to the repository at least at the same time as the implementation code. Otherwise, code will not be tested (at least not in the same build that delivered it to production). Without the code being tested, we’re running the risk of delivering to production the code that does not meet quality requirements. Remember, the final goal is to deliver to production the code after every push unless one of the steps in the pipeline failed. Even if you opt for slightly easier forms like continuous deployment or integration, unit tests pushed at the same time as the implementation code are a must. For this, and many other reasons, test-driven development is one of the crucial elements in the continuous delivery. While the minimum requirement is to push tests no later than the implementation code, with test-driven development we’re getting additional benefits like improved design, reliable documentation, etc. So, what is TDD? Test-driven development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or ‘rediscovered’ the technique, stated in 2003 that TDD encourages simple designs and inspires confidence. Please consult the example walk-through and Java best practices for more info. Jenkins For those of you who did not follow the previous articles in detail or those who failed to reproduce all exercises, please install Vagrant and clone the code from the repository TechnologyConversationsCD. To create a virtual machine with Jenkins and myFirstJob job that performs static analysis, please go to the directory servers/jenkinsWithAnalysis inside the cloned repository and execute the following: vagrant up If this is not the first time you’re starting this vagrant VM (“vagrant up” is configured to run the Docker container only the first time) or for some reason Docker container is not running, it can run with the following shortcut script. /opt/start_docker_with_jenkins.sh We can check which containers are running by executing: docker ps When finished, virtual machine with Jenkins and myFirstJob job will be up and running. Results can be seen by opening http://localhost:8080 in browser. We’ll continue using Gradle as our build tool. In the last article we were executing the Gradle task check to run static analysis (CheckStyle, FindBugs and PMD). Good thing about the check task is that it runs “all verification tasks in the project, including test.” So, there’s nothing for us to do to run the tests. They are already running. We should, however, put a bit of effort to display test results in a bit nicer way. Currently, they are only visible as the build status (red if tests failed) and through the logs. To publish the test results, we should add the post-build action called “Publish JUnit test results report”. As “Test report XMLs” we should put the path “build/test-results/**/*.xml”. Jenkins JUnit reports have few things left to be desired. For that reason, in case of Gradle, I prefer using “HTML Publisher Plugin”. Please install it in the same way as other plugins (Manage Jenkins > Manage Plugins > Available). It can be added to the job in the similar manner as the previous plugin. Select “Publish HTML Reports” from the “Add post-build action” list. Click the “Add” button and as the directory set “build/reports/tests/”. “Keep past HTML reports” is useful if historical data is required. To see the reports we just made we need to execute a build either by pushing something to the repository and waiting until Jenkins picks it up or by running it manually using the “Build Now” button1. If you’re using the same repository as the one used by this article, manual build is the way to go. Reports are located in the job itself (“HTML Report” and “Latest Test Result” links). There is also a graph located on the right side below static analysis. Moreover, the same links appear inside each build only this time located on the left-hand side of the screen. Big question is whether we need reports at all. In case of true continuous integration, most of the time there should be no errors at all (developers should be running unit tests locally before pushing the code). When errors do occur, they should be limited in number (usually only one) and of short duration (priority should be to fix failures as soon as possible). Those characteristics mean that we often need only logs (to see details of some error) and reports are most of the time the same (everything is green). However, many organizations cannot get to this point easily. There will be more than few errors, it will take hours or days instead of minutes to fix them and management will require reports no matter the results. If that’s the case, Jenkins reporting plugins come in handy. That does not mean that the suggestion is to go down this route (a lot of errors and a lot of time until they’re fixed) but that the reality is often different and that it can take considerable time to reach the end goal: true continuous integration, deployment or delivery. Travis There is nothing we should do in Travis. Travis is already running the Gradle “check” task that, among other things, executes tests. Unlike Jenkins’ plugins that allow nice visualization of test reports, Travis has only logs and successful or failed statuses. We already discussed pros and cons of not having reports. Circle As with Travis, Circle does not require anything special to be done to run tests. It already knows that Gradle has the test task and executes it. We already saw in the previous article that Circle is very similar to Travis in its approach to CI. Major difference was in the stability and speed (it is several times faster than Travis). With tests we can explore another difference. Circle has the option to store build artifacts. With this option we can accomplish the similar effect as what we did with Jenkins. We’ll allow users to see the Gradle tests reports. All we need to do is to modify our circle.yml file. [circle.yml] test: post: - cp -R build/reports/tests/* $CIRCLE_ARTIFACTS Full source code can be found in the circle.yml. Once this change is pushed to the repository Circle will store all our reports and make them available through the “Build Artifacts” option available in each build. By opening index.html we can see the nice report Gradle generated for us. Summary It was fairly easy to set up the execution of tests in all three tools (Jenkins, Travis and Circle). Actually, the execution was already done for us by the Gradle “check” task we were using for the static analysis. The only thing we had to do was to tell Jenkins and Circle where our reports are (Travis does not have that luxury). Jenkins continues to shine with its plugin system. Circle’s “Build Artifacts” was a pleasant surprise and another handy addition to its speed. The more I’m using it, the more I can see the advantages when compared to Travis. On the other hand, Travis’ price is unbeatable (public repositories are free). In the next article we’ll explore code coverage. It will help us determine how much of our code is actually tested. Stay tuned.Word of caution: make sure to wipe the workspace ([JOB] > Workspace > Wipe Out Current Workspace) if running the same build multiple times without actually making any changes to the code. Gradle will detect that there are no changes and do nothing. This is a good thing since that way a lot of time can be saved but it can cause a bit of confusion when testing Jenkins job configuration.Reference: Continuous Delivery: Unit Tests from our JCG partner Viktor Farcic at the Technology conversations blog....

Hibernate Debugging – Finding the origin of a Query

It’s not always immediate why and in which part of the program is Hibernate generating a given SQL query, especially if we are dealing with code that we did not write ourselves. This post will go over how to configure Hibernate query logging, and use that together with other tricks to find out why and where in the program a given query is being executed. What does the Hibernate query log look like Hibernate has built-in query logging that looks like this: select /* load your.package.Employee */ this_.code, ... from employee this_ where this_.employee_id=?TRACE 12-04-2014@16:06:02 BasicBinder - binding parameter [1] as [NUMBER] - 1000 Why can’t Hibernate log the actual query ? Notice that what is logged by Hibernate is the prepared statement sent by Hibernate to the JDBC driver plus it’s parameters. The prepared statement has ? in the place of the query parameters, the parameter values themselves are logged just bellow the prepared statement. This is not the same as the actual query sent to the database, as there is no way for Hibernate to log the actual query. The reason for this is that Hibernate only knows about the prepared statements and the parameters that it sends to the JDBC driver, and it’s the driver that will build the actual queries and then send them to the database. In order to produce a log with the real queries, a tool like log4jdbc is needed, which will be the subject of another post. How to find out the origin of the query The logged query above contains a comment that allows to identify in most cases the origin of the query: if the query is due to a load by ID the comment is /* load your.entity.Name */, if it’s a named query then the comment will contain the name of the query. If it’s a one to many lazy initialization the comment will contain the name of the class and the property that triggered it, etc. Setting up the Hibernate query log In order to obtain a query log, the following flags need to be set in the configuration of the session factory: <bean id= "entityManagerFactory" class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean" > ... <property name="jpaProperties" > <props> <prop key="hibernate.show_sql" >true</ prop> <prop key="hibernate.format_sql" >true</ prop> <prop key="hibernate.use_sql_comments">true</prop> </props> </property> The example above is for Spring configuration of an entity manager factory. This is the meaning of the flags:show_sql enables query logging format_sql pretty prints the SQL use_sql_comments adds an explanatory commentIn order to log the query parameters, the following log4j or equivalent information is needed: <logger name="org.hibernate.type"> <level value="trace" /> </logger > If everything else fails In many cases the comment created by use_sql_comments is enough to identify the origin of the query. If this is not sufficient, then we can start by identifying the entity returned by the query based on the table names involved, and put a breakpoint in the constructor of the returned entity. If the entity does not have a constructor, then we can create one and put the breakpoint in the call to super(): @Entity public class Employee { public Employee() { super(); // put the breakpoint here } ... } When the breakpoint is hit, go to the IDE debug view containing the stack call of the program and go through it from top to bottom. The place where the query was made in the program will be there in the call stack.Reference: Hibernate Debugging – Finding the origin of a Query from our JCG partner Aleksey Novik at the The JHades Blog blog....

Creating files and directories in NIO.2

Great number of applications nowadays create files or directories for very wide range of purposes. Whether it is to generate a report, export piece of configuration or simply to store some data it is important to be able to handle these tasks. Creating files and directories is one of the most heavily used functionality while working with a file system. This part of library underwent quite a modernization. Updates in this area include guarantee of atomicity of certain operations, creation of files and directories with preset file attributes, performance optimization as well as introduction of exception hierarchy that replaced boolean returning methods from prior versions of IO library.     Checking methods Before we get down to any code or explanation, let me take a step back and focus on something that will be essential not only to this post but also a number of posts to come. I find it important to be familiar a few methods usually called checking methods. Checking methods include all those methods used to perform various checks before calling the actual file system manipulation code. For convenience, they are all in class java.nio.file.Files. Using these methods will help you prevent unexpected behavior of your application. Since these methods are really simple, I will skip examples dedicated to them and instead use them in later examples.Checking methodsMethod name Descriptionexists(Path path, LinkOption... options) Tests whether a file exists.isExecutable(Path path) Tests whether a file is executable.isHidden(Path path) Tells whether or not a file is considered hidden.isReadable(Path path) Tests whether a file is readable.isRegularFile(Path path, LinkOption... options) Tests whether a file is a regular file with opaque content.isSameFile(Path path, Path path2) Tests if two paths locate the same file.isWritable(Path path) Tests whether a file is writable.notExists(Path path, LinkOption... options) Tests whether the file located by this path does not exist.  Creating a new directory One of the most important uses of class Files is to create new directories using method createDirectory. Directory creation is pretty simple and straight forward process so there is not much to explain. As usual it is always a good idea to use checking method exists from class Files to ensure that it is possible to create a directory with given path and also to prevent FileAlreadyExistsException. Whole situation is presented in the following code snippet: Path newDirectoryPath = Paths.get("/home/jstas/directory");if (!Files.exists(newDirectoryPath)) { try { Files.createDirectory(newDirectoryPath); } catch (IOException e) { System.err.println(e); } } The code sample is pretty simple – it creates a directory with provided path given no other file system entry resides on provided path. If we need to create whole directory hierarchy then we need to switch to method createDirectories which behaves similarly and creates whole hierarchy defined by a path instance. Since a directory is a type of file we are able to set its own metadata (file attributes). Not only are we able to do this, we might even create metadata definition beforehand and create a directory with initial file attributes in an atomic operation preventing any inconsistencies along the way. As mentioned in my previous article, there are two supported standards for managing file system permissions: POSIX and ACL. POSIX file permissions First, lets look at how we can manage file system permissions on POSIX-compliant systems like Linux-based systems and Mac OS. Thanks to the fact that POSIX file permissions are rather simple to understand, library creators provide us with convenience tools such as direct translation from string representation to a set of PosixFilePermissions or conversion tool to convert said set into FileAttribute object. This is not the only way to create FileAttribute object as we will see in next chapter. Getting back to the example at hand, lets look at the following code. Using convenience method fromString of class PosixFilePermissions we are able to create a set of PosixFilePermissions. Now it is necessary to create FileAttribute instance to be passed to createDirectory method that creates our test directory. Lets look at following snippet of code: Path newDirectoryPath = Paths.get("/home/jstas/testPosix");if (!Files.exists(newDirectoryPath)) { Set<PosixFilePermission> permissions = PosixFilePermissions.fromString("r-xr-----"); FileAttribute<Set<PosixFilePermission>> fileAttributes = PosixFilePermissions.asFileAttribute(permissions);try { Files.createDirectory(newDirectoryPath, fileAttributes); } catch (IOException e) { System.err.println(e); } } It is easy to validate whether our permissions were set correctly. You can either read file attributes directly from Java code like I presented in File attributes article or do it manually. I used systems terminal to check them with following output: dr-xr-----. 2 jstas jstas 4096 Jan 5 13:34 testPosix ACL file permissions Things get a little bit more complex when managing file system permissions on ACL-compliant systems such as Windows (NT, 2000, XP and newer). ACL lists can get pretty complex and robust so there are no shortcuts here like with POSIX file permissions. The key here is to use an anonymous class definition based on the interface FileAttribute. This interface defines only two methods: name returns the name of a file attribute and value returns value of this attribute. When working with ACL, the name of an attribute we are interested in is ‘acl:acl’. value method just returns list of constructed ACL entries. Lets take a look at what’s hidden inside an ACL entry and how to create an instance of AclEntry. First of all, ACL entry consists of several objects:FlagsThe flags component is a set of flags to indicate how entries are inherited and propagated Values: DIRECTORY_INHERIT, FILE_INHERIT, INHERIT_ONLY, NO_PROPAGATE_INHERITPermissionsThe permissions component is a set of permissions Values: APPEND_DATA, DELETE, DELETE_CHILD, EXECUTE, READ_ACL, READ_ATTRIBUTES, READ_DATA, READ_NAMED_ATTRS, SYNCHRONIZE, WRITE_ACL, WRITE_ATTRIBUTES, WRITE_DATA, WRITE_NAMED_ATTRS, WRITE_OWNERTypeThe type component determines if the entry grants or denies access. Values: ALARM, ALLOW, AUDIT, DENYPrincipalThe principal component, sometimes called the “who” component, is a UserPrincipal corresponding to the identity that the entry grants or denies access Values retrieved using UserPrincipalLookupServiceGiven the complexity of a single ACL entry, creators of NIO.2 library saw a very suitable candidate for implementation of a builder pattern. Visit following page for more information on design patterns and builder pattern. So the implementation selects appropriate flags and permissions, binds them with an user principal and sets the type of entry. Please study following code snippet to get familiar with ACL permissions: Path newDirectoryPath = Paths.get("c:", "testACL");if (!Files.exists(newDirectoryPath)) { FileAttribute<List<AclEntry>> fileAttributes = new FileAttribute<List<AclEntry>>() {@Override public List<AclEntry> value() { // lookup user principal FileSystem fileSystem = FileSystems.getDefault(); UserPrincipalLookupService userPrincipalLookupService = fileSystem.getUserPrincipalLookupService(); UserPrincipal userPrincipal = null; try { userPrincipal = userPrincipalLookupService.lookupPrincipalByName("JStas"); } catch (IOException e) { throw new RuntimeException(e); }// select ACL flags Set<AclEntryFlag> flags = EnumSet.of(AclEntryFlag.FILE_INHERIT, AclEntryFlag.DIRECTORY_INHERIT);// select ACL permission Set<AclEntryPermission> permissions = EnumSet.of(AclEntryPermission.READ_DATA, AclEntryPermission.WRITE_DATA, AclEntryPermission.EXECUTE);// build ACL entry Builder builder = AclEntry.newBuilder(); builder.setFlags(flags); builder.setPermissions(permissions); builder.setPrincipal(userPrincipal); builder.setType(AclEntryType.DENY);AclEntry entry = builder.build(); List<AclEntry> aclEntryList = new ArrayList<>(); aclEntryList.add(entry);return aclEntryList; }@Override public String name() { return "acl:acl"; } };try { Files.createDirectory(newDirectoryPath, fileAttributes); } catch (IOException e) { System.err.println(e); } } To verify successful creation of a directory and its file attributes in Windows 7, select security tab in properties of given folder and click on Advanced. Your newly created entry should be listed in presented table with detail view similar to this one:Creating a new file The core part of any file system related code usually involves code that creates single or more files. To create a file we need to use class Files again and call method createFile. Just like a directory, a file can be created with initial file attributes and same restrictions apply. Having said that I’m not going to demonstrate the work with file attributes since it is the same as in directory example. Once again this is really simple method with no catch to it so everything is presented in following example: Path newFilePath = Paths.get("C:", "a.txt");if (!Files.exists(newFilePath)) { try { Files.createFile(newFilePath); } catch (IOException e) { System.err.println(e); } } Please notice the use of exists checking method that prevents FileAlreadyExistsException.Reference: Creating files and directories in NIO.2 from our JCG partner Jakub Stas at the Jakub Stas blog....

Pitfalls of the Hibernate Second-Level / Query Caches

This post will go through how to setup the Hibernate Second-Level and Query caches, how they work and what are their most common pitfalls. The Hibernate second level cache is an application level cache for storing entity data. The query cache is a separate cache that stores query results only. The two caches really go together, as there are not many cases where we would like to use one without the other. When well used these caches provide improved performance in a transparent way, by reducing the number of SQL statements that hit the database.   How does the second level-cache work? The second level cache stores the entity data, but NOT the entities themselves. The data is stored in a ‘dehydrated’ format which looks like a hash map where the key is the entity Id, and the value is a list of primitive values. Here is an example on how the contents of the second-level cache look: *-----------------------------------------* | Person Data Cache | |-----------------------------------------| | 1 -> [ "John" , "Q" , "Public" , null ] | | 2 -> [ "Joey" , "D" , "Public" , 1 ] | | 3 -> [ "Sara" , "N" , "Public" , 1 ] | *-----------------------------------------* The second level cache gets populated when an object is loaded by Id from the database, using for example entityManager.find(), or when traversing lazy initialized relations. How does the query cache work? The query cache looks conceptually like an hash map where the key is composed by the query text and the parameter values, and the value is a list of entity Id’s that match the query: *----------------------------------------------------------* | Query Cache | |----------------------------------------------------------| | ["from Person where firstName=?", ["Joey"] ] -> [1, 2] ] | *----------------------------------------------------------* Some queries don’t return entities, instead they return only primitive values. In those cases the values themselves will be stored in the query cache. The query cache gets populated when a cacheable JPQL/HQL query gets executed. What is the relation between the two caches? If a query under execution has previously cached results, then no SQL statement is sent to the database. Instead the query results are retrieved from the query cache, and then the cached entity identifiers are used to access the second level cache. If the second level cache contains data for a given Id, it re-hydrates the entity and returns it. If the second level cache does not contain the results for that particular Id, then an SQL query is issued to load the entity from the database. How to setup the two caches in an application The first step is to include the hibernate-ehcache jar in the classpath: <dependency> <groupId>org.hibernate</groupId> <artifactId>hibernate-ehcache</artifactId> <version>SOME-HIBERNATE-VERSION</version> </dependency> The following parameters need to be added to the configuration of your EntityManagerFactory or SessionFactory: <prop key="hibernate.cache.use_second_level_cache">true</prop> <prop key="hibernate.cache.use_query_cache">true</prop> <prop key="hibernate.cache.region.factory_class">org.hibernate.cache.ehcache.EhCacheRegionFactory</prop> <prop key="net.sf.ehcache.configurationResourceName">/your-cache-config.xml</prop> Prefer using EhCacheRegionFactory instead of SingletonEhCacheRegionFactory. Using EhCacheRegionFactory means that Hibernate will create separate cache regions for Hibernate caching, instead of trying to reuse cache regions defined elsewhere in the application. The next step is to configure the cache regions settings, in file your-cache-config.xml: <?xml version="1.0" ?> <ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" updateCheck="false" xsi:noNamespaceSchemaLocation="ehcache.xsd" name="yourCacheManager"><diskStore path="java.io.tmpdir"/><cache name="yourEntityCache" maxEntriesLocalHeap="10000" eternal="false" overflowToDisk="false" timeToLiveSeconds="86400" /><cache name="org.hibernate.cache.internal.StandardQueryCache" maxElementsInMemory="10000" eternal="false timeToLiveSeconds="86400" overflowToDisk="false" memoryStoreEvictionPolicy="LRU" /><defaultCache maxElementsInMemory="10000" eternal="false" timeToLiveSeconds="86400" overflowToDisk="false" memoryStoreEvictionPolicy="LRU" /> </ehcache> If no cache settings are specified, default settings are taken, but this is probably best avoided. Make sure to give the cache a name by filling in the name attribute in the ehcache element. Giving the cache a name prevents it from using the default name, which might already be used somewhere else on the application. Using the second level cache The second level cache is now ready to be used. In order to cache entities, annotate them with the @org.hibernate.annotations.Cache annotation: @Entity @Cache(usage=CacheConcurrencyStrategy.READ_ONLY, region="yourEntityCache") public class SomeEntity { ... } Associations can also be cached by the second level cache, but by default this is not done. In order to enable caching of an association, we need to apply @Cache to the association itself: @Entity public class SomeEntity { @OneToMany @Cache(usage=CacheConcurrencyStrategy.READ_ONLY, region="yourCollectionRegion") private Set<OtherEntity> other; } Using the query cache After configuring the query cache, by default no queries are cached yet. Queries need to be marked as cached explicitly, this is for example how a named query can be marked as cached: @NamedQuery(name="account.queryName", query="select acct from Account ...", hints={ @QueryHint(name="org.hibernate.cacheable", value="true") } }) And this is how to mark a criteria query as cached: List cats = session.createCriteria(Cat.class) .setCacheable(true) .list(); The next section goes over some pitfalls that you might run into while trying to setup these two caches. These are behaviors that work as designed but still can be surprising. Pitfall 1 – Query cache worsens performance causing a high volume of queries There is an harmful side-effect of how the two caches work, that occurs if the cached query results are configured to expire more frequently than the cached entities returned by the query. If a query has cached results, it returns a list of entity Id’s, that is then resolved against the second level cache. If the entities with those Ids where not configured as cacheable or if they have expired, then a select will hit the database per entity Id. For example if a cached query returned 1000 entity Ids, and non of those entities where cached in the second level cache, then 1000 selects by Id will be issued against the database. The solution to this problem is to configure query results expiration to be aligned with the expiration of the entities returned by the query. Pitfall 2 – Cache limitations when used in conjunction with @Inheritance It is currently not possible to specify different caching policies for different subclasses of the same parent entity. For example this will not work: @Entity @Inheritance @Cache(CacheConcurrencyStrategy.READ_ONLY) public class BaseEntity { ... }@Entity @Cache(CacheConcurrencyStrategy.READ_WRITE) public class SomeReadWriteEntity extends BaseEntity { ... }@Entity @Cache(CacheConcurrencyStrategy.TRANSACTIONAL) public class SomeTransactionalEntity extends BaseEntity { ... } In this case only the @Cache annotation of the parent class is considered, and all concrete entities have READ_ONLY concurrency strategy. Pitfall 3 – Cache settings get ignored when using a singleton based cache It is advised to configure the cache region factory as a EhCacheRegionFactory, and specify an ehcache configuration via net.sf.ehcache.configurationResourceName. There is an alternative to this region factory which is SingletonEhCacheRegionFactory. With this region factory the cache regions are stored in a singleton using the cache name as a lookup key. The problem with the singleton region factory is that if another part of the application had already registered a cache with the default name in the singleton, this causes the ehcache configuration file passed via net.sf.ehcache.configurationResourceName to be ignored. Conclusion The second level and query caches are very useful if set up correctly, but there are some pitfalls to bear in mind in order to avoid unexpected behaviors. All in all it’s a feature that works transparently and that if well used can increase significantly the performance of an application.Reference: Pitfalls of the Hibernate Second-Level / Query Caches from our JCG partner Aleksey Novik at the The JHades Blog blog....

Don’t be a Slave to Your Tools

Developers attach quickly to tools because they are concrete and have well defined behavior.  It is easier to learn a tool than to learn best practices or methodology. Tools only assist in solving problems, they can’t solve the problem by themselves. A developer who understands the problem can use tools to increase productivity and quality. Poor developers don’t invest the time or effort to understand how to code properly and avoid defects.  They spend their time learning how to use tools without understanding the purpose of the tool or how to use it effectively. To some degree, this is partially the fault of the tool vendors. The tool vendors perceive an opportunity to make $$$$$ based on providing support for a common problems, such as:defect trackers to help you manage defect tracking version control systems to manage source code changes tools to support Agile development (Version One, JIRA) debuggers to help you find defectsThere are many tools out there, but let’s just go through this list and point out where developers and organizations get challenged.  Note, all statistics below are derived from over 15,000 projects over 40 years.1 Defect Trackers Believe it or not, some organizations still don’t have defect tracking software. I’ve run into a couple of these companies and you would not believe why. The effect of not having a defect tracking system is pretty severe, and there is evidence to prove this. Inadequate defect tracking methods: productivity -15%, quality -21%So we are pretty much all in agreement that we need to have defect tracking; we all know that the ability to manage more than a handful of defects is impossible without some kind of system. Automated defect tracking tools: productivity +18%, quality +26%   The problem is that developers disagree over what might be the best defect tracking system. The real problem is that almost every defect tracking system is poorly set-up, leading to poor results. Virtually every defect tracking system when configured properly will yield tremendous benefits. The most common pitfalls are:Introducing irrelevant attributes into the defect lifecycle status, i.e. creation of statuses like deferred, won’t fix, or functions as designed Not being able to figure out if something is fixed or not Not understanding who is responsible for addressing a defectThe tool vendors are happy to continue to provide new versions of defect trackers. However, using a defect tracker effectively has more to do with how the tool is used rather than which tool is selected. One of the most fundamental issues that organizations wrestle with is what is a defect?  A defect only exists if the code does not behave according to specifications. But what if there are no specifications or the specifications are bad?  See It’s not a bug, it’s… for more information. Smart organizations understand that the way in which the defect tracker is used will make the biggest difference.  Discover how to get more out of you defect tracking system in Bug Tracker Hell and How to Get Out. Another common problem is that organizations try to manage enhancements and requirements in the defect tracking system. After all whether it is a requirement or a defect it will lead to a code change, so why not put all the information into the defect tracker?  Learn why managing requirements and enhancements in the defect tracking system is foolish in Don’t manage enhancements in the bug tracker. Version Control Systems Like defect tracking systems most developers have learned that version control is a necessary hygiene procedure.  If you don’t have one then you are likely to catch a pretty serious disease (and at the least convenient time). Inadequate change control: productivity -11%, quality -16% Virtually all developers dislike version control systems and are quite vocal about what they can’t do with their version control system.  If you are the unfortunate person who made the final decision on which version control system is used just understand that their are hordes of developers out their cursing you behind your back. Version control is simply chapter 1 of the story.  Understanding how to chunk code effectively, integrate with continuous build technology, and making sure that the defects in the defect tracker refers to the correct version are just as important as the choice of version control system. Tools to support Agile Sorry Version One and JIRA, the simple truth is that using an Agile tool does not make you agile, see this. These tools are most effective when you actually understand Agile development. One of my most effective Agile implementations only used Google docs in the implementation. Enough said. Debuggers I have written extensively about why debuggers are not the best primary tools to track down defects.  So I’ll try a different approach here! One of the most enduring sets of ratios in software engineering has been 1:10:100.  That is, if the cost of tracking down a defect pre-test (i.e. before QA) is 1, then it will cost 10x if the defect is found by QA, and 100x if the defect is discovered in deployment by your customers. Most debuggers are invoked when the cost function is in the 10x or 100x part of the process. It is not that I dislike debuggers — I simply believe in using pre-test defect removal strategies because they cost less and lead to higher code quality. Pre-test defect removal strategies include:Planning code, i.e. PSP Test driven development, TDD Design by Contract (DbC) Code inspections Pair programming for complex sections of codeYou can find more information about this in:Defects are for Losers Not planning is for Losers Debuggers are for Losers Are debuggers crutchesSeldom Used Tools Tools that can make a big difference but many developers don’t use them:Automated static analysis: productivity +21%, quality +31% Automated unit testing: productivity +17%, quality +24% Automated unit testing generally involves using test driven development (TDD) or data driven development together with continual build technology. Automated sizing in function points: productivity +17%, quality +24% Automated quality and risk prediction: productivity +16%, quality +23% Automated test coverage analysis: productivity +15%, quality +21% Automated deployment support: productivity +15%, quality +20% Automated cyclomatic complexity computation: productivity +15%, quality +20%Important Techniques with No Tools There are a number of techniques available in software development that tool vendors have not found a way to monetize on. These techniques tend to be overlooked by most developers, even though they can make a huge difference in productivity and quality. The Personal Software Process and Team Software Process were developed by Watts Humphrey, one of the pioneers of building quality software.Personal software process: productivity +21%, quality +31%2 Team software process: productivity +21%, quality +31%3The importance of inspections is covered in:Inspections are not Optional Software Professionals do InspectionsCode inspections: productivity +21%, quality +31%4 Requirement inspections: productivity +18%, quality +27%4 Formal test plans: productivity +17%, quality +24% Function point analysis (IFPUG): productivity +16%, quality +22%ConclusionThere is definitely a large set of developers that assume that using a tool makes them competent. The reality is that learning a tool without learning the principles that underlie the problem you are solving is like assuming you can beat the great Michael Jordan at basketball just because you have great running shoes. Learning tools is not a substitute for learning how do do something competently. Competent developers are continually learning about techniques that lead to higher productivity and quality, whether or not that technique is supported by a tool. ResourcesGilb, Tom and Graham, Dorothy. Software Inspections 1Jones, Capers. SCORING AND EVALUATING SOFTWARE METHODS, PRACTICES, AND RESULTS. 2008. Jones, Capers. The Economics of Software Quality. 2011 Radice, Ronald A. High Quality 2Watts, Humphrey. Introduction to the Personal Software Process 3Watts, Humphrey. Introduction to the Team Software ProcessReference: Don’t be a Slave to Your Tools from our JCG partner Dalip Mahal at the Accelerated Development blog....

Working with files and directories in NIO.2

In previous articles I discussed creation (Creating files and directories) and selection (Listing and filtering directory contents) of files and directories. The last logical step to take is to explore what can we do with them and how. This is a part of the library that was redesigned in a big way. Updates in this area include guarantee of atomicity of certain operations, API improvements, performance optimization as well as introduction of proper exception hierarchy that replaced boolean returning methods from prior versions of IO library.         Opening a file Before we get down to reading from and writing to a file, we need to cover one common ground of these operations – the way files are opened. The way files are opened directly influences results of these operations as well as their performance. Lets take a look at standard options of opening files contained in enum java.nio.file.StandardOpenOption:Standard open optionsValue DescriptionAPPEND If the file is opened for WRITE access then bytes will be written to the end of the file rather than the beginning.CREATE Create a new file if it does not exist.CREATE_NEW Create a new file, failing if the file already exists.DELETE_ON_CLOSE Delete on close.DSYNC Requires that every update to the file’s content be written synchronously to the underlying storage device.READ Open for read access.SPARSE Sparse file.SYNC Requires that every update to the file’s content or metadata be written synchronously to the underlying storage device.TRUNCATE_EXISTING If the file already exists and it is opened for WRITE access, then its length is truncated to 0.WRITE Open for write access.These are all standard options that you as a developer may need to properly handle opening of files whether it is for reading or writing. Reading a file When it comes to reading files NIO.2 provides several ways to do it – each with its pros and cons. These approaches are as follows:Reading a file into a byte array Using unbuffered streams Using buffered streamsLets take a look at first option. Class Files provides method readAllBytes to do exactly that. Reading a file into a byte array seems like a pretty straight forward action but this might be suitable only for a very restricted range of files. Since we are putting the entire file into the memory we must mind the size of that file. Using this method is reasonable only when we are trying to read small files and it can be done instantly. It is pretty simple operation as presented in this code snippet: Path filePath = Paths.get("C:", "a.txt");if (Files.exists(filePath)) { try { byte[] bytes = Files.readAllBytes(filePath); String text = new String(bytes, StandardCharsets.UTF_8);System.out.println(text); } catch (IOException e) { throw new RuntimeException(e); } } The code above first reads a file into a byte array and then constructs string object containing contents of said file with following output: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam sit amet justo nec leo euismod porttitor. Vestibulum id sagittis nulla, eu posuere sem. Cras commodo, massa sed semper elementum, ligula orci malesuada tortor, sed iaculis ligula ligula et ipsum. When we need to read the contents of a file in string form we can use the code above. However, this solution is not that clean and we can use readAllLines from class Files to avoid this awkward construction. This method serves as a convenient solution to reading files when we need human-readable output line by line. The use of this method is once again pretty simple and quite similar to the previous example (same restrictions apply): Path filePath = Paths.get("C:", "b.txt");if (Files.exists(filePath)) { try { List<String> lines = Files.readAllLines(filePath, StandardCharsets.UTF_8);for (String line : lines) { System.out.println(line); } } catch (IOException e) { throw new RuntimeException(e); } } With following output: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aliquam sit amet justo nec leo euismod porttitor. Vestibulum id sagittis nulla, eu posuere sem. Cras commodo, massa sed semper elementum, ligula orci malesuada tortor, sed iaculis ligula ligula et ipsum. Reading a file using streams Moving on to more sophisticated approaches we can always use good old streams just like we were used to from prior versions of the library. Since this is a well-known ground I’m only going to show how to get instances of these streams. First of all, we can retrieve InputStream instance from class Files by calling newInputStream method. As usual, one can further play with a decorator pattern and make a buffered stream out of it. Or for a convenience use method newBufferedReader. Both methods return a stream instance that is plain old java.io object. Path filePath1 = Paths.get("C:", "a.txt"); Path filePath2 = Paths.get("C:", "b.txt");InputStream is = Files.newInputStream(filePath1); InputStreamReader isr = new InputStreamReader(is); BufferedReader br = new BufferedReader(isr);BufferedReader reader = Files.newBufferedReader(filePath2, StandardCharsets.UTF_8); Writing to a file Writing to a file is similar to reading process in a range of tools provided by NIO.2 library so lets just review:Writing a byte array into a file Using unbuffered streams Using buffered streamsOnce again lets explore the byte array option first. Not surprisingly, class Files has our backs with two variants of method write. Either we are writing bytes from an array or lines of text, we need to focus on StandardOpenOptions here because both methods can be influenced by custom selection of these modifiers. By default, when no StandardOpenOption is passed on to the method, write method behaves as if the CREATE, TRUNCATE_EXISTING, and WRITE options were present (as stated in Javadoc). Having said this please beware of using default (no open options) version of write method since it either creates a new file or initially truncates an existing file to a zero size. File is automatically closed when writing is finished – both after a successful write and an exception being thrown. When it comes to file sizes, same restrictions as in readAllBytes apply. Following example shows how to write an byte array into a file. Please note the absence of any checking method due to the default behavior of write method. This example can be run multiple times with two different results. First run creates a file, opens it for writing and writes the bytes from the array bytes to this file. Any subsequent calling of this code will erase the file and write the contents of the bytes array to this empty file. Both runs will result in closed file with text ‘Hello world!’ written on the first line. Path newFilePath = Paths.get("/home/jstas/a.txt"); byte[] bytes = new byte[] {0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x77, 0x6f, 0x72, 0x6c, 0x64, 0x21};try { Files.write(newFilePath, bytes); } catch(IOException e) { throw new RuntimeException(e); } When we need to write lines instead of bytes we can convert a string to byte array, however, there is also more convenient way to do it. Just prepare a list of lines and pass it on to write method. Please note the use of two StandardOpenOptions in following example. By using these to options I am sure to have a file present (if it does not exist it gets created) and a way to append data to this file (thus not loosing any previously written data). Whole example is rather simple, take a look: Path filePath = Paths.get("/home/jstas/b.txt");List<String> lines = new ArrayList<>(); lines.add("Lorem ipsum dolor sit amet, consectetur adipiscing elit."); lines.add("Aliquam sit amet justo nec leo euismod porttitor."); lines.add("Vestibulum id sagittis nulla, eu posuere sem."); lines.add("Cras commodo, massa sed semper elementum, ligula orci malesuada tortor, sed iaculis ligula ligula et ipsum.");try { Files.write(filePath, lines, StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.APPEND); } catch (IOException e) { throw new RuntimeException(e); } Writing to a file using streams It might not be a good idea to work with byte arrays when it comes to a larger files. This is when the streams come in. Similar to reading chapter, I’m not going to explain streams or how to use them. I would rather focus on a way to retrieve their instances. Class Files provides method newOutputStream that accepts StandardOpenOptions to customize streams behavior. By default, when no StandardOpenOption is passed on to the method, streams write method behaves as if the CREATE, TRUNCATE_EXISTING, and WRITE options are present (as stated in Javadoc). This stream is not buffered but with a little bit of decorator magic you can create BufferedWriter instance. To counter this inconvenience, NIO.2 comes with newBufferWriter method that creates buffered stream instance right away. Both ways are shown in following code snippet: Path filePath1 = Paths.get("/home/jstas/c.txt"); Path filePath2 = Paths.get("/home/jstas/d.txt");OutputStream os = Files.newOutputStream(filePath1); OutputStreamWriter osw = new OutputStreamWriter(os); BufferedWriter bw = new BufferedWriter(osw);BufferedWriter writer = Files.newBufferedWriter(filePath2, StandardCharsets.UTF_8, StandardOpenOption.CREATE, StandardOpenOption.APPEND); Copying and moving files and directories Copying files and directories One of most welcomed features of NIO.2 is updated way of handling copying and moving files and directories. To keep everything nicely in line, designers decided to introduce two parent (marker) interfaces into new file system API: OpenOption and CopyOption (both interfaces from package java.nio.file). StandardOpenOption enum mentioned in previous chapter implements OpenOption interface. CopyOption interface on the other hand has two implementations, one of which we have already met in post about Links in NIO.2. Some of you may recall LinkOption enum which is said implementation guiding methods handling link related operations. However, there is another implementation – StandardCopyOption enum from package java.nio.file. Once again, we are presented with yet another enumeration – used to guide copy operations. So before we get down to any code lets review what we can achieve using different options for copying.Standard copy optionsValue DescriptionATOMIC_MOVE Move the file as an atomic file system operation.COPY_ATTRIBUTES Copy attributes to the new file.REPLACE_EXISTING Replace an existing file if it exists.  Using these options to guide your IO operations is quite elegant and also simple. Since we are trying to copy a file, ATOMIC_MOVE does not make much sense to use (you can still use it, but you will end up with java.lang.UnsupportedOperationException: Unsupported copy option). Class Files provides 3 variants of copy method to serve different purposes:copy(InputStream in, Path target, CopyOption... options)Copies all bytes from an input stream to a file.copy(Path source, OutputStream out)Copies all bytes from a file to an output stream.copy(Path source, Path target, CopyOption... options)Copy a file to a target file.Before we get to any code I believe that it is good to understand most important behavioral features of copy method (last variant out of three above). copy method behaves as follows (based on Javadoc):By default, the copy fails if the target file already exists or is a symbolic link. If the source and target are the same file the method completes without copying the file. (for further information check out method isSameFile of class Files) File attributes are not required to be copied to the target file. If the source file is a directory then it creates an empty directory in the target location (entries in the directory are not copied). Copying a file is not an atomic operation. Custom implementations may bring new specific options.These were core principals of inner workings of copy method. Now is a good time to look at code sample. Since its pretty easy to use this method lets see it in action (using the most common form of copy method). As expected, following code copies the source file (and possibly overwrites the target file) preserving file attributes: Path source = Paths.get("/home/jstas/a.txt"); Path target = Paths.get("/home/jstas/A/a.txt");try { Files.copy(source, target, StandardCopyOption.COPY_ATTRIBUTES, StandardCopyOption.REPLACE_EXISTING); } catch (IOException e) { throw new RuntimeException(e); } No big surprises here – code copies source file with its file attributes. If you feel I forgot about (not empty) directories, let me assure you that I did not. It is also possible to use NIO.2 to copy, move or delete populated directories but this is what I am going to cover in the next post so you gonna have to wait a couple of days. Moving files and directories When it comes to moving files we again need to be able to specify options guiding the process for the method move from Files class. Here we make use of StandardCopyOptions mentioned in previous chapter. Two relevant options are ATOMIC_MOVE and REPLACE_EXISTING. First of all, lets start with some basic characteristics and then move on to a code sample:By default, the move method fails if the target file already exists. If the source and target are the same file the method completes without moving the file. (for further information check out method isSameFile of class Files) If the source is symbolic link, then the link itself is moved. If the source file is a directory than it has to be empty to be moved. File attributes are not required to be moved. Moving a file can be configured to be an atomic operation but doesn’t have to. Custom implementations may bring new specific options.Code is pretty simple so lets look at following code snippet: Path source = Paths.get("/home/jstas/b.txt"); Path target = Paths.get("/home/jstas/A/b.txt");try { Files.move(source, target, StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING); } catch(IOException e) { throw new RuntimeException(e); } As expected, code moves source file in an atomic operation. Removing files and directories Last part of this article is dedicated to deleting files and directories. Removing files is, once again, pretty straight forward with two possible methods to call (both from Files class, as usual):public static void delete(Path path) public static boolean deleteIfExists(Path path)Same rules govern both methods:By default, the delete method fails with DirectoryNotEmptyException when the file is a directory and it is not empty. If the file is a symbolic link then the link itself is deleted. Deleting a file may not be an atomic operation. Files might not be deleted if they are open or in use by JVM or other software. Custom implementations may bring new specific options.Path newFile = Paths.get("/home/jstas/c.txt"); Path nonExistingFile = Paths.get("/home/jstas/d.txt");try { Files.createFile(newFile); Files.delete(newFile);System.out.println("Any file deleted: " + Files.deleteIfExists(nonExistingFile)); } catch(IOException e) { throw new RuntimeException(e); } With an output: Any file deleted: falseReference: Working with files and directories in NIO.2 from our JCG partner Jakub Stas at the Jakub Stas blog....

Automating the Continuous Integration of Android Projects With Gradle Using Jenkins on Windows

This post will show how to automate the deployment process of an Android Application using Jenkins Continuous Integration – to build the project, run the unit tests (if any), archive the built artifacts and run the Android lint reports. 1. Install Jenkins as a Windows Service Navigate to jenkins-ci.org website using an Internet browser and download the Windows native package (the link is underlined for easy identification) as shown from the right side pane of the Download Jenkins tab.    Once the download is complete, uncompress the zip file and click on the jenkins-1.xxx.msi file. Proceed through the configuration steps to install the Jenkins as a Windows service. 2. Modify Default Jenkins Port By default Jenkins runs on the port 8080. In order to avoid conflict with other applications, the default port can be modified by editing the jenkins.xml found under C:\Program Files (x86)\Jenkins location. As shown below, modify the httpPort to 8082. <service> <id>jenkins</id> <name>Jenkins</name> <description>This service runs Jenkins continuous integration system.</description> <env name="JENKINS_HOME" value="%BASE%"/> <!-- if you'd like to run Jenkins with a specific version of Java, specify a full path to java.exe. The following value assumes that you have java in your PATH. --> <executable>%BASE%\jre\bin\java</executable> <arguments>-Xrs -Xmx256m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8082</arguments> <!-- interactive flag causes the empty black Java window to be displayed. I'm still debugging this. <interactive /> --> <logmode>rotate</logmode><onfailure action="restart" /> </service> Once the modification is saved in jenkins.xml file, restart the Jenkins service from the Windows Task Manager->Services and right clicking on the Jenkins service and choose Stop Service to stop the service as shown below.Once the status of the service changes to stopped, restart the service by right clicking on the Jenkins service and choose Start Service to start the service again.Navigate to localhost:8082 to verify if the Jenkins restart was successful as shown below – Jenkins Dashboard will be displayed. Note that it takes a while before the Jenkins service becomes available.3. Install Plugins On the Jenkins Dashboard, navigate to Manage Jenkins –> Manage Plugins as shown in the snapshot below.Install the following plugins and restart Jenkins for the changes to take effect.Git Plugin (for integrating Git with Jenkins) Gradle Plugin (for integrating Gradle with Jenkins) Android Lint Plugin (for integration Lint with Jenkins)4. Configure System On the Jenkins Dashboard, navigate to Manage Jenkins –> Configure System as shown in the snapshot below.Navigate to the Global Properties section and click on Add to add an Environment Variable ANDROID_HOME as shown in the snapshot below. Enter the name as ANDROID_HOME and enter the path of the location where the Android SDK is stored on windows.Navigate to the JDK section and click on “Add JDK” to add the JDK installation as shown in the snapshot below. Specify a JDK Name, choose the JDK Version to install and follow the on-screen instructions to save the Oracle Login credentials. Save the changes.Next, proceed to the Git section and click on “Add Git” to add the Git installation as shown in the snapshot below. Specify Git Name, specify the path to Git executable and Save the changes.Next, proceed to the Gradle section and click on “Add Gradle” to add the Gradle installation as shown in the snapshot below. Specify Gradle Name, choose the appropriate version (at the time of writing, I used Gradle 1.10) and Save the changes.Next, proceed to the Email Notification section and enter the SMTP Server details as shown below. Click on the Advanced button to add the further details required and Save the changes. Click on “Test configuration by sending test e-mail”, enter the test e-mail recipient and click on “Test configuration” to see if the email is successfully sent.5. Create a New Jenkins Job From the Jenkins Dashboard, click on “New Job” to create a new job. Enter a name for the job and choose “Build a free-style software project” as option and click on OK as shown below.From the New Job Configuration screen, proceed to the Source Code Management section. Save the Git credentials by clicking on “Add” as shown below and entering the details in the following dialog. Save the changes by clicking on “Add” as shown below.Specify the Git Repository URL for the project, choose the saved credentials from the drop-down list as shown in the snapshot below. Save the changes.Next, from the Build Triggers section, select the options desired as shown below and Save the changes.Proceed to the Build section, choose “Invoke Gradle script” from the drop-down list of choices for “Add build step”. Choose the appropriate Gradle version which is configured, enter the tasks to be built and select the options as desired. Save the changes.Proceed to the Post-build Actions section, click on “Publish Android Lint results” from the drop-down list of choices for “Add post-build action” and specify the location where the Lint results should be stored in the Jenkins workspace for the job. Similarly, click on “Archive the artifacts” from the drop-down list of choices for “Add post-build action” and the specify the format of APK files to be archived after every build. Additionally, options from Advanced section such as “Discard all but the last successful/stable artifact to save disk space” could be enabled for saving disk space. Click on “E-mail Notification” from the drop-down list of choices for “Add post-build action” and enter the values for the email recipients as shown below. Save the changes.6. Build Now Once the above configuration steps are complete, click on “Build Now” under the Jenkins –> Build Android Application (or the respective Job name) to build the project based on the configuration. The console output has the detailed logs of what steps were initiated by the configuration and the outcome of the entire build. Clicking on any successful build outcome shows the artifacts that were archived as part of the build, the change that started the build and the lint results as shown below.Thus the entire process of building the project an Android application project whenever a SCM change is triggered or under another condition, running lint reports, archiving the artifacts built, publishing lint reports and triggering emails to the recipients can be automated with a click of a button through Jenkins.Reference: Automating the Continuous Integration of Android Projects With Gradle Using Jenkins on Windows from our JCG partner Elizabeth Thomas at the My Experiments with Technology blog....

RSS Reader Using: ROME, Spring MVC, Embedded Jetty

In this post I will show some guidelines to create a Spring web application, running it using Jetty and using an external library called ROME for RSS reading. General I have recently created a sample web application that acts as an RSS reader. I wanted to examine ROME for RSS reading. I also wanted to create the application using Spring container and MVC for the simplest view. For rapid development, I used Jetty as the server, using a simple java class for it. All the code can be found at GitHub, eyalgo/rss-reader.   ContentMaven Dependencies Jetty Server Spring Dependency Spring MVC ROMEMaven Dependencies At first, I could not get the correct Jetty version to use. There is one with group-id mortby, and another by eclipse. After some careful examination and trial and error, I took the eclipse’s library. Spring is just standard. I found ROME with newest version under GutHub. It’s still a SNAPSHOT. Here’s the list of the dependencies:Spring jetty rome and rome-fetcher logback and slf4j For TestingJunit mockito hamcrest spring-testThe project’s pom file can be found at: https://github.com/eyalgo/rss-reader/blob/master/pom.xmlJetty Server A few years ago I’ve been working using Wicket framework and got to know Jetty, and its easy usage for creating a server. I decided to go in that direction and to skip the standard web server running with WAR deployment. There are several ways to create the Jetty server. I decided to create the server, using a web application context. First, create the context: private WebAppContext createContext() { WebAppContext webAppContext = new WebAppContext(); webAppContext.setContextPath("/"); webAppContext.setWar(WEB_APP_ROOT); return webAppContext; } Then, create the server and add the context as handler: Server server = new Server(port); server.setHandler(webAppContext); Finally, start the server: try { server.start(); } catch (Exception e) { LOGGER.error("Failed to start server", e); throw new RuntimeException(); } Everything is under https://github.com/eyalgo/rss-reader/tree/master/src/test/java/com/eyalgo/rssreader/server. Spring Project StructureSpring Dependency In web.xml I am declaring application-context.xml and web-context.xml . In web-context.xml , I am telling Spring were to scan for components: <context:component-scan base-package="com.eyalgo.rssreader"/>. In application-context.xml I am adding a bean, which is an external class and therefore I can’t scan it (use annotations): <bean id="fetcher" class="org.rometools.fetcher.impl.HttpURLFeedFetcher"/>. Besides scanning, I am adding the correct annotation in the correct classes. @Repository @Service @Controller @Autowired Spring MVC In order to have some basic view of the RSS feeds (and atoms), I used a simple MVC and JSP pages. To create a controller, I needed to add @Controller for the class. I added @RequestMapping("/rss") so all requests should be prefixed with rss. Each method has a @RequestMapping declaration. I decided that everything is GET. Adding a Parameter to the Request Just add @RequestParam("feedUrl") before the parameter of the method. Redirecting a Request After adding an RSS location, I wanted to redirect the answer to show all current RSS items. So the method for adding an RSS feed needed to return a String. The returned value is: “redirect:all”. @RequestMapping(value = "feed", method = RequestMethod.GET) public String addFeed(@RequestParam("feedUrl") String feedUrl) { feedReciever.addFeed(feedUrl); return "redirect:all"; } Return a ModelAndView Class In Spring MVC, when a method returns a String, the framework looks for a JSP page with that name. If there is none, then we’ll get an error. (If you want to return just the String, you can add @ResponseBody to the method.) In order to use ModelAndView, you need to create one with a name: ModelAndView modelAndView = new ModelAndView("rssItems"); The name will tell Spring MVC which JSP to refer to. In this example, it will look for rssItems.jsp. Then you can add to the ModelAndView “objects”: List<FeedItem> items = itemsRetriever.get(); ModelAndView modelAndView = new ModelAndView("rssItems"); modelAndView.addObject("items", items); In the JSP page, you need to refer the names of the objects you added. And then, you can access their properties. So in this example, we’ll have the following in rssItems.jsp: <c:forEach items="${items}" var="item"> <div> <a href="${item.link}" target="_blank">${item.title}</a><br> ${item.publishedDate} </div> </c:forEach> Note Spring “knows” to add jsp as a suffix to the ModelAndView name because I declared it in web-context.xml. In the bean of class: org.springframework.web.servlet.view.InternalResourceViewResolver. By setting the prefix this bean also tells Spring were to look for the jsp pages. Please look:https://github.com/eyalgo/rss-reader/blob/master/src/main/java/com/eyalgo/rssreader/web/RssController.java https://github.com/eyalgo/rss-reader/blob/master/src/main/webapp/WEB-INF/views/rssItems.jspError Handling There are several ways to handle errors in Spring MVC. I chose a generic way, in which for any error, a general error page will be shown. First, add @ControllerAdvice to the class you want to handle errors. Second, create a method per type of exception you want to catch. You need to annotate the method with @ExceptionHandler. The parameter tells which exception this method will handle. You can have a method for IllegalArgumentException and another for different exception and so on. The return value can be anything and it will act as normal controller. That means, having a jsp (for example) with the name of the object the method returns. In this example, the method catches all exception and activates error.jsp, adding the message to the page. @ExceptionHandler(Exception.class) public ModelAndView handleAllException(Exception e) { ModelAndView model = new ModelAndView("error"); model.addObject("message", e.getMessage()); return model; } ROME ROME is an easy to use library for handling RSS feeds : https://github.com/rometools/rome. rome-fetcher is an additional library that helps getting (fetching) RSS feeds from external sources, such as HTTP, or URL : https://github.com/rometools/rome-fetcher As of now, the latest build is 2.0.0-SNAPSHOT . An example on how to read an input RSS XML file can be found at: https://github.com/eyalgo/rss-reader/blob/master/src/test/java/com/eyalgo/rssreader/runners/MetadataFeedRunner.java To make life easier, I used rome-fetcher. It gives you the ability to give a URL (RSS feed) and have all the SyndFeed out of it. If you want, you can add caching, so it won’t download cached items (items that were already downloaded). All you need is to create the fetcher with FeedFetcherCache parameter in the constructor. Usage: @Override public List<FeedItem> extractItems(String feedUrl) { try { List<FeedItem> result = Lists.newLinkedList(); URL url = new URL(feedUrl); SyndFeed feed = fetcher.retrieveFeed(url); List<SyndEntry> entries = feed.getEntries(); for (SyndEntry entry : entries) { result.add(new FeedItem(entry.getTitle(), entry.getLink(), entry.getPublishedDate())); } return result; } catch (IllegalArgumentException | IOException | FeedException | FetcherException e) { throw new RuntimeException("Error getting feed from " + feedUrl, e); } }https://github.com/eyalgo/rss-reader/blob/master/src/main/java/com/eyalgo/rssreader/service/rome/RomeItemsExtractor.javaNote If you get a warning message (looks as System.out) that tells that fetcher.properties is missing, just add an empty file under resources (or in the root of the classpath). Summary This post covered several topics. You can also have a look at the way a lot of the code is tested. Check Matchers and mocks. If you have any remarks, please drop a note.Reference: RSS Reader Using: ROME, Spring MVC, Embedded Jetty from our JCG partner Eyal Golan at the Learning and Improving as a Craftsman Developer blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: