Featured FREE Whitepapers

What's New Here?


Installing Drupal on FreeBSD

Drupal ports have been available on FreeBSD since quite a long time, and binary packages can be installed very quickly. However, manual setup is required to connect Drupal to the database and have Apache serve the Drupal website. In this post I’ll describe the setup procedure of Drupal 7 on FreeBSD 10.0. The process will not be very different if different versions of Drupal or FreeBSD are used. Installing Drupal Drupal ports are available on FreeBSD and, in fact, multiple versions are available:     # pkg search drupal drupal6-6.31 [...snip...] drupal7-7.31 [...snip...] Unless there is a compelling reason not to do so, install the latest one: # pkg install drupal7 To successfully run Drupal, you need:The Apache HTTP Server. PHP. A supported database server (PostgreSQL or MySQL).The Drupal port, however, does not enforce these dependencies, so that you have to satisfy them manually. Installing the Apache HTTP Server Unless there is a compelling reason not to do so, install the latest available Apache port (apache24 at the time of writing): # pkg install apache24 Once the port is installed, enable the corresponding service adding the following line to /etc/rc.conf: apache24_enable="YES" Installing the Database Drupal supports both PostgreSQL and MySQL but the Drupal port does not install any, by default, although it installs the MySQL client utilities. In this post MySQL will be used but if you prefer using PostgreSQL instead, just skip this section and read this article instead. Since the Drupal port by default defines the MYSQL option, when you install the binary package using pkg you’ll also get a MySQL client port, such as what I got at the time of writing: mysql55-client-5.5.40 As a consequence, you have to install the matching mysqlXX-server port: # pkg install mysql55-server-5.5.40 If you try to install a different version (at the time of writing mysql56 is available), you may be requested to remove Drupal itself because of the inter-dependencies between the client and server MySQL packages. Once MySQL is installed, enable the corresponding service adding the following line to /etc/rc.conf: mysql_enable="YES" Installing PHP The installation of PHP is taken care of by the Drupal port. However, the PHP module for the Apache HTTP Server is not installed and must be installed manually. Make sure you install the PHP module that corresponds with the PHP version installed by the Drupal port. At the time of writing, the following modules are available: # pkg search mod_php mod_php5-5.4.33_1,1 mod_php55-5.5.17_1 mod_php56-5.6.1 Since the port depends on php5, then mod_php5-5.4.33_1,1 should be installed: # pkg install mod_php5-5.4.33_1,1 The port takes care of modifying the Apache HTTP Server configuration file so that the PHP module is loaded. If you did not install the packages in order suggested by this post, then you may have lost that piece of configuration. In any case, make sure a line similar to the following is present in /usr/local/etc/apache24/httpd.conf: LoadModule php5_module        libexec/apache24/libphp5.so Installing drush drush is an optional package offering an amazingly good command line interface to perform many Drupal-related management tasks: drush could even be used to install Drupal, but this topic is not covered by this post, since I prefer relying on a port tested specifically on FreeBSD. However, once you have tested that a specific release works correctly, you may find drush very useful to streamline your installations. If you are interested in using drush, you will find plenty of good information on the Internet. To install drush, the following command may be used: # pkg install drush Creating a Database for Drupal A database for Drupal must be created in the DB server installed in the previous section. The MySQL port sets no password for the root user connecting from localhost; for this reason, setting its password is recommended (in bold the text input by the user): # mysql -u root Welcome to the MySQL monitor. Commands end with ; or \g. [...snip...] mysql> set password for root@localhost=PASSWORD('your-password') mysql> exit Bye Once the password is set, you can try to reconnect (in italic bold the variables whose name should be changed with your values of choice): # mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with ; or \g. [...snip...] mysql> create database drupal_database_name; mysql> create user 'drupal_user'@'localhost' identified by 'password'; mysql> grant all privileges on drupal_database_name.* to 'drupal_user'@'localhost' with grant option; mysql> flush privileges; Configuring the Apache HTTP Server Now that everything is in place, we can configure the web server so that it starts serving the Drupal web application. The tasks to perform are the followingConfiguring the required modules. Configuring a virtual host to serve Drupal. Configuring a MIME/type for PHP.The modules required to run Drupal are mod_rewrite and the PHP module. The latter was configured automatically by the PHP module port, and the latter can be configured uncommenting the following line from /usr/local/etc/apache24/httpd.conf: LoadModule rewrite_module libexec/apache24/mod_rewrite.so The cleanest way to segregate the Drupal configuration is creating a virtual host for it. An additional advantage of this approach is that Drupal will be served from the root path (/) and you won’t need to use any rewrite rule to achieve the same result. Assuming the host name and the port where Drupal will be published is drupal.host.name:80, then create a file in /usr/local/etc/apache24/Includes named drupal.conf and define the skeleton of the virtual host: <VirtualHost *:80> ServerName drupal.host.name# Drupal directory configuration placeholder ErrorLog ${APACHE_LOG_DIR}/drupal-error.log LogLevel warn CustomLog ${APACHE_LOG_DIR}/drupal-access.log combined </VirtualHost> In the default configuration of Apache in FreeBSD, any .conf file in this directory is included automatically, so that no additional code is required to add the virtual host to the web server configuration. In this fragment I’ve used an environment variable (${APACHE_LOG_DIR}) to separate some server configuration variables that could be reused in external scripts. To define environment variables, a .env file must be created in /usr/local/etc/apache24/envvars.d such as: The Drupal directory fragment defines the DocumentRoot of the virtual host and some of the required options: DocumentRoot /usr/local/www/drupal7 <Directory "/usr/local/www/drupal7"> Options Indexes FollowSymLinks AllowOverride All Require all granted </Directory>The option AllowOverride set to All is required so that .htaccess files shipped with Drupal are taken into account by the Apache HTTP Server. In this fragment, the path of the Drupal installation directory of the FreeBSD port is used. If you installed Drupal using alternative methods (such as drush), update the path accordingly. The complete virtual host configuration file is:ServerName drupal.host.nameDocumentRoot /usr/local/www/drupal7 Options Indexes FollowSymLinks AllowOverride All Require all grantedErrorLog ${APACHE_LOG_DIR}/drupal-error.log LogLevel warn CustomLog ${APACHE_LOG_DIR}/drupal-access.log combinedFinally, the Apache HTTP Server must be instructed to execute the PHP code contained in PHP pages and to do so we need to add a MIME/type for them adding the following line in httpd.conf: <IfModule mime_module># Content has been trimmed# Add MIME type for PHP AddType application/x-httpd-php .php</IfModule> Once all the settings are in place, Apache can be restarted and you can point your browser to http://drupal.host.name/ where the Drupal installation wizard will welcome you and will require you to input the database configuration and other Drupal website settings. To restart the Apache HTTP Server, the following command can be used: # service apache24 restart Configuring Drupal behind a Proxy Server Machines connecting to enterprise network often are not connected directly to the Internet but require the use of a web proxy server instead. Drupal can be configured to use a web proxy server by setting the following variables in ${DRUPAL_HOME}/sites/default/settings.php. If this file does not exist, copy the file default.settings.php (shipped with Drupal) into settings.php. The configuration variable that enable proxy support are the following: /** * External access proxy settings: * * If your site must access the Internet via a web proxy then you can enter * the proxy settings here. Currently only basic authentication is supported * by using the username and password variables. The proxy_user_agent variable * can be set to NULL for proxies that require no User-Agent header or to a * non-empty string for proxies that limit requests to a specific agent. The * proxy_exceptions variable is an array of host names to be accessed directly, * not via proxy. */ $conf['proxy_server'] = 'web-proxy'; $conf['proxy_port'] = 3128; # $conf['proxy_username'] = ''; # $conf['proxy_password'] = ''; # $conf['proxy_user_agent'] = ''; # $conf['proxy_exceptions'] = array('', 'localhost'); Depending on your proxy settings, different values may be used. Beware that although Drupal itself (Core) supports a proxy, many third-party modules still do not. One notable exception at the time of writing is the reCaptcha module which will not work without a direct Internet connection. Setting Up Clean URLs Last but not least, clean URLs support may be enabled. Drupal performs a sanity check and will not allow you to enable the Clean URLs feature if the test does not pass. However, I have found plenty of false negatives when running Drupal 7 on FreeBSD: if the Clean URL test fails in your installation, try checking if clean URLs are working and use the workaround described in the official Drupal documentation and forcibly enable Clean URLs.Reference: Installing Drupal on FreeBSD from our JCG partner Enrico Crisostomo at the The Grey Blog blog....

Spring Boot / Java 8 / Tomcat 8 on Openshift with DIY

DIY cartridge is an experimental cartridge that provides a way to test unsupported languages on OpenShift. It provides a minimal, free-form scaffolding which leaves all details of the cartridge to the application developer. This blog post illustrates the use of Spring Boot / Java 8 / Tomcat 8 application with PostgreSQL service bound to it. Creating new application Prerequisite Before we can start building the application, we need to have an OpenShift free account and client tools installed. Step 1: Create DIY application To create an application using client tools, type the following command: rhc app create boot diy-0.1 This command creates an application boot using DIY cartridge and clones the repository to boot directory. Step 2: Add PostgreSQL cartridge to application The application we are creating will use PostgreSQL database, hence we need to add appropriate cartridge to the application: rhc cartridge add postgresql-9.2 --app boot After creating the cartridge, it is possible to check its status with the following command: rhc cartridge status postgresql-9.2 --app boot Step 3: Delete Template Application Source code OpenShift creates a template project that can be freely removed: git rm -rf .openshift README.md diy misc Commit the changes: git commit -am "Removed template application source code" Step 4: Pull Source code from GitHub git remote add upstream https://github.com/kolorobot/openshift-diy-spring-boot-sample.git git pull -s recursive -X theirs upstream master Step 5: Push changes The basic template is ready to be pushed: git push The initial deployment (build and application startup) will take some time (up to several minutes). Subsequent deployments are a bit faster, although starting Spring Boot application may take even more than 2 minutes on small Gear: Tomcat started on port(s): 8080/http Started Application in 125.511 seconds You can now browse to: http://boot-yournamespace.rhcloud.com/manage/health and you should see: { "status": "UP", "database": "PostgreSQL", "hello": 1 } You can also browser the API. To find out what options you have, navigate to the root of the application. You should see the resource root with links to available resources: { "_links" : { "person" : { "href" : "http://boot-yournamespace.rhcloud.com/people{?page,size,sort}", "templated" : true } } } Navigating to http://boot-yournamespace.rhcloud.com/people should return all people from the database. Step 6: Adding Jenkins Using Jenkins has some advantages. One of them is that the build takes place in it’s own Gear. To build with Jenkins, OpenShift needs a server and a Jenkins client cartridge attached to the application. Creating Jenkins application: rhc app create ci jenkins And attaching Jenkins client to the application: rhc cartridge add jenkins-client --app boot You can now browse to: http://ci-.rhcloud.com and login with the credentials provided. When you make next changes and push them, the build will be triggered by Jenkins: remote: Executing Jenkins build. remote: remote: You can track your build at https://ci-<namespace>.rhcloud.com/job/boot-build remote: remote: Waiting for build to schedule......... And when you observe the build result, the application starts a bit faster on Jenkins. Under the hood Why DIY? Spring Boot application can be deployed to Tomcat cartridge on OpenShift. But at this moment no Tomcat 8 and Java 8 support exists, therefore DIY was selected. DIY has limitations: it cannot be scaled for example. But it is perfect for trying and playing with new things. Application structure The application is a regular Spring Boot application, that one can bootstrapped with http://start.spring.io. Build system used is Maven, packaging type is Jar. Tomcat 8 with Java 8 used. Spring Boot uses Tomcat 7 by default, to change it the following property was added: <properties> <tomcat.version>8.0.9</tomcat.version> </properties> The Maven was selected, since currently only Gradle 1.6 can be used on OpenShift. This is due to a bug in Gradle. Gradle 2.2 fixes this issue. Maven settings.xml The settings.xml file is pretty important, as it contains the location of Maven repository: ${OPENSHIFT_DATA_DIR}/m2/repository. On OpenShift, write permissions are only in $OPENSHIFT_DATA_DIR. Data source configuration The application uses Spring Data REST to export repositories over REST. The required dependencies are:spring-boot-starter-data-jpa – repositories configuration spring-boot-starter-data-rest – exposing repositoties over REST hsqldb – for embedded database support postgresql – for PostgreSQL support. Since currently OpenShift uses PostgreSQL 9.2, the appropriate driver’s version is usedCommon properties – application.properties By default (default profile, src/main/resources/application.properties), the application will use embedded HSQLDB and populate it with the src/main/resources/data.sql. The data file will work on both HSQLDB and PostrgeSQL, so we don’t need to provide platform specific files (which is possible with Spring Boot).spring.datasource.initialize = true must be used, so Spring Boot picks up the data file and loads it to the database. spring.jpa.generate-ddl = true makes sure that the schema will be exported.OpenShift properties – application-openshift.properties OpenShift specific configuration (src/main/resources/application-openshift.properties) allows the use of PostgreSQL service. The configuration uses OpenShift env variables to setup the connection properties:$OPENSHIFT_POSTGRESQL_DB_HOST – for the database host $OPENSHIFT_POSTGRESQL_DB_PORT – for the database port $OPENSHIFT_APP_NAME – for the database name $OPENSHIFT_POSTGRESQL_DB_USERNAME – for the database username $OPENSHIFT_POSTGRESQL_DB_PASSWORD – for the database passwordSpring allows to use env variables directly in properties with ${} syntax, e.g.: spring.datasource.username = ${OPENSHIFT_POSTGRESQL_DB_USERNAME} To let Spring Boot activate OpenShift profile, the spring.profiles.active property is passed to the application at startup: java -jar <name>.jar --spring.profiles.active=openshift. Logging on OpenShift The logging file will be stored in $OPENSHIFT_DATA_DIR: logging.file=${OPENSHIFT_DATA_DIR}/logs/app.log Actuator Actuator default management context path is /. This is changed to /manage, because OpenShift exposes /health endpoint itself that covers Actuator’s /health endpoint . management.context-path=/manage OpenShift action_hooks OpenShift executes action hooks script files at specific points during the deployment process. All hooks are placed in the .openshift/action_hooks directory in the application repository. Files must have be executable. In Windows, in Git Bash, the following command can be used: git update-index --chmod=+x .openshift/action_hooks/* Deploying the application The deploy script downloads Java and Maven, creates some directories and exports couple of environment variables required to properly run Java 8 / Maven build. The final command of the deployment is to run Maven goals: mvn -s settings.xml clean install Starting the application When deploy script finishes successfully, the target directory will contain a single jar with the Spring Boot application assembled. The application is started and bound to the server address and port provided by OpenShift. In addition, the profile name is provided, so a valid data source will be created. The final command that runs the application: nohup java -Xms384m -Xmx412m -jar target/*.jar --server.port=${OPENSHIFT_DIY_PORT} --server.address=${OPENSHIFT_DIY_IP} --spring.profiles.active=openshift & Stopping the application The stop script is looking for a Java process and when it finds it… you know what happens. Summary I am pretty happy with the evaluation of OpenShift with Do It Yourself cartridge. Not everything went smooth as I expected, mostly due to memory limitations on small Gear. I spent some time to figure it out and have proper configuration. But still, OpenShift with DIY is worth trying and playing with for a short while. Especially, that to get started is completely for free. ReferencesThe project source code, used throughout this article, can be found on GitHub: https://github.com/kolorobot/openshift-diy-spring-boot-sample. Spring Boot documentation: http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#cloud-deployment-openshift Some OpenShift references used while creating this article: https://blog.openshift.com/run-gradle-builds-on-openshift https://blog.openshift.com/tips-for-creating-openshift-apps-with-windowsReference: Spring Boot / Java 8 / Tomcat 8 on Openshift with DIY from our JCG partner Rafal Borowiec at the Codeleak.pl blog....

Let’s Stream a Map in Java 8 with jOOλ

I wanted to find an easy way to stream a Map in Java 8. Guess what? There isn’t! What I would’ve expected for convenience is the following method:                   public interface Map<K, V> {default Stream<Entry<K, V>> stream() { return entrySet().stream(); } } But there’s no such method. There are probably a variety of reasons why such a method shouldn’t exist, e.g.:There’s no “clear” preference for entrySet() being chosen over keySet() or values(), as a stream source Map isn’t really a collection. It’s not even an Iterable That wasn’t the design goal The EG didn’t have enough timeWell, there is a very compelling reason for Map to have been retrofitted to provide both an entrySet().stream() and to finally implement Iterable<Entry<K, V>>. And that reason is the fact that we now have Map.forEach(): default void forEach( BiConsumer<? super K, ? super V> action) { Objects.requireNonNull(action); for (Map.Entry<K, V> entry : entrySet()) { K k; V v; try { k = entry.getKey(); v = entry.getValue(); } catch(IllegalStateException ise) { // this usually means the entry is no longer in the map. throw new ConcurrentModificationException(ise); } action.accept(k, v); } } forEach() in this case accepts a BiConsumer that really consumes entries in the map. If you search through JDK source code, there are really very few references to the BiConsumer type outside of Map.forEach() and perhaps a couple of CompletableFuture methods and a couple of streams collection methods. So, one could almost assume that BiConsumer was strongly driven by the needs of this forEach() method, which would be a strong case for making Map.Entry a more important type throughout the collections API (we would have preferred the type Tuple2, of course). Let’s continue this line of thought. There is also Iterable.forEach(): public interface Iterable<T> { default void forEach(Consumer<? super T> action) { Objects.requireNonNull(action); for (T t : this) { action.accept(t); } } } Both Map.forEach() and Iterable.forEach() intuitively iterate the “entries” of their respective collection model, although there is a subtle difference:Iterable.forEach() expects a Consumer taking a single value Map.forEach() expects a BiConsumer taking two values: the key and the value (NOT a Map.Entry!)Think about it this way:This makes the two methods incompatible in a “duck typing sense”, which makes the two types even more differentBummer! Improving Map with jOOλ We find that quirky and counter-intuitive. forEach() is really not the only use-case of Map traversal and transformation. We’d love to have a Stream<Entry<K, V>>, or even better, a Stream<Tuple2<T1, T2>>. So we implemented that in jOOλ, a library which we’ve developed for our integration tests at jOOQ. With jOOλ, you can now wrap a Map in a Seq type (“Seq” for sequential stream, a stream with many more functional features): Map<Integer, String> map = new LinkedHashMap<>(); map.put(1, "a"); map.put(2, "b"); map.put(3, "c");assertEquals( Arrays.asList( tuple(1, "a"), tuple(2, "b"), tuple(3, "c") ),Seq.seq(map).toList() ); What you can do with it? How about creating a new Map, swapping keys and values in one go: System.out.println( Seq.seq(map) .map(Tuple2::swap) .toMap(Tuple2::v1, Tuple2::v2) );System.out.println( Seq.seq(map) .toMap(Tuple2::v2, Tuple2::v1) ); Both of the above will yield: {a=1, b=2, c=3} Just for the record, here’s how to swap keys and values with standard JDK API: System.out.println( map.entrySet() .stream() .collect(Collectors.toMap( Map.Entry::getValue, Map.Entry::getKey )) ); It can be done, but the every day verbosity of standard Java API makes things a bit hard to read / write.Reference: Let’s Stream a Map in Java 8 with jOOλ from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog....

Stop Claiming that you’re Using a Schemaless Database

One of MongoDB’s arguments when evangelising MongoDB is the fact that MongoDB is a “schemaless” database: Why Schemaless? MongoDB is a JSON-style data store. The documents stored in the database can have varying sets of fields, with different types for each field.     And that’s true. But it doesn’t mean that there is no schema. There are in fact various schemas:The one in your head when you designed the data structures The one that your database really implemented to store your data structures The one you should have implemented to fulfill your requirementsEvery time you realise that you made a mistake (see point three above), or when your requirements change, you will need to migrate your data. Let’s review again MongoDB’s point of view here: With a schemaless database, 90% of the time adjustments to the database become transparent and automatic. For example, if we wish to add GPA to the student objects, we add the attribute, resave, and all is well — if we look up an existing student and reference GPA, we just get back null. Further, if we roll back our code, the new GPA fields in the existing objects are unlikely to cause problems if our code was well written. Everything above is true as well. “Schema-less” vs. “Schema-ful” But let’s translate this to SQL (or use any other “schema-ful” database instead): ALTER TABLE student ADD gpa VARCHAR(10); And we’re done! Gee, we’ve added a column, and we’ve added it to ALL rows. It was transparent. It was automatic. We “just get back null” on existing students. And we can even “roll back our code”: ALTER TABLE student DROP gpa; Not only are the existing objects unlikely to cause problems, we have actually rolled back our code AND database. Let’s summarise:We can do exactly the same in “schema-less” databases as we can in “schema-ful” ones We guarantee that a migration takes place (and it’s instant, too) We guarantee data integrity when we roll back the changeWhat about more real-world DDL? Of course, at the beginning of projects, when they still resemble the typical cat/dog/pet-shop, book/author/library sample application, we’ll just be adding columns. But what happens if we need to change the student-teacher 1:N relationship into a student-teacher M:N relationship? Suddenly, everything changes, and not only will the relational data model prove far superior to a hierarchical one that just yields tons of data duplication, it’ll also be moderately easy to migrate, and the outcome is guaranteed to be correct and tidy! CREATE TABLE student_to_teacher AS SELECT id AS student_id, teacher_id FROM student;ALTER TABLE student DROP teacher_id; … and we’re done! (of course, we’d be adding constraints and indexes) Think about the tedious task that you’ll have transforming your JSON to the new JSON. You don’t even have XSLT or XQuery for the task, only JavaScript! Let’s face the truth Schemalessness is about a misleading term as much as NoSQL is: History of NoSQL according to @markmadsen #strataconf pic.twitter.com/XHXMJsXHjV — Edd Dumbill (@edd) November 12, 2013And again, MongoDB’s blog post is telling the truth (and an interesting one, too): Generally, there is a direct analogy between this “schemaless” style and dynamically typed languages. Constructs such as those above are easy to represent in PHP, Python and Ruby. What we are trying to do here is make this mapping to the database natural. When you say “schemaless”, you actually say “dynamically typed schema” – as opposed to statically typed schemas as they are available from SQL databases. JSON is still a completely schema free data structure standard, as opposed to XML which allows you to specify XSD if you need, or operate on document-oriented, “schema-less” (i.e. dynamically typed) schemas. (And don’t say there’s json-schema. That’s a ridiculous attempt to mimick XSD) This is important to understand! You always have a schema, even if you don’t statically type it. If you’re writing JavaScript, you still have types, which you have to be fully aware of in your mental model of the code. Except that there’s no compiler (or IDE) that can help you infer the types with 100% certainty. An example: LOL JavaScript: > null * 1 0 > null == 0 false pic.twitter.com/Hc2NR2tsMP — Lukas Eder (@lukaseder) October 15, 2014… and more:So, there’s absolutely nothing that is really easier with “schemaless” databases than with “schemaful” ones. You just defer the inevitable work of sanitising your schema to some other later time, a time when you might care more than today, or a time when you’re lucky enough to have a new job and someone else does the work for you. You might have believed MongoDB, when they said that “objects are unlikely to cause problems”. But let me tell you the ugly truth: Anything that can possibly go wrong, does – Murphy We wish you good luck with your dynamically typed languages and your dynamically typed database schemas – while we’ll stick with type safe SQL.Reference: Stop Claiming that you’re Using a Schemaless Database from our JCG partner Lukas Eder at the JAVA, SQL, AND JOOQ blog....

ConEmu – Windows console emulator with tabs

After switching to Git some time ago, I started working more and more with Git Bash on Windows. Git Bash is pretty cool as it provides (apart from Git) Bash supported with basic Unix tools including curl or ssh. Git Bash in Windows has some limitation though including limited customization options and lack of good copy & paste options supported with keyboard shortcuts. Fortunately, there is ConEmu that not only fills that gap but adds various features that make working with console applications more productive and more enjoyable for me.         Introduction ConEmu is a Windows console emulator with tabs, which presents multiple consoles and simple GUI applications as one customizable GUI window with various features. And not only working with Git Bash is far better with ConEmu, but with other tools I use too:Far Manager – a program for managing files and archives in Windows – handy Notepad++ – source code editor and Notepad replacement – naturally! cmd (Windows command prompt) – I still use it, rarely but stillPractically, running any tool should not be a problem. Let’s say I want to run my favorite password manager in ConEmu I can execute the following command: $ <KeePassHome>/keepass.exe -new_console -new_console switch instructs ConEmu to start an application in a new console. Working with tabs Controlling tabs I disabled most confirmation on tabs actions (Settings > Main > Confirm), so now I can fully control tab creation, closing and switching between them with shortcuts without additional confirmations. Most commonly used shortcuts for working with tabs:Win + N – show New Console Dialog (e.g. for running tasks with no shortcuts assigned) Win + X – new cmd console Win + Delete – for closing active tab Win + <Num> – switch between tabs (alternative Ctrl+Tab)Split Screen ConEmu may split any tab into several panes:The most common shortcuts to work with Split Screen:Win+N – show New Console Dialog and select Split Screen options Ctrl+Shift+O – duplicate shell from active pane and split horizontally Ctrl+Shift+E – duplicate shell from active pane and split verticallyYou navigate between screens in Split Screen mode just like you navigate between tabs. Tasks Git Bash is my favorite shell on Windows, therefore I made it a startup task in ConEmu:In addition I added Far Manager and Notepad++ tasks and I associated hot keys for them:Win+B,F,P – Git Bash, FAR and Notepad++.Even if you choose shortcuts that are used by Windows, ConEmu will intercept them (once active). Working with text Highlighting, copying & pasting with mouse and keyboard shortcuts makes it really convenient. This is one of the features I appreciate most in ConEmu.Shortcuts:copying current selection with Ctrl + C pasting with Shift+Insert, Ctrl+V (only first line) or with right mouse click, selecting text Shift+Arrow Keys/Home/End or with right click and dragAdditionally, scrolling buffer is also easy withCtrl+Up/Down/PgUp/PgDown shortcuts. Notepad++ Notepad++ is one of my favorite editors for Windows. ConEmu can run Notepad++ in a tab with no problem. I created a task for Notepad++ so I can start it in a new tab whenever I want. In addition, I made possible to run it in console with the file loaded that is passed as argument. This is very easy with Git Bash. Make sure Notepad++ is in the PATH and create an alias: alias edit="notepad++ -new_console" Now, edit filename will run Notepad++ with filename loaded in a new tab.In case you want this alias to be always available, create .bashrc file in your home directory (if does not exist) and add the alias so it is automatically loaded on Git Bash startup. Summary I’ve been using ConEmu for several weeks now and I am far from knowing everything about it, but I already can’t imagine my Windows without it! With ConEmu I can use my favorite tools like Git Bash, cmd, Far Manager, and Notepad++ in one application with great tabbing experience supported with shortcuts. Font anti-aliasing, transparency (can be configured for active and inactive window separately), full screen, split screen and great mark, highlight, copy & paste options makes ConEmu is a complete application and a great choice for developers looking for improved productivity in Windows. I truly recommend ConEmu to every professional! References Project home page: https://code.google.com/p/conemu-maximus5Reference: ConEmu – Windows console emulator with tabs from our JCG partner Rafal Borowiec at the Codeleak.pl blog....

Neo4j: Cypher – Avoiding the Eager

  Although I love how easy Cypher’s LOAD CSV command makes it to get data into Neo4j, it currently breaks the rule of least surprise in the way it eagerly loads in all rows for some queries even those using periodic commit. This is something that my colleague Michael noted in the second of his blog posts explaining how to use LOAD CSV successfully:  The biggest issue that people ran into, even when following the advice I gave earlier, was that for large imports of more than one million rows, Cypher ran into an out-of-memory situation. That was not related to commit sizes, so it happened even with PERIODIC COMMIT of small batches.I recently spent a few days importing data into Neo4j on a Windows machine with 4GB RAM so I was seeing this problem even earlier than Michael suggested. Michael explains how to work out whether your query is suffering from unexpected eager evaluation:If you profile that query you see that there is an “Eager” step in the query plan. That is where the “pull in all data” happens.You can profile queries by prefixing the word ‘PROFILE’. You’ll need to run your query in the console of /webadmin in your web browser or with the Neo4j shell. I did this for my queries and was able to identify query patterns which get evaluated eagerly and in some cases we can work around it. We’ll use the Northwind data set to demonstrate how the Eager pipe can sneak into our queries but keep in mind that this data set is sufficiently small to not cause issues. This is what a row in the file looks like: $ head -n 2 data/customerDb.csv OrderID,CustomerID,EmployeeID,OrderDate,RequiredDate,ShippedDate,ShipVia,Freight,ShipName,ShipAddress,ShipCity,ShipRegion,ShipPostalCode,ShipCountry,CustomerID,CustomerCompanyName,ContactName,ContactTitle,Address,City,Region,PostalCode,Country,Phone,Fax,EmployeeID,LastName,FirstName,Title,TitleOfCourtesy,BirthDate,HireDate,Address,City,Region,PostalCode,Country,HomePhone,Extension,Photo,Notes,ReportsTo,PhotoPath,OrderID,ProductID,UnitPrice,Quantity,Discount,ProductID,ProductName,SupplierID,CategoryID,QuantityPerUnit,UnitPrice,UnitsInStock,UnitsOnOrder,ReorderLevel,Discontinued,SupplierID,SupplierCompanyName,ContactName,ContactTitle,Address,City,Region,PostalCode,Country,Phone,Fax,HomePage,CategoryID,CategoryName,Description,Picture 10248,VINET,5,1996-07-04,1996-08-01,1996-07-16,3,32.38,Vins et alcools Chevalier,59 rue de l'Abbaye,Reims,,51100,France,VINET,Vins et alcools Chevalier,Paul Henriot,Accounting Manager,59 rue de l'Abbaye,Reims,,51100,France,,,5,Buchanan,Steven,Sales Manager,Mr.,1955-03-04,1993-10-17,14 Garrett Hill,London,,SW1 8JR,UK,(71) 555-4848,3453,\x,"Steven Buchanan graduated from St. Andrews University, Scotland, with a BSC degree in 1976. Upon joining the company as a sales representative in 1992, he spent 6 months in an orientation program at the Seattle office and then returned to his permanent post in London. He was promoted to sales manager in March 1993. Mr. Buchanan has completed the courses ""Successful Telemarketing"" and ""International Sales Management."" He is fluent in French.",2,http://accweb/emmployees/buchanan.bmp,10248,11,14,12,0,11,Queso Cabrales,5,4,1 kg pkg.,21,22,30,30,0,5,Cooperativa de Quesos 'Las Cabras',Antonio del Valle Saavedra,Export Administrator,Calle del Rosal 4,Oviedo,Asturias,33007,Spain,(98) 598 76 54,,,4,Dairy Products,Cheeses,\x MERGE, MERGE, MERGE The first thing we want to do is create a node for each employee and each order and then create a relationship between them. We might start with the following query: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) This does the job but if we profile the query like so… PROFILE LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row WITH row LIMIT 0 MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) …we’ll notice an ‘Eager’ lurking on the third line: ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | employee, order, UNNAMED216 | MergePattern | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | employee, employee, order, order | MergeNode; :Employee; MergeNode; :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+----------------------------------+-----------------------------------------+ You’ll notice that when we profile each query we’re stripping off the periodic commit section and adding a ‘WITH row LIMIT 0′. This allows us to generate enough of the query plan to identify the ‘Eager’ operator without actually importing any data. We want to split that query into two so it can be processed in a non eager manner: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row WITH row LIMIT 0 MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, employee, order, order | MergeNode; :Employee; MergeNode; :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+----------------------------------+-----------------------------------------+ Now that we’ve created the employees and orders we can join them together: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, order, UNNAMED216 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(0) | 0 | 0 | order, order | :Order | ==> | Filter(1) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(1) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ Not an Eager in sight! MATCH, MATCH, MATCH, MERGE, MERGE If we fast forward a few steps we may now have refactored our import script to the point where we create our nodes in one query and the relationships in another query. Our create query works as expected: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (employee:Employee {employeeId: row.EmployeeID}) MERGE (order:Order {orderId: row.OrderID}) MERGE (product:Product {productId: row.ProductID}) ==> +-------------+------+--------+----------------------------------------------------+--------------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+----------------------------------------------------+--------------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, employee, order, order, product, product | MergeNode; :Employee; MergeNode; :Order; MergeNode; :Product | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+----------------------------------------------------+------------------------------------------------------------ We’ve now got employees, products and orders in the graph. Now let’s create relationships between the trio: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MATCH (product:Product {productId: row.ProductID}) MERGE (employee)-[:SOLD]->(order) MERGE (order)-[:PRODUCT]->(product) If we profile that we’ll notice Eager has sneaked in again! ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | order, product, UNNAMED318 | MergePattern | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | employee, order, UNNAMED287 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(product,productId) == Property(row,ProductID) | ==> | NodeByLabel(0) | 0 | 0 | product, product | :Product | ==> | Filter(1) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(1) | 0 | 0 | order, order | :Order | ==> | Filter(2) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(2) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ In this case the Eager happens on our second call to MERGE as Michael identified in his post:The issue is that within a single Cypher statement you have to isolate changes that affect matches further on, e.g. when you CREATE nodes with a label that are suddenly matched by a later MATCH or MERGE operation.We can work around the problem in this case by having separate queries to create the relationships: LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (employee:Employee {employeeId: row.EmployeeID}) MATCH (order:Order {orderId: row.OrderID}) MERGE (employee)-[:SOLD]->(order) ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | employee, order, UNNAMED236 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(0) | 0 | 0 | order, order | :Order | ==> | Filter(1) | 0 | 0 | | Property(employee,employeeId) == Property(row,EmployeeID) | ==> | NodeByLabel(1) | 0 | 0 | employee, employee | :Employee | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+-------------------------------+-----------------------------------------------------------+ USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MATCH (order:Order {orderId: row.OrderID}) MATCH (product:Product {productId: row.ProductID}) MERGE (order)-[:PRODUCT]->(product) ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | order, product, UNNAMED229 | MergePattern | ==> | Filter(0) | 0 | 0 | | Property(product,productId) == Property(row,ProductID) | ==> | NodeByLabel(0) | 0 | 0 | product, product | :Product | ==> | Filter(1) | 0 | 0 | | Property(order,orderId) == Property(row,OrderID) | ==> | NodeByLabel(1) | 0 | 0 | order, order | :Order | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+------------------------------+--------------------------------------------------------+ MERGE, SET I try to make LOAD CSV scripts as idempotent as possible so that if we add more rows or columns of data to our CSV we can rerun the query without having to recreate everything. This can lead you towards the following pattern where we’re creating suppliers: USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (supplier:Supplier {supplierId: row.SupplierID}) SET supplier.companyName = row.SupplierCompanyName We want to ensure that there’s only one Supplier with that SupplierID but we might be incrementally adding new properties and decide to just replace everything by using the ‘SET’ command. If we profile that query, the Eager lurks: ==> +----------------+------+--------+--------------------+----------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +----------------+------+--------+--------------------+----------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph(0) | 0 | 0 | | PropertySet | ==> | Eager | 0 | 0 | | | ==> | UpdateGraph(1) | 0 | 0 | supplier, supplier | MergeNode; :Supplier | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +----------------+------+--------+--------------------+----------------------+ We can work around this at the cost of a bit of duplication using ‘ON CREATE SET’ and ‘ON MATCH SET': USING PERIODIC COMMIT 1000 LOAD CSV WITH HEADERS FROM "file:/Users/markneedham/projects/neo4j-northwind/data/customerDb.csv" AS row MERGE (supplier:Supplier {supplierId: row.SupplierID}) ON CREATE SET supplier.companyName = row.SupplierCompanyName ON MATCH SET supplier.companyName = row.SupplierCompanyName ==> +-------------+------+--------+--------------------+----------------------+ ==> | Operator | Rows | DbHits | Identifiers | Other | ==> +-------------+------+--------+--------------------+----------------------+ ==> | EmptyResult | 0 | 0 | | | ==> | UpdateGraph | 0 | 0 | supplier, supplier | MergeNode; :Supplier | ==> | Slice | 0 | 0 | | { AUTOINT0} | ==> | LoadCSV | 1 | 0 | row | | ==> +-------------+------+--------+--------------------+----------------------+ With the data set I’ve been working with I was able to avoid OutOfMemory exceptions in some cases and reduce the amount of time it took to run the query by a factor of 3 in others. As time goes on I expect all of these scenarios will be addressed but as of Neo4j 2.1.5 these are the patterns that I’ve identified as being overly eager. If you know of any others do let me know and I can add them to the post or write a second part.Reference: Neo4j: Cypher – Avoiding the Eager from our JCG partner Mark Needham at the Mark Needham Blog blog....

Akka Notes – Child Actors and ActorPath – 6

Actors are completely hierarchical. Whatever Actors that you create HAS to be a child of some other Actor. Let’s analyze that a bit : Path Say, we create an ActorRef using ActorSystem.actorOf and try to print it’s path.       val actorSystem=ActorSystem("SupervisionActorSystem") val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor]) println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/$a As you see, a path looks very similar to a file path in a file system.akka here is fixed because all these are addresses of Akka Actors – more like file:// or http:// prefix (nothing to do with protocol though). The SupervisionActorSystem is just the name of your ActorSystem that you created. We’ll talk about the user in the next section. The $a is the name of your Actor that the system generated for you. How would you like if your operating system generated random file names for your files? You’d obviously hate it because you would want to refer to that name in future. So, let’s give it a proper meaningful name to it :val actorRef=actorSystem.actorOf(Props[BasicLifecycleLoggingTeacherActor], "teacherActor") println (actorRef.path) // (prints) akka://SupervisionActorSystem/user/teacherActor That’s it. Now, the path actually makes sense.Child Actors Similar to top level actors which we create out of ActorSystem, we could also create child actors out of the ActorContext. In fact, the power of Actors’s fault tolerance primarily lie within leveraging the Actor hierarchy and the ability of a parent to manage the life of child actors. Assume you have a TeacherSupervisor and you would like to create a TeacherActor to be a child of the Supervisor, you do a ActorContext.actorOf instead of a ActorSystem.actorOf: class TeacherSupervisor extends Actor with ActorLogging { val teacherActor=context.actorOf(Props[TeacherActor], "teacherActor") ... ... Frankly, in any application, you will be creating a whole lot of child actors than top level actors – which means that you’ll be calling a lot more actorContext.actorOf than actorSystem.actorOf.You’ll notice that the path of the child actor is akka://SupervisionActorSystem/user/teacherSupervisor/teacherActor, which is very similar to the path you get when you create a child folder within a parent folder. When do you create child Actors? You generally create child actors when a particular task is composed of a subtask or multiple subtasks. You also create a child actor when a particular task to be executed by the parent is error-prone and you would want to isolate it (so that if it fails, you could recover). When there is no parent child relationship between tasks, then you DON’T create child actors. Also, there’s nothing which is stopping a child actor from creating children to delegate its subtasks. Actors and their creation are really cheap but the power that comes with it is amazing (we’ll see about this while we talk about supervision). Now what is that user in the path? For lack of a creativity, let me compare the ActorSystem to a Unix file system – with a / root folder and all those /etc, /usr, /bin and various other folders. ActorSystem are much like that. It creates a few top level Actors – the most important being the root Actor with the path /, the user Actor with the path /user and a system Actor with the path /system. (there’s also a /deadLetters that represent the DeadLetterActorRef. We saw this in our previous post) Codewise, the ActorSystem composes three Actors inside it (via ActorRefProvider). Those are the root for ALL the Actors that gets created under the ActorSystem.systemGuardian actor – root of all actors under /system guardian actor – root of all actors under /user and rootGuardian Actor – root of both the systemGuardian and the userGuardian actors./** * Reference to the supervisor of guardian and systemGuardian; .... */ def rootGuardian: InternalActorRef/** * Reference to the supervisor used for all top-level user actors. */ def guardian: LocalActorRef/** * Reference to the supervisor used for all top-level system actors. */ def systemGuardian: LocalActorRef/user (aka) user guardian Any Actor that you create in your program like the StudentActor or the TeacherActor using the ActorSystem‘s actorOf method would directly fall under /user. That’s the reason your teacherActor in the first part of this write-up had the path slug /user/teacherActor. /system (aka) system guardian The system guardian shuts itself down when it notices that the userGuardian is dead. Makes sense considering if the userGuardian is down, then all the business actors under it is also down and therefore all administrative actors needs to be gone too. We could see two distinct places where System Actors are being created – I mean actors under the /system hierarchy.Like we saw earlier, any message that you send to an Actor that is terminated gets forwarded to the mailbox of an internal Actor called DeadLetterActor. The DeadLetter Actor wraps each message as a DeadLetter and publishes it to the EventStream. One other Actor called DeadLetterListener consumes all DeadLetters and publishes that as a log message. Now, the DeadLetterListener is a system Actor with path /system/deadLetterListener. Remember the TestEventListener that we created in our previous write-up to subscribe to log messages in the EventStream? They are System actors too. In fact, all akka.loggers are created as System actors.class TeacherTest extends TestKit(ActorSystem("UniversityMessageSystem", ConfigFactory.parseString("""akka.loggers = ["akka.testkit.TestEventListener"]"""))) ... ... The documentation here says that any Actor that is configured in the configuration files and created and deployed into the ActorSystem while it starts, falls under the /system umbrella. Let me update this post when I find something interesting around this / (aka) root guardian As we saw earlier, the / Actor is the parent of the user and the system guardians. TRIVIA Technically, there’s a superfluous parent for the root actor too. This Actor’s only job is to shut down the entire ActorSystem if the root actor fails. Since it’s strictly not considered within the Actor hierarchy, the Akka team calls it : private[akka] val theOneWhoWalksTheBubblesOfSpaceTime: InternalActorRef = new MinimalActorRef { ...Reference: Akka Notes – Child Actors and ActorPath – 6 from our JCG partner Arun Manivannan at the Rerun.me blog....

Why to use String

Recently I was tutoring juniors during a training session. One of the task was to write a class that can dwarwle maps based on some string key. The result one of the juniors created contained the following method:                 void dwarwle(HashMap<String,Dwarwable> mapToDwarwle, String dwarwleKey){ for( final Entry<String, Dwarwable> entry : mapToDwarwle.entrySet()){ dwarwle(entry.getKey(),entry.getValue(),dwarwleKey); } } The code was generally ok. The method to dwarwle an individual dwarwable entry using the actual key it is assigned to in the hash map and the dwarwle key is factored to a separate method. It is so simple that I do not list here. The variable names are also meaningful so long as long you know what actually dwarwling is. The method is short and readable, but the argument list expects a HashMap instead of a Map. Why do we want to restrict the caller to use a HashMap? What if the caller has a TreeMap and for good reason. Do we want to have separate method that can dwarwle TreeMap? Certainly not.Expect the interface, pass the implementation.The junior changed the code replacing HashMap to Map but after five minutes or so this clever lady raised her hand and had the following question: “If we changed HashMap to Map, why did not we change String to CharSequence?” It is not so easy to answer a question like that when it comes out of the blue. The first thing that came up in my mind is that the reason is that we usually do it that way and that is why. But that is not a real argument, at least I would not accept anything like that and I also except my students not to accept such answer. It would be very dictator style anyway. The real answer is that the parameter is used as a key in a map and the key of a map should be immutable (at least mutation should be resilient to equals and hashcode calculation). CharSequence is an interface and an interface in Java (unfortunately) can not guarantee immutability. Only implementation can. String is a good, widely known and well tested implementation of this interface and therefore can be a good choice. There is a good discussion about it on stackoverflow. In this special case we expect the implementation because we need something immutable and we “can not” trust the caller to pass a character sequence implementation that is immutable. Or: we can, but it has a price. If a StringBuilder is passed and modified afterwards our dwarwling library may not work and a blame war may start. When we design an API and a library we should also think about not only the possible but also about the actual, average use.A library is as good as it is used and not as it could be used.This can also be applied to other products, not only libraries but this may lead too far (physics and weapon).Reference: Why to use String from our JCG partner Peter Verhas at the Java Deep blog....

Java Extension Mechanism Loads All JARs

The Java Extension Mechanism is described in the Java Tutorial as a “standard, scalable way to make custom APIs available to all applications running on the Java platform.” As described in Understanding Extension Class Loading, “the extension framework makes use of the class-loading delegation mechanism” with extension classes loaded after the bootstrap classes in rt.jar (and related JARs) but before the classes loaded from the typical classpath. The extension directory works a bit like the classpath in that its part of the class loading mechanism and classes available within JARs in the extension directory are made available to Java applications. There are some key, differences, however, and some of these are highlighted next.  Characteristic Classpath Extension Mechanism (Optional Packages)Scope Typically Application-Specific-classpath/-cp java.class.path Executable JAR Manifest’s Class-PathPotentially All JREs on HostCLASSPATH environment variableAll JVMs Running in Specific JREjava.ext.dirsAll Host’s JREsSolaris: /usr/jdk/packages/lib/ext Linux: /usr/java/packages/lib/ext Windows: %SystemRoot%\Sun\Java\lib\extHow Specified .jar FilesExplicitly specified by name (including .jar) Wildcard (*) matching all all JAR files with .jar extensions.class FilesDirectory containing .class files specifiedAll JAR files (even if extension other than .jar or no extension at all) in designated directories are loaded.Class Loading Order After bootstrap and extensions loading. After bootstrap but before classpath.One of the most significant observations worth some more emphasis is that the extension mechanism will pick up all JAR format files in the extension directory even if the file does not have a .jar extension. The implication of this is that while one can change the name of a JAR located in a classpath directory to have an extension other than .jar so that the wildcard does not pick it up, this technique will not work with the extension directory. I’m going to use some simple examples in this post to demonstrate some of these differences. The next two code listings are for a very simple HelloWorld class and a main application class called Main that uses the HelloWorld class. HelloWorld.java public class HelloWorld { @Override public String toString() { return "Hello, World!"; } } Main.java import static java.lang.System.out;public class Main { public static void main(final String[] arguments) { out.println(new HelloWorld()); } } To demonstrate a primary difference between classpath and the extension mechanism (optional packages), I will archive the compiled HelloWorld.class file into a JAR called HelloWorld.jar and place it in a different directory than the compiled Main.class file. To demonstrate the use of the traditional classpath, I place the HelloWorld.jar file in a directory called C:\hello and will access that JAR via wildcard (*) for Main to use. This is demonstrated in the next two screen snapshots.The two previous images demonstrate that the Java Main application can still load the HelloWorld.class file even though I had deleted it from the current directory because the Java launcher was explicitly told (via the -classpath option) to look for it in C:\hello. Using the extensions mechanism (optional packages), it is possible to have the class loaded without it being in the same directory and without explicit classpath specification. This is shown in the next screen snapshot.The previous screen snapshot demonstrates that the Java launcher doesn’t even need the HelloWorld.class in the same directory or specified on its classpath when that class is inside a JAR that is in the extensions (optional packages) directory. This is often cited as a benefit of using the extensions mechanism because all applications using that JRE (or potentially all applications on the host) can see the same classes without need to explicitly specify them on the classpath. With the traditional classpath approach of instructing an application to load classes from JARs, the JAR file containing the .class file needs to end with the .jar extension. The next screen snapshot demonstrates what happens when the HelloWorld.jar is renamed HelloWorld.backup in the same classpath-referenced directory.The last image demonstrates that a NoClassDefFoundError is encountered when the JAR file in the classpath-referenced directory does not have a .jar extension. Perhaps a bit surprisingly, the extensions (optional packages) mechanism does not work the same way. Instead, all JAR files in the extensions specified directory are loaded regardless of their extension and regardless of even if they have a file extension. This is demonstrated in the next screen image.The previous image demonstrates that renaming the JAR file that resides within the extensions directory to not have any file extension whatsoever does not prevent the classloader from loading the classes of that JAR. In other words, the classloading mechanism loads all JAR files in the specified extensions directory based on file type rather than on file name or extension. As the Optional Packages Overview summarizes, “There is nothing special about any particular JAR file itself or the classes it contains that makes it an installed optional package. It is an installed optional package by virtue of its location in jre/lib/ext.” There are some risks and downsides associated with placing too many class definitions in JARs inside the extensions directory. It can be maddening to wonder why NoSuchMethodErrors, for example, are occurring when one can see that a class specified explicitly on the classpath has the method in question. I have previously written about one of the many potential causes of NoSuchMethodError, but forgotten outdated and obsolete class definitions residing inside of JAR files in the extensions directory are another potential cause. This is demonstrated next. The next two code listings show revised versions of Main.java and HelloWorld.java. In particular, HelloWorld has an all-new method that the new version of Main invokes. In this case, I’m going to leave the newly compiled HelloWorld.class file in the same directory when I run the Main to demonstrate that the old, busted version of HelloWorld.class in the JAR in the extensions directory takes precedence over the new hotness HelloWorld.class in the current directory. Revised Hello World.java (New Method) public class HelloWorld { @Override public String toString() { return "Hello, World!"; }public String directedHello(final String name) { return "Hello, " + name; } } Revised Main.java import static java.lang.System.out;public class Main { public static void main(final String[] arguments) { final HelloWorld helloWorld = new HelloWorld(); out.println(helloWorld); out.println(helloWorld.directedHello("Dustin")); } }The last image demonstrates that the now obsolete class definition of HelloWorld in the extensions directory takes precedence over the new class definition of HelloWorld in the same directory. Even when I specify the current directory on the classpath, the old version in the extensions directory takes precedence. This is shown in the next screen snapshot, which also shows that the JAR in the extensions directory that is “hiding” the newer JAR and its class’s newer method is still not even named with a .jar extension.The example just demonstrated is not even the most difficult situation a forgotten JAR in the specified extensions directory (or directories) can cause. In that example, at least I had a NoSuchMethodError to alert me to a problem. A potentially even more difficult situation to debug can exist when the old class definition has the same method signature but has an outdated method implementation. In such cases, there may be no error, exception, or throwable of any kind, but the application logic will simply not work correctly or as expected. The old functionality could exist in the code base for some time before it’s even recognized as an issue, especially if unit tests and other testing is lacking. Use of the extensions directory can make things easier on developers because classes in JAR files residing in the extensions directory (or directories) are available to all applications in the JRE associated with the extensions directory (or with all JREs on the host if the operating system-based host-wide extensions directory is used). However, there are definite risks associated with too liberal use of the directory. It can be easy to forget that outdated class definitions residing in JARs in that directory are preventing classloaders from loading the newer and seemingly obvious versions of the class definitions. When this happens, the very extensions (optional packages) mechanism that made developers’ lives easier now make it more difficult. Elliotte Rusty Harold provides a warning about use of the extensions (optional packages) mechanism, “While this seems convenient, it is also a long-term mistake… Sooner or later (probably sooner), you’ll load the wrong version of a class from a place you aren’t even thinking about and waste hours debugging.” The Java Tutorial also recommends caution (I added the emphasis), “Since this mechanism extends the platform’s core API, its use should be judiciously applied. Most commonly it is used for well standardized interfaces such as those defined by the Java Community Process, although it may also be appropriate for site wide interfaces.” Although the extensions (optional packages) mechanism is similar to the classpath mechanism and both are used as part of class loading, the differences between the two are important to note. In particular, it is important to remember that all JAR files (even if they don’t have .jar file extensions) that reside in the directory referenced as an extension directory will be loaded. Renaming these JARs and even changing their file extension will not be sufficient to have the classloading ignore them. With classpath, on the other hand, renaming the JAR is sufficient to prevent loading when the classpath specifies individual JAR files explicitly and changing the file extension is typically sufficient to prevent loading even when the classpath uses the wildcard (*) to specify all JARs in a directory. There are situations when the extensions (optional packages) mechanism is the appropriate choice, but these seem fairly rare. It is also important to keep in mind the extensions (optional packages) mechanism in mind when dealing with unexplained NoSuchMethodErrors so that one can check it out to see if the offender lives in that directory or directories.Reference: Java Extension Mechanism Loads All JARs from our JCG partner Dustin Marx at the Inspired by Actual Events blog....

Docker RabbitMQ cluster

I have been trying to create a Docker based RabbitMQ cluster on and off for sometime and got it working today – fairly basic and flaky but could be a good starting point for others to improve on. This is how the sample cluster looks on my machine, this is a typical cluster described in the RabbitMQ clustering guide available here -https://www.rabbitmq.com/clustering.html. As recommended at the site, there are 2 disk based nodes and 1 RAM based node here.        To quickly replicate this, you only need to have fig in your machine, just create a fig.yml file with the following entry: rabbit1: image: bijukunjummen/rabbitmq-server hostname: rabbit1 ports: - "5672:5672" - "15672:15672"rabbit2: image: bijukunjummen/rabbitmq-server hostname: rabbit2 links: - rabbit1 environment: - CLUSTERED=true - CLUSTER_WITH=rabbit1 - RAM_NODE=truerabbit3: image: bijukunjummen/rabbitmq-server hostname: rabbit3 links: - rabbit1 - rabbit2 environment: - CLUSTERED=true - CLUSTER_WITH=rabbit1 and in the folder holding this file, run: fig up That is it!, the entire cluster should come up. If you need more nodes, just modify the fig.yml file. The docker files for creating the dockerized rabbitmq-server is available at my github repo here: https://github.com/bijukunjummen/docker-rabbitmq-cluster and the “rabbitmq-server” image itself is here at the docker hub. ReferencesThe base rabbitmq image is somewhat based on cthulhuology’s docker rabbitmq image: https://github.com/cthulhuology/docker-rabbitmq Docker file: https://github.com/bijukunjummen/docker-rabbitmq-cluster Docker image: https://registry.hub.docker.com/u/bijukunjummen/rabbitmq-server/Reference: Docker RabbitMQ cluster from our JCG partner Biju Kunjummen at the all and sundry blog....
Java Code Geeks and all content copyright © 2010-2014, Exelixis Media Ltd | Terms of Use | Privacy Policy | Contact
All trademarks and registered trademarks appearing on Java Code Geeks are the property of their respective owners.
Java is a trademark or registered trademark of Oracle Corporation in the United States and other countries.
Java Code Geeks is not connected to Oracle Corporation and is not sponsored by Oracle Corporation.
Do you want to know how to develop your skillset and become a ...
Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you two of our best selling eBooks for FREE!

Get ready to Rock!
You can download the complementary eBooks using the links below: