A week ago James Duncan Davidson and Mike Clark launched @dailyshoot, a Twitter feed that posts daily photo assignments. The idea is to encourage people who want to learn photography to practice it every day with the help of a simple assignment that fits a single tweet. I’m following Duncan’s blog, so I found out about Daily Shoot the day it was launched.

So far I’ve completed all the assignments and I’ve already learned quite a bit doing so. It’s very interesting to see how other people interpret the same assignments. I avoid looking at other responses before completing an assignment so that I don’t end up just copying someone else’s approach. Once I’m done I look at what other’s have done for some nice insight on what I could have done differently. The process is quite educational.

Here’s what I’ve shot this week:

You can click on the pictures for more background on each assignment and how I approached it. For more information on Daily Shoot, see the recently launched website.

Two weeks ago during the BarCamp at the ApacheCon US I chaired a short session titled “The RESTful Content Repository”. The idea of the session was to discuss the various ways that existing content repositories support RESTful access over HTTP and to perhaps find some common ground from which a generic content repository protocol could be formulated.

The REST architectural style was generally accepted as a useful set of constraints for the architecture of distributed content-based applications, but as an architectural style it doesn’t define what the bits on the wire should look like. This is what we set out to define with the HTTP protocol as a baseline. We didn’t get too far, but see below for some collected thoughts and a useful set of “test cases” that I hope to use to further investigate this idea.

Existing solutions

Many existing content repositories and related products already support one or more HTTP-based access patterns: Apache Jackrabbit exposes two slightly different WebDAV-based access points. Apache Sling adds the SlingPostServlet and default JSON and XML renderings of content. Apache CouchDB uses JSON over HTTP as the primary access protocol. Apache Solr uses XML over HTTP. Midgard doesn’t have a built-in HTTP binding for content, but makes it very easy to implement such bindings. This list just scratches the surface…

There are even existing generic protocols that match at least parts of what we wanted to achieve. WebDAV has been around for ten years already, but the way it extends HTTP with extra methods makes it harder to use with existing HTTP clients and libraries. The AtomPub protocol solves that issue, but being based on the Atom format and leaving much of the server behaviour undefined, AtomPub may not be the best solution for generic content repositories.

Content repository operations over HTTP

To better understand the needs and capabilities of existing solutions, we should come up with a simple set of content operations and find out if and how different systems support those operations over HTTP. The most basic such set of operations is CRUD, i.e. how to create, read, update, and delete a document, so let’s start with that. I’m giving each operation a key (CRn, as in “Content Repository operation N”) and a brief description of what’s expected. In later posts I hope to explore how these operations can be implemented with curl or some other simple HTTP client accessing various kinds of content repositories. I’m also planning to extend the set of required operations to cover features like search, linking, versioning, transactions, etc.

CR1: Create a document

Documents with simple properties like strings and dates are basic building blocks of all content applications. How can I create a new document with the following properties?

• title = “Hello, World!” (string)
• date = 2009-11-17 (date)

At the end of this operation I should have a URL that I can use to access the created document.

CR2: Read a document

Given the URL of a document (see CR1), how do I read the properties of that document?

The retrieved property values should match the values given when the document was created.

CR3: Update a document

Given the URL of a document (see CR1), how do update the properties of that document? For example, I want to update the existing date property and add a new string property:

• date = 2009-11-18 (date)
• history = “Document date updated” (string)

When the document is read (see CR2) after this update, the retrieved information should contain the original title and the above updated date and history values.

CR4: Delete a document

Given the URL of a document (see CR1), how do I delete that document?

Once deleted, it should no longer be possible to read (see CR2) or update (see CR3) the document.

We’re organizing a NoSQL meetup in Oakland on Monday next week. In addition to helping set the meetup agenda, the “Topics you are interested in” question in the sign up form provides some interesting insight on the current interests of the NoSQL community. Here’s a quick breakdown of the key terms distilled from the 88 signups we’ve received so far.

Note that the data is biased towards Apache projects due to the meetup being organized at ApacheCon US 2009.

Projects

The following open source projects were mentioned. The list is in alphabetical order, as the data set is too small to make any reasonable ordering by popularity.

• Cassandra
• CouchDB
• Hadoop
• HBase
• HDFS
• Jackrabbit
• Lucene
• Mahout
• memcached
• MongoDB
• Redis
• Riak
• Scalaris
• Sling
• Tokyo Cabinet
• Voldemort

Topics

Many responses were about the “big data” aspect of the NoSQL movement. Some frequent keywords: distributed storage, large transactional data, consistency, failover, availability, reliability, stability, failure detection, failed node replacement, (petabyte) scalability, consistency levels, storage technology, performance, benchmarks, optimization, backup and recovery, map/reduce

Another common theme were the various database types and the NoSQL “development model”. Keywods: document stores, key/value stores, consistent hashing, graph databases, object databases, persistent queues, content modeling, migration from the relational model, social graphs, streaming, software as a service, offline applications, full text search, natural language processing

Beyond the above big themes, I found it interesting that the following technologies were specifically named: Erlang, Java, WebSimpleDB, WebDAV

In addition to specific topics, many people were asking for case studies or “lessons learned” -type presentations.

The Apache POI team is doing an amazing job at making Microsoft Office file formats more accessible to the open source Java world. One of the projects that benefits from their work is Apache Tika that uses POI to extract text content and metadata from all sorts of Office documents.

However, there’s one problem with POI that I’d like to see fixed: It’s too big.

More specifically, the ooxml-schemas jar used by POI for the pre-generated XMLBeans bindings for the Office Open XML schemas is taking up over 50% of the 25MB size of the current Tika application. The pie chart below illustrates the relative sizes of the different parser library dependencies of Tika:

Both PDF and the Microsoft Office formats are pretty big and complex, so one can expect the relevant parser libraries to be large. But the 14MB size of the ooxml-schemas jar seems excessive, especially since the standard OOXML schema package from which the ooxml-schemas jar is built is only 220KB in size.

Does anyone have good ideas on how to best trim down this OOXML dependency?

I’ve recently spent some effort in improving the look of the Apache Jackrabbit website. I’m no designer, so the results aren’t that great, but it’s been a nice break from the regular project work. And I got to brush up my Photoshop and Gimp skills.

One part of the effort was creating an icon for the site. Previously the site used the feather icon used as the default on all Apache project sites, but I wanted a Jackrabbit-specific icon that helps me to quickly identify and access Jackrabbit pages among the numerous tabs I usually have open in my browser. The work is a good example of incremental improvements in action:

I started with a copy of the Jackrabbit logo with nice alpha-layered transparent background. It looked great until I noticed that some browsers lost the smooth alpha layer and instead resulted in a rather badly aliased icon seen above.

The straightforward solution was to add a white background as can be seen in step 2. That worked already pretty well in all browsers.

After a few days of watching the icon I found it a bit too blocky to my taste, so I tried to restore some of the nice transparency effect by rounding the corners a bit. I’m pretty happy with the result.

Of course, if you have design talent and think you can do better, go for it!

There’s lots of upcoming release activity at the Apache projects I’m more or less involved with:

• The incubating Apache PDFBox project is just about to release the eagerly anticipated 0.8.0 release. I’m expecting to see the release announcement on Tuesday next week. PDFBox is a Java library for working with PDF documents.
• Another incubating project, Apache UIMA, is working towards the 2.3.0 release. I’m looking forward to seeing both UIMA and PDFBox graduating from the Apache Incubator shortly after the respective releases. UIMA is a framework and a set of components for analyzing large volumes of unstructured information.
• The Apache Sling project is a component-based project like Apache Felix, so there is no clear project-wide release cycle.  Instead Sling is about to start releasing new versions of most of the components changed since the all-inclusive incubator releases. Sling is a JCR-based web framework.
• Apache Tika uses PDFBox for extracting text content from PDF documents. I’m hoping to see a Tika 0.5 release soon with the latest PDFBox dependency and the design improvements I’ve been working on. Tika is a toolkit for extracting text and metadata from all kinds of documents.
• Apache Solr is about to enter code freeze in preparation for the 1.4 release that will include the “Solar Cell” feature based on Tika. Solr is a search server based on Lucene.
• The Commons IO project has been upgraded to use Java 5 features and I’m starting to push it towards a 2.0 release. Commons IO is a library of Java IO utilities.
• Lucene Java is gearing up for the 2.9 release, and will soon follow up with the 3.0 release. The trie range feature is an especially welcome addition for many use cases. Lucene is a feature-rich high performance search engine.
• And last but not least, Apache Jackrabbit is getting ready to release the 2.0 version based on the recently approved JCR 2.0 standard. Jackrabbit is a feature-complete JCR content repository implementation.

I’m hoping to see most of these releases happening in time for the ApacheCon US 2009 conference in early November.

The Apache Jackrabbit project has just released Jackrabbit version 1.6.0. This release will most likely be the latest JCR 1.0 -based Jackrabbit 1.x minor release before the upcoming Jackrabbit 2.0 and the upgrade to JCR version 2.0. The purpose goal of this release is to push out as many of the recent Jackrabbit trunk improvements as possible so that the number of new things in Jackrabbit 2.0 remains manageable.

The most notable changes and new features in this release are:

• The RepositoryCopier tool makes it easy to backup and migrate repositories (JCR-442). There is also improved support for selectively copying content and version histories between repositories (JCR-1972).
• A new WebDAV-based JCR remoting layer has been added to complement the existing JCR-RMI layer (JCR-1877, JCR-1958).
• Query performance has been further optimized (JCR-1820, JCR-1855 and JCR-2025).
• Added support for Ingres and MaxDB/SapDB databases (JCR-1960, JCR-1527).
• Session.refresh() can now be used to synchronize a cluster node with changes from the other nodes in the cluster (JCR-1753).
• Unreferenced version histories are now automatically removed once all the contained versions have been removed (JCR-134).
• Standalone components like the JCR-RMI layer and the OCM framework have been moved to a separate JCR Commons subproject of Jackrabbit, and are not included in this release. Updates to those components will be distributed as separate releases.
• Development preview: There are even more JSR 283 features in Jackrabbit 1.6 than were included in the 1.5 version. These new features are accessible through special “jsr283″ interfaces in the Jackrabbit API. Note that none of these features are ready for production use, and will be replaced with final JCR 2.0 versions in Jackrabbit 2.0.

This release is the result of contributions from quite a few people. Thanks to everyone involved, this is open source in action!

The JCR 2.0 API specified by JSR 283 has been in Proposed Final Draft (PFD) stage since March, and Apache Jackrabbit developers have been busy implementing all the specified new features and adding compliance test cases for them.

Both the Reference Implementation (RI) and the Technology Compatibility Kit (TCK) of JSR 283 will be based on Jackrabbit code, and we expect the final version of the specification to be released shortly after Jackrabbit trunk becomes feature-complete and the API coverage of the TCK reaches 100%. The following two graphs illustrate our progress on both these fronts.

First a track of all the JCR 2.0 implementation tasks we’ve filed under the JCR-1104 collection issue. The amount of work per each sub-task is not uniform, so this graph only shows the general trend and does not suggest any specific completion date.

The second graph tracks the TCK API coverage. We started with the JCR 1.0 TCK, so the first 300-400 method signatures were already covered with few changes to existing test code. Based on Julian’s API coverage reports in JSR-2085, this graph tracks progress in covering the 100+ new method signatures introduced in JCR 2.0. Again, the graph is meant to show just a general trend and should not be used to extrapolate future progress.

Wan’t to see JCR 2.0 in action? The latest Jackrabbit 2.0 alpha releases are available for download!

The punchcard graphs at Github are a nice way to quickly detect the rough geographical distribution (or nighttime coding habits) of the key contributors of an open source project. Here’s a few selected examples from the ASF.

Apache HTTP Server

Apache Maven (core)

Apache Jackrabbit

Apparently they’re now setting up a Pirate Party also in Finland. I guess it’s good to have a political force that questions the appropriateness of traditional copyright in the digital world. However, as a knowledge worker I’m not that excited about drastic changes in the protection of immaterial rights.

Anyway, my appreciation for the movement in Finland went down considerably when I saw their spokesman in the news today. When asked about the main goals of the new party he only mentioned freedom of speech and protection of privacy. Did he just forget the massive overhaul of copyright and patent laws that they’re primarily after?