Last modified 3 years ago

http://www.eobjects.dk/datatransformer/DataTransformerLogo.png

DataTransformer

DataTransformer is a lightweight, highly reliable and robust ETL (Extract-Transform-Load) and DI (Data Integration) tool designed for easy integration with other applications.

DataTransformer is based on MetaModel for querying and modelling. Furthermore it is based on Hsqldb for storing of intermediate transformation results and modification of data.

Here are some initial DataTransformer resources for those who want to participate in the development:

What are the main design ideas behind DataTransformer

DataTransformer seeks to solve two high-level issues in most Open Source ETL/DI tools:

  • Limitations caused by memory consumption: Most Open Source (if not all) ETL/DI have difficulties coping with very large amounts of data. The way that DataTransformer tries to solve this problem is by using the embedded database  Hsqldb for almost every transformation. Hsqldb can manage millions of rows by caching when memory consumption is growing too large. Additionally most transformations can be expressed in SQL making it easy to maintain a consistent and simple way of implementing a high level tool on top of the well-known and thoroughly tested database backend.
  • Embedability caused by complicated designs and infrastructure: DataTransformer consists of only a few classes and a lot of work and prototyping has gone into simplyfying and correcting both the interface that is used and the interfaces required to implement new features. Additionally we place a high mark on easy distribution and embedability of the project as middleware in other projects through Maven and an easily understandable integration API.

Building DataTransformer

Building DataTransformer is very simple. If you have the tools described in the ToolBase page, you just need to check out the source code: 

svn checkout http://eobjects.org/svn/DataTransformer/trunk DataTransformer

... and build the project: 

cd DataTransformer
mvn install