- Bring your company into the century of Data Science
- Add Data Science to the strategic capabilities of your company
- Avoid expensive pitfalls during the introduction
- Bridge the Knowledge Gap before falling for expensive solutions
- Identify Use Cases within your company
- Training for your In House Data Scientists
- Joint Projects for efficient Knowledge Transfer
- Delivery of Key-Turn Solutions
- Help with the Infrastructure Installation
- Support your Data Scientists with our Development Service
- Add features to your platform that are specific to your company
- Avoid breaking the tool chain
- Reduce maintenance costs
Today we are happy to announce the Advanced Reporting Extension for RapidMiner. With it's three operators, it looks tiny in comparison to the bulky Jackhammer Extension, but it adds them in a blind spot of RapidMiner and are designed to take away some worries from the common data scientist.
The idea is to use the capabilities of RapidMiner to automate any regular reporting task that results in an Excel sheet. There have been many projects and data science departments that simply drown in these kind of request, consuming all resource before you can get to the really fun part of data science. Now you can simply start at the beginning to create a nearly zero overhead reporting, even if you don't have or can't use real business intelligence tools like tableau or qlik.
How does that work?
Step 1: Create a template in Excel
First we create a dummy sheet and add all of the desired layout components, diagrams, texts and of course areas for data.
We can use any formatting, chart type or conditional coloring that we like, including the nice spark lines. Just one thing is important: We need to reserve space for inserting the data. What will happen later is, that we overwrite parts of the content of the table with data from RapidMiner. So if we have more than three employees, we would need either let more space between the table and the diagram, or just put the data into a separate sheet and reference this in the diagram. But if you are used to Excel reporting, you probably know all these tricks...
Insert some dummy values so that you can see the charts in action.
Don't forget to save the file. We will need it later.
Step 2: Create a process in RapidMiner to load the data
RapidMiner is very versatile to get the data into the shape you want. It can read and combine many different formats and sources and then aggregate, join, pivot and process the data into the shape that you need it.
On the right you see a process combining data from four different sources with multiple joins and preprocessing steps to match the data. Such a process could just deliver us the data we want to put into our nice Worktime sheet.
Of course it could be much simpler and just contain a single SQL query or also be very much more complex involving calling of webservices, Big Data and analytics on hadoop, some machine learning or whatever. The trick is that we can leverage the entire flexibility of RapidMiner to get the data we want to put into an Excel sheet.
Step 3: Open Report
Once we have the data in the desired format, we add an Open Report (Excel) operator from our extension. You see it on the right hand side in the operator tree. We need to point the operator on two files: The template file we created and saved in Step 1. You can either use the parameter form template file or the tem input port. The second file can be specified as target file parameter or by using the tar output port.
Why are there ports for the files? Because it allows you to handle the files conveniently in scenarios where you want to do stuff with them in the process later. You could even create a template file in a RapidMiner process, or less fancy and more realistic: Store the file in the repository of a RapidMiner Server to share among many users. The output file port is most useful if you want to either zip the result or return it as a webservice result in a RapidMiner Server Webservice or Web Application.
Any data we want to insert into the Excel file, we need to forward to the input ports of the Open Report (Excel) operator. Don't worry, there will always be another input port if you connect the last one. We will use the data delivered to these ports in the inner subprocess to do the actual insertion.
Step 4: Insert Tabular Data
If we entered the inner process of the Open Report (Excel), we can add the Write Data Entry (Excel) operator to insert an ExampleSet into the excel. We have done so with the first ExampleSet on the screenshot on the right. The operator allows to select which attributes to use and where to place it. Therefore you specify the sheet where it will be insert by it's index. Then you point it to a fill range. A range can be either open ended by specifying the left upper cell of the area or closed, if followed by a colon and the right lower cell. So B2 would start in the second column, second row. B2:D4 would allow to fill 2 rows and 2 columns.
For our little employee table from Step 1, we set it to B11:C13. Unless we select fit to range, the process will now fail if our data does not fit into this range.
We will add another operator of this type to output the second table.
Step 5: Insert Data
The only thing missing is the version tag, so that people know what this report was about, when they open them at some point later.
Therefore we first use a Generate Macro operator from RapidMiner's core functionality to create a process variable (or macro as they call it) containing the current date and time. We then add a Write Cell (Excel) operator from the Advanced Reporting Extension and connect the ports. Although there will be no data flowing from the Generate Macro operator to the Write Cell (Excel) operator, the connection makes sure that the Generate Macro will be executed first and set the process variable before it is read.
Then we just need to point the Write Cell (Excel) operator to the right fill position, which is F5 in our case. Setting the value and type correctly and we are good to go.
Short notice on dates: There is an unlimited number of different date formats out there. If you want to write a date to excel, you first need to parse the date format that the value has in RapidMiner. So if you enter something like 2017-03-29 23:59:59 as value, you should enter "yyyy-MM-dd HH:mm:ss" in the date format parameter of the Write Cell (Excel) operator. Once it knows the date, it will automatically transform it in the correct format of the Excel Template Sheet, where you set it with the Cell Format.
Once the subprocess is finished the target file will be written and you just need to mail it to someone else and be done with it.
We would like to recommend to just automate about everything right from the beginning. There will be nothing like a "I just need to do this once". In 90% of all cases, you will need to do it twice and then the additional overhead of the automation already would have paid off. So please feel free to download the extension, order a license and ask any questions you might have. In case you are not convinced, yet, the free version let's you access the full functionality and only limits the number of Write operators to one within each subprocess.
Undoubtedly, Data Science is one of the most important new capabilities in companies that will shape the future over the next decades. But how to establish Data Science with all its facets from the craftsmanship of big data architecture over the art of business intelligence towards the magic of predictive analytics in a company? Where do you start? Which steps do you need to take when you want to get there? And what actually is "there" at all?
The Düsseldorf Data Science Meetup invited our Lead Data Scientist Sebastian Land to give some answers. His talk aggregated over ten years of experience in the field and joins it with in-depth understanding of the involved technologies. Enriched with our practical project experiences at Old World Computing, he, as a true data scientist by heart applied his predictive powers to picture possible ways with their chances but also cautioned about probable pitfalls and the cliffs of costly decisions on this journey into the future.
We thank the organizers for the opportunity to speak on such a great event in a great location and hope that the surprisingly many visitors could take home at least some light to shed at their own problems. Thank you for coming!
To preserve the knowledge hopefully gained there, we have made the slides available here.
(Taken from Düsseldorf Data Science Meetup)
We just published an update of the Statistics Extension that should make it compatible with RapidMiner 7.3. However, several restrictions of the new RapidMiner Security Mechanism are preventing existing functionality to work. While all operators are working as expected, you will not be able to enter the license key in the dialogue.
Hence for now you need to go over the RapidMiner Settings / Preferences menu, switch to the Licenses tab and past your email and license key there as shown on the right.
While our Jackhammer performed very well in our projects, we are relieving it from the active duty. For now.
This actually does sound much more sad than it really is. The simple reality is that it won't be compatible with future versions of RapidMiner in it's current form and we don't want to promise something to our customers that we cannot stay up to. Therefore we will take some time to spit polish everything that is in there and will remain working and then he will be back, hammering away your data science problems. In the meantime we discovered so many useful features for large scale data science projects that we are going to implement, you should watch his return anyway.
So, finally we have done it! If you have followed our twitter posts or some remarks in the RapidMiner Community, you already may have been aware that we have been working on something larger. Now, after more than a month of spit polishing our brand new extension with documentation, tests and improved user interface, it's done. It's available now!
Wow, this really is a great feeling right now. I guess you would have an easier time sharing my feelings and excitement, if you knew what the extension is actually about. So please find below a very short collection of images. A long description of the features can be found here.
Loops to save time and complexity
The new loop operators allow to use arbitrary input ports. If you need to continuously change one specific object, you can get it from the loop port which is in the first iteration holding a copy of the same object as on the first input port. But once the first iteration is executed, the results from the first iteration will be forwarded back to the input loop port for the next one. No Remember / Recall needed to continuously aggregate data as above.
The screenshot shows how data is appended to previous results and then swapped onto the disk.
Continuous Tests for your process collection
Here you see a simple process that combines our new Loop Repository operator, that allows you to iterate over processes with the testing operator. It loads the process and the respective result and then compares it. If the result does not match, regardless whether it is data, performance vector or model, an exception is thrown and you know that something went wrong in your infrastructure.
On Disk Memory and Caching
The Storage Statistics Panel gives you an overview of how much memory you are consuming on your hard disk (o better solid state disk) and how many cache entries are currently stored.
Above you see how ten thousands of columns are stored, each of them in a data set with around 60000 rows. And this extra memory just comes with a speed penalty of factor three.
The new Cross Validation
The new cross validation looks nearly the same from the inside...
...but allows to generate final preprocessing models if they are built during the training phase. Training phase may also use other external input. Finally the cross validation can output test results for all rows like an X-Prediction with now time overhead. But that's not all...
Unlike the default version, it will make of your powerful multi core CPU like here on my Laptop.
The same is true for the loop operators that can execute iterations in parallel if the results are not depending on each other! A great way to speed up complex operations on larger (but not yet big) data sets.
As you can not only use your multi core cpu for speeding up some operators, but you can also run multiple processes at once, right within your studio. Simply drag them onto the panel or select the play button!
You see above that they will be shown beside the tasks of operators. Finished processes allow to access their results until you clean the list. You can even access the logs of the processes during runtime!
Free Demo Version
The good news is: Everybody who wants to try it can just proceed with the download from our product page here. While the free demo version is restricted in some features, most of them are free to use for everybody! But if you like it, we will be very happy to welcome you as paying customer...
Of course we are also happy to receive some feedback about the features, what's great, what's missing and so on, but first you should get yourself familiar with the extension: Here's the list of the features.
As the benefit of some of the features may be not obvious for everybody, we will release a series of videos, explaining when, why and how to use them. Most of them are really meant for the hard boiled data scientist in a productive environment that not all of us might face everyday. On the other hand, the extension also features several operators that are very useful for the beginner as well as they make things simply easier to achieve.