The technologic evolution of the last decades triggered an increase in the volume of time series to process, in numerous fields such as experimental science. The experimental time series data are processed to find a mathematical model (usually a differential equation) fitting the series. Due to the amount of available data and technologies to process them, the amount of available mathematical models is increasing as well. Therefore, storing and organizing those models to ease their management and retrieval has become an essential challenge.
The MathMOuse project (Mathematical MOdels warehoUSE) is an enriched Data Warehouse structure able to:
- store mathematical models in a relational database,
- provide the user a "query by data" system to query the Warehouse using raw time series as input parameter.
Software requirements¶For all OS:
- Java JRE >= 1.8 : http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
- Docker (for people who can't install Docker, a Vagrant configuration file is supplied) : https://www.docker.com/community-edition#/download
NOTE: If you want to use Vagrant, you will need:
- Virtualbox >= 5 : https://www.virtualbox.org/
- Vagrant >= 1.8.7 : https://www.vagrantup.com/downloads.html
In a command prompt, browse to your local copy of the project and type:
$ vagrant up
A virtual machine, will be automatically set up. Then:
$ vagrant ssh
NOTE: If the vagrant ssh command failed, maybe you need to install a SSH client.
The command will connect you to the newly created virtual machine.
The MathMOuse project binary files can be downloaded here. The source code will be available for download after the end of the thesis.
The archive file (mathmouse-bda-2017.zip) is composed of:
- docker-compose.yml: is a Docker configuration file, that will be used to create the necessary services.
- dockermanager-0.0.1-SNAPSHOT.jar: a Java jar file, that contains binary code for building the services and manage them.
- Vagrantfile: a Vagrant configuration file to create a virtual machine
- bootstrap.sh: an additional configuration file to set up a virtual machine created by Vagrant
- javarabbitmq: a directory containing files (Dockerfile and init.sh) to configure the rabbitmq service.
- mathmousedb: a directory containing files (init-db.sh and init-db.sql) to configure the postgreSQL service and the database.
- mathmouse: a directory containing binary Java jars for other microservices (comparatormanager, comparator, generator, dbcore, dbgui, dockermanager, populator) of the projects and support projects (dbcoreapi, dockermanagerapi, javarabbitmqapi, timeseriesreductorslib). As well as Dockerfile configuration files, for the services. In the dbgui there is an additional resources directory with XML and CSV files, used as data samples of equations and time series, respectively. In the populator directory, there is another resources files, with XML files, that can be used to quickly populate the database for tests purposes.
The prototype where developed and tested under Linux.
Using a command prompt, browse to your local copy of the project and type:
$ cd mathmouse-bda-2017 $ java -jar dockermanager-0.0.1-SNAPSHOT.jar &
This command will build and set up the necessary services (rabbitmq, postgresql, comparator manager, comparator, generator). Your prompt should be similar to the following image:
NOTE: you can check if the services are running by typing:
$ sudo docker ps
You should see the following:
To start the GUI, you will need to use the command prompt and browse to your local copy of the project. Then, type:
$ cd microservices/mathmouse/dbgui
$ java -jar dbgui-0.0.1-SNAPSHOT.jar
to start the GUI. The following window should open:
Add a user-defined equation in the database¶
Three XML files are available in the microservices/mathmouse/dbgui/resources/xmls directory. They contain the XML representation of three equations, given by automatic control experts. To add an equation in the database, you will have to select one of the mentioned files, with the GUI:
- Click on the "Upload" (above the navigation tree in the Equation tab) button of the GUI window
- Select a file
- Click on "Open"
- Click on "Validate"
You can check if the new equation appeared in the navigation tree, like on the following image:
Finding an equation matching a time series¶
Three time series files are are available in the microservices/mathmouse/dbgui/resources/csvs. They contains series of measurements from a real automatic control experiement. You can use one of the series to search the database for an equation matching the desired series. To do so, you will have to go through the following steps:
- Click on the "Create Jobs" Tab
- Click on "Open"
- Select a file
- Click on "Open"
- A line should have appeared in the main window
- Cross the box on the right
- Click "Start"
If you click on the "Jobs Manager" tab, you will see a new line, with the job's state and progression, like you can see on the given preview below:
To check the results, click on "Details" and a new window should pop-up. The results should be constantly updating. The window is divided in three tabs.
- The tab "Accepted Models" is the list of models that fits the data.
- The tab "Rejected Models" is the list of models that does not fit the data.
- The tab "Undetermined Models" is the list of models that could not be classified into the two first categories.
Here is what you should see:
NOTE: if you want to quickly populate the database, a sample of 100 models is ready and available. You can check the XML files in the microservices/mathmouse/populator/resources/xmls directory.
If you want to add them quickly in the database, you will have to close the GUI, if it is running. Then browse to the microservices/mathmouse/populator directory and type:
$ java -jar populator-0.0.1-SNAPSHOT.jar
- Cyrille PONCHATEAU, Ladjel BELLATRECHE, Carlos Ordonez, Mickael BARON, A Database Model for Time Series : From a traditional Data Warehouse to a Mathematical Models Warehouse, 32th French Conference On Advanced Databases (BDA 2016), 2016
- Cyrille PONCHATEAU, Ladjel BELLATRECHE, Mickael BARON, Entrepôt de Données dans l'ère Data Science : De la Donnée au Modèle, 12émes Journées Francophone sur les Entrepôts de Données et analyse en ligne (EDA 2016), edited by RNTI, 2016, pp. 65-79
- Lines of Code: 13 077
- Programming Language: Java