Goal
To give any person without programming skills the ability to create a Big Data analysis system.
Solution
jStart Beacon is a system for designing, building, and running applications that collect, analyze, and visualize Big Data. The system includes a web-based visual application editor that has a number of ready-made components for integration with data sources (files, databases, and social networks). The users can either build their models from ready-made components or modify the ways the data stream is processed within the components of their model by themselves. The system supports viewing the data processing results in each component in real time, which helps debug the model quickly.
The system can run user models on the Storm cluster or on Cloud Foundry. It is also possible to deploy a cluster of your own (based on Docker containers) to execute the models. Data streams are processed in real time within one of the distributed systems mentioned above, and the results are visualized in the web client. Models and topologies can be launched on schedule.
The main advantage of this solution is that a person without any programming skills can create a Big Data analysis system.
The system design has the potential to be extended using additional components that provide access to more data sources and receivers. Our company developed a basic set of standard components that provide access to information from the most popular data sources, such as file systems, SQL databases, and social networks. Third parties working on the project are currently developing extra components, for example, to work with mongoDB, a NoSQL database.