05. Januar 2017

Big Data - Many Talk About It, We Do It!

Analytics 750x410

We’ve been working on big data topics in our labs and with our clients for quite a while now. Over time, we built a framework of technologies and utilities we can build data driven projects on. We call it ti&m analytics.

The Architecture

ti&m analytics is based on the Lambda Architecture, an industry best practice using only open source technologies. Data driven or big data ventures have requirements in two categories:

  • Real-time: monitoring, recommendations, fraud detection
  • Historic: analysis of classifications, statistics, exit points and so on

In order to fulfill those requirements, Lambda uses the following four main building blocks:

  • Data integration integrates data from all sources. These can be REST services or sensors (push services) as well as file loads and other periodic loads (pull loads).
    Technologies: kafka, Storm
  • Real time / speedlayer is mainly used for monitoring and alerting use cases and is not able to calculate anything backwards in time.
    Technologies: spark streaming, atmosphere.io
  • Batch / serving layer is used for calculations on historic data and the storing of such. Technologies: HDFS, Spark, HBase
  • Visualization / data access results are displayed either in dashboards or interactive tools as Tableau or using Hive.
    Technologies: ti&m dashboard, Tableau, Hive

Our approach

Using the Lambda Architecture, we built numerous solutions ranging from web analytics to social media monitors and banking appliances. This allowed us to extract a reusable framework including integration for social media (Twitter/Facebook), social collaboration (jive), user tracking across multiple channels, core banking solutions and many more to come.

We worked with a solid library of already implemented analyses, for example process analysis to visualize where clients/users get stuck during processes or where they exit. Other examples would be sentiment analysis, usage statistics by geo location as well as identification of top users and what they are focused on.

Visualization is the key to any data driven initiative! We built a customizable dashboard to visualize real time and historical data. As we don’t believe in anything proprietary, the collected data can also be used with visualization tools like Tableau or interactive querying using Hive, Pig or Drill. For recommender systems or personalized content, data can also be fed into business applications.

Regulation or data governance might require to have all data on premise. Our solution ships using virtual machines and can be installed on the existing infrastructure. We also offer to have shared and privately hosted infrastructure in our datacenters. Consequently, the solution is perfectly suitable to build MVPs, but also for long term production usage.

Implementation

Digital analytics as a whole (organisational and technical) is very complex, therefore implementation has to be done step by step. Primary data sources and how they yield business values have to be identified. Imminent business value can’t always be guaranteed as value in data grows over time. A good example would be collecting usage data, which over time allows deeper insights into how users behave and how they react to changes.

Once the primary goal and the long-term vison are agreed upon, implementation starts. We set up the needed infrastructure either on or off the premise, with the main goal being the full integration within the first iteration. Additionally, the data architecture is defined, including how and where data is stored and in what access patterns it is used. Last but not least, we set up the visualisation using our ti&m dashboard or Tableau/Hive etc.

Please get in touch – come by for a Lab Visit at ti&m!


Martin Fabini
Martin Fabini

Martin Fabini ist seit mehr als 20 Jahren in der IT tätig. Bei ti&m führt er Kunden an neue Business Cases mit neuen Technologien heran.

Ähnliche Artikel

Building an Efficient Chat Tool Prototype<br/>
ManMachine: Building an Efficient Chat Tool Prototype

Artificial Intelligence (AI) has been a major theme in the last decade and numerous big companies have invested a lot of effort into the technology. Within the scope of our last ti&m garage project, we too developed a small but efficient chat tool prototype for a big company in Switzerland.

Mehr erfahren
Cloud Act Lead image
CLOUD Act und Data Residency – Ist amerikanischen Cloud Providern noch zu trauen?

Durch den CLOUD Act müssen US-Firmen Daten auch aus ausländischen Rechenzentren bei Anfrage an US-Behörden ausliefern. Firmen, die mit US-Cloud-Providern zusammenarbeiten, müssen sich dieser Gefahr bewusst sein, wie ti&m-Surfer Bernd Leinfelder in seinem Blog-Beitrag herausarbeitet.

Mehr erfahren
Trust 750x410
Authentication Is Good, Trust Is Better. What About Trusting Delegated Identity?

Trust in a relationship is a must and this is not only holds true for private lives but also in the virtual life. While trustworthiness for established authentication protocols is mainly based on agreement between entities, certificates and keys, trust in the identity delegation context is ambiguous because the owner might not be the consumer of the API. This post addresses some trust concerns when introducing protocols based on identity delegation that de-facto lead to an identity paradigm shift.

Mehr erfahren
Mobile-Applikationen: Sicherer als gedacht <br/>
Mobile-Applikationen: Sicherer als gedacht

Mobile-Applikationen haftet immer noch der Ruf an, weniger sicher zu sein als ihre webbasierten Gegenstücke. In der Realität verhält es sich jedoch genau umgekehrt. Mobile-Applikationen sind aufgrund moderner Sicherheitskonzepte mobiler Betriebssysteme bedeutend sicherer als jeder PC oder jedes Notebook. Dies selbst dann, wenn manche Sicherheitsvorkehrungen vom Benutzer durch so genanntes Jailbreaking bzw. Rooting der Geräte aktiv unterwandert worden sind.

Mehr erfahren
Microservice-Architekturen
Single Sign-On als Authentisierungslösung für Microservice-Architekturen

Eine der wichtigsten Säulen der IT-Sicherheit jeder Applikation ist die zuverlässige Authentifizierung eines Benutzers und seine Autorisierung für einzelne Operationen. Während bei monolithischen Systemen oft eine einfache Loginseite und eine Session-ID in einem Cookie ausreicht, um dies sicher und benutzerfreundlich umzusetzen, ist dies in einer modernen Microservice-Architektur nicht ganz so einfach.

Mehr erfahren