Modern Big Data Architectures. Dominik Ryzko

Чтение книги онлайн.

PREFACE

      Over several years of my career in IT, I have observed how various ideas and technologies have come and gone, taking different paths, from being new and innovative to maturity and adoption, only to be replaced by even newer concepts as they arrive. Some gained popularity very quickly and became the buzzwords of their time, something that everybody tries and claims to be doing. Such is the case of big data, the popularity of which skyrocketed and was embraced by research, industry, and governments alike. In 2012 the Obama Administration announced the Big Data Research and Development Initiative [2012] acknowledging it as a key enabler to accelerate the pace of discovery in science and engineering, strengthen our national security, and transform teaching and learning. Only recently has big data been overshadowed by the widespread adoption of artificial intelligence (AI), which by the way, builds on the foundations of big data. However, big data will remain strong for the foreseeable future.

      Other promising technologies from the past have not stood the test of time. The rise of statistical approaches to AI, and deep learning in particular in the past decade, gave the final blow to the symbolic methods, which I found elegant and fascinated me at the time of my undergraduate studies in the 1990s. Have the logical systems passed forever? Possibly not; after all, there are still open questions as on how humans analyze facts, reason, and make decisions, which we are not yet able to model purely by statistical methods. Only the future will show us in which direction science will progress.

      Another interesting story is related to Multi-Agent Systems (MAS), the field I picked for my PhD and later research. While the concept of an agent, or closely related actor, go way back to the 1970s, it never gained wide popularity outside the relatively narrow research community and some niche business applications. Despite bringing in innovative views on information system paradigms and promises of solving some of its challenges, the idea never became widely adopted.

As my focus started to switch toward industry projects and I became more and more involved with building large scale big data and AI systems, I observed that some of the fundamental assumptions behind MAS have made their way into mainstream information systems. Monolith systems, dominant at the beginning of my career, started giving way to services and later to micro-services. Physical devices distributed in the physical environment gained computational power and thus built-in intelligence and increased autonomy. These changes were happening not because MAS were gaining in popularity, but due to the fact that it made sense to solve real issues in this way.

      Finally, a few years back, I was able to formulate a more concrete conclusion, which can be used as a working thesis for this book – mainstream computer science is on a convergence path with multi-agent paradigms. Or to be more specific: the fundamental building blocks of modern information systems have been gaining the properties of those attributed to agents in MAS and thus the whole system has become more adaptive, autonomous, and intelligent. I decided to devote some time to studying these analogies, by comparing the fundamental assumptions and paradigms as well as by looking at the applications of MAS in solving various problems in the big data area. This book summarizes this research by taking a journey through modern big data architectures viewed through the eyes of the MAS domain.

      I hope the view taken in this book will be fresh and interesting and will inspire further critical thinking about the evolution of contemporary information systems and the direction they are heading.

      Dominik Ryżko

      Warsaw

      August 2019