Open Source ETL-The Prodigy Kids

By Binayaka Mishra

It’s said, ETL concept gained popularity in the 1970s when organizations began using multiple data repositories, or databases, to store different types of business information but only as a technical concept. The need to integrate data that was spread across these databases grew quickly. ETL became the standard method for taking data from disparate sources and transforming it before loading it to a target source, or destination. In the late 1980s and early 1990s, data warehouses came onto the scene. A distinct type of database, data warehouses provided integrated access to data from multiple systems – mainframe computers, minicomputers, personal computers and spreadsheets. Coupled with mergers and acquisitions, many organizations wound up with several different ETL solutions that were not integrated.

• Language: English
• Publisher: Binayaka Mishra
• Release date: Jun 19, 2017
• ISBN: 9781370352647

Binayaka Mishra

Binayaka Mishra is an experienced IT professional, in various tools and technologies like Data Warehousing, BigData Analytics, Cloud Computing, Reporting Analytics & Project Management documentation with 14+ years’of experience. He was Graduated in Computer Science & Engineering from National Institute Of Science & Technology, Berhampur, Odisha, India in 2002.He has worked in several critical roles with MNC’s like Tech Mahindra, Oracle Corporation, Wipro Technology,CapGemini UK,CapGemini India Pvt Ltd, UBS , AoN Hewitt Associates India Pvt Ltd, HMRC -UK and TUI Travel Plc -UK. Apart from technical details, his mastery are into functional domains like Payroll Processing, Tax Calculation, UK NI, BFSI,Telecommunication, Corporate Tax measurements divisions, Investment Banking, Automotive, Asset management , Security and Travel & Tourisim.Currently working as a Solution Architect / Project Manager in Tech Mahindra, India, loves to listen to music, play snooker, Bowling and a desperate swimmer like a shark.More Information could be found about him in his Linkedin Profile : https://www.linkedin.com/in/binayaka-mishra-b09612142/For any comments or advise, please feel free to write to: [e-mail]

Book preview

ACKNOWLEDGMENTS

This paper is for informational purposes only. THIS DOCUMENT IS PROVIDED AS IS WITH NO WARRANTIES WHATSOEVER, INCLUDING ANY WARRANTY OF MERCHANTABILITY, NONINFRINGEMENT, FITNESS FOR ANY PARTICULAR PURPOSE, OR ANY WARRANTY OTHERWISE ARISING OUT OF ANY PROPOSAL, SPECIFICATION, OR SAMPLE. The author of this paper disclaims all liability, including liability for infringement of any property rights, relating to use of this information. No license, express or implied, by estoppel or otherwise, to any intellectual property rights is granted herein. I would like to personally thank the various Informational Journals as furnished on Reference & Citation to aid me produce such quality content and also let me learn new ideas.

DEDICATION

Description: BigB

I Would like to dedicate this manuscript to my lovely parents, Mr. Baikuntheswari Prasad Mishra and Mrs. Sakuntala Mishra for their outstanding contribution to me at every poker moments of my life and lessoning to live with hope and dreams where no one else could aid. Not too stunningly mention, If I am being little eligible to write this book to represent before you, it’s become successful due to the painful journey of my parents which always inspired me.

Chapter 1: FAIRYTALE STYLE

For those who are confused about the headline of this article, please don’t be. The sole reason for the headline is to give you an idea about the kind of un-boundary world ETL universe possesses. Since the term ETL as we understand as Extract, Transform & Load and primarily used on the arena of DWH [Data Warehouse], it’s not so true since the time it was born. Although most of you would like to get many insight thoughts of ETL with DWH or BigData or Cloud or Business Intelligence or Data quality or Data Integration or may be other software tools, but the actual history of ETL is still a myth whether ETL was invented or discovered?It’s said, ETL concept gained popularity in the 1970s when organizations began using multiple data repositories, or databases, to store different types of business information but only as a technical concept. The need to integrate data that was spread across these databases grew quickly. ETL became the standard method for taking data from disparate sources and transforming it before loading it to a target source, or destination. In the late 1980s and early 1990s, data warehouses came onto the scene. A distinct type of database, data warehouses provided integrated access to data from multiple systems – mainframe computers, minicomputers, personal computers and spreadsheets. Coupled with mergers and acquisitions, many organizations wound up with several different ETL solutions that were not integrated. As most of us trying to follow Bill Inmon’s theory of ETL definition as proposed on 08th May 2008 on http://www.b-eye-network.com/view/6680 or its next modified version from Ralph Kimball, who is another legendary person on the DWH world, still there are various visions of ETL since ancient times 1970. The relational database revolution in the early 1980s ushered in an era of improved access to the valuable information contained deep within data. Still improvements were needed. It was soon discovered that databases modeled to be efficient at transactional processing were not always optimized for complex reporting or analytical needs. But the practice known today as Data Warehousing really saw its genesis in the late 1980s. An IBM Systems Journal article published in 1988, An architecture for a business information system[http://ieeexplore.ieee.org/document/5387658/], coined the term business data warehouse, although a future progenitor of the practice, Bill Inmon, used a similar term in the 1970s.

The abstract for the IBM article perfectly describes the problem and ultimate solution that spawned today’s modern data warehousing industry and ETL concept.The transaction-processing environment in which companies maintain their operational databases was the original target for computerization and is now well understood. On the other hand, access to company information on a large scale by an end user for reporting and data analysis is relatively new. Within IBM, the computerization of informational systems is progressing, driven by business needs and by the availability of improved tools for accessing the company data.[and] "It is now apparent that architecture is needed to draw together the various strands of informational system activity within the company. IBM Europe, Middle East, and Africa (E/ME/A)