Dominik Ryzko - Modern Big Data Architectures

1 COVER

2 LIST OF FIGURES

3 LIST OF TABLES

4 PREFACE

5 ACKNOWLEDGMENTS

6 ACRONYMS

7 CHAPTER 1: Introduction1.1 Motivation 1.2 Assumptions 1.3 For Whom Is This Book? 1.4 Book Structure

8 CHAPTER 2: Evolution of IT Architectures and Paradigms2.1 Evolution of IT Architectures 2.2 Actors and Agents 2.3 From ACID to BASE, CAP, and NoSQL – The Database (R)evolution 2.4 The Cloud 2.5 From Distributed Sensor Networks to the Internet of Things and Cyber-Physical Systems 2.6 The Rise of Big Data

9 CHAPTER 3: Sources of Data 3.1 The Internet 3.2 Scientific Data 3.3 Industrial Data 3.4 Internet of Things

10 CHAPTER 4: Big Data Tasks 4.1 Recommender Systems 4.2 Search 4.3 Ad-tech and RTB Algorithms 4.4 Cross-Device Graph Generation 4.5 Forecasting and Prediction Systems 4.6 Social Media Big Data 4.7 Anomaly and Fraud Detection 4.8 New Drug Discovery 4.9 Smart Grid Control and Monitoring 4.10 IoT and Big Data Applications

11 CHAPTER 5: Cloud Computing 5.1 Cloud Enabled Architectures 5.2 Agents and the Cloud

12 CHAPTER 6: Big Data Architectures 6.1 Big Data Computation Models 6.2 Publish-Subscribe Systems 6.3 Stream Processing 6.4 Higer Level Big Data Architectures 6.5 Industry and Other Approaches 6.6 Actor and Agent-Based Big Data Architectures

13 CHAPTER 7: Big Data Analytics, Mining, and Machine Learning 7.1 To SQL or Not to SQL 7.2 Big Data Mining and Machine Learning

14 CHAPTER 8: Physically Distributed Systems – Mobile Cloud, Internet of Things, Edge Computing 8.1 Mobile Cloud 8.2 Edge and Fog Computing 8.3 Internet of Things

15 CHAPTER 9: Summary

16 BIBLIOGRAPHY

17 INDEX

18 END USER LICENSE AGREEMENT

1 Chapter 3Table 3.1 Social media users.Table 3.2 The amount of data collected by 1 million metering devices in a yea...

2 Chapter 5Table 5.1 Cloud computing versus multi-agent systems

3 Chapter 6Table 6.1 Spark versus Flink versus Storm

1 Chapter 2 Figure 2.1 BI in monolith architecture. Figure 2.2 Data warehouse architecture. Figure 2.3 Akka actor hierarchy. Figure 2.4 BDI architecture. Figure 2.5 Vertical layered architecture. Figure 2.6 Horizontal layered architecture.

2 Chapter 3Figure 3.1 Semantic web stack.Figure 3.2 RDF graph.Figure 3.3 LOD cloud.Figure 3.4 The cost of sequencing per genome over time.Figure 3.5 The millenium simulation.Figure 3.6 Big data value potential index.

3 Chapter 4Figure 4.1 Search engine – indexing.Figure 4.2 Search engine – query.Figure 4.3 Ad-tech ecosystem.Figure 4.4 RTB message flow.

4 Chapter 5Figure 5.1 OpenStack MapFigure 5.2 Deployment method evolution.Figure 5.3 Auto scaling groups.Figure 5.4 Distributed DB architectures.Figure 5.5 HDFS architecture.

5 Chapter 6Figure 6.1 MapReduceFigure 6.2 Graph operation in DryadFigure 6.3 Finding minimal value with PregelFigure 6.4 Roaring bitmaps.Figure 6.5 Summary reduction.Figure 6.6 Kafka topic.Figure 6.7 Spark Streaming flow.Figure 6.8 Unbounded table.Figure 6.9 Flink architecture.Figure 6.10 Storm topology.Figure 6.11 Storm task grouping.Figure 6.12 S4 processing node.Figure 6.13 Mantis architecture.Figure 6.14 Spark stack.Figure 6.15 The lambda architecture.Figure 6.16 Architecture for multi-agent big data processing.Figure 6.17 Kappa architecture.Figure 6.18 Delta architecture.Figure 6.19 Realtime data processing at Facebook.Figure 6.20 Starfish ecosystem

6 Chapter 7Figure 7.1 Hive architecture (MapReduce)Figure 7.2 Tez data processing.Figure 7.3 High-level architecture of Kylin.Figure 7.4 Shark architecture.Figure 7.5 Spark SQL data flows.Figure 7.6 Spark SQL query planningFigure 7.7 Twitter ML architecture – integration of learners into Pig storag...Figure 7.8 Pipelined MapReduce.

7 Chapter 8Figure 8.1 Mobile cloud computing architecture.Figure 8.2 Mobile cloud computing architecture with backend and local clouds...Figure 8.3 Cloud of things architecture.Figure 8.4 IoTCloud architecture.

1 Cover

2 Table of Contents

3 Begin Reading

1 ii

2 iii

3 iv

4 ix

5 x

6 xi

7 xiii

8 xiv

9 xv

10 xvii

11 xviii

12 1

13 2

14 3

15 4

16 5

17 7

18 8

19 9

20 10

21 11

22 12

23 13

24 14

25 15

26 16

27 17

28 18

29 19

30 20

31 21

32 22

33 23

34 24

35 25

36 26

37 27

38 28

39 29

40 31

41 32

42 33

43 34

44 35

45 36

46 37

47 38

48 39

49 40

50 41

51 42

52 43

53 44

54 45

55 46

56 47

57 48

58 49

59 51

60 52

61 53

62 54

63 55

64 56

65 57

66 58

67 59

68 60

69 61

70 62

71 63

72 64

73 65

74 66

75 67

76 68

77 69

78 70

79 71

80 72

81 73

82 74

83 75

84 76

85 77

86 78

87 79

88 80

89 81

90 82

91 83

92 84

93 85

94 87

95 88

96 89

97 90

98 91

99 92

100 93

101 94

102 95

103 96

104 97

105 98

106 99

107 100

108 101

109 102

110 103

111 104

112 105

113 106

114 107

115 108

116 109

117 110

118 111

119 112

120 113

121 114

122 115

123 116

124 117

125 118

126 119

127 121

128 122

129 123

130 124

131 125

132 126

133 127

134 128

135 129

136 130

137 131

138 132

139 133

140 134

141 135

142 136

143 137

144 138

145 139

146 140

147 141

148 142

149 143

150 144

151 145

152 146

153 147

154 148

155 149

156 150

157 151

158 152

159 153

160 154

161 155

162 156

163 157

164 158

165 159

166 160

167 161

168 162

169 163

170 164

171 165

172 166

173 167

174 168

175 169

176 170

177 171

178 172

179 173

180 174

181 175

182 176

183 177

184 179

185 180

186 181

187 182

188 183

189 184

190 185

191 186

192 187

193 188

194 189

Founded in 1807, John Wiley & Sons is the oldest independent publishing company in the United States. With offices in North America, Europe, Asia, and Australia, Wiley is globally committed to developing and marketing print and electronic products and services for our customers' professional and personal knowledge and understanding.

The Wiley CIO series provides information, tools, and insights to IT executives and managers. The products in this series cover a wide range of topics that supply strategic and implementation guidance on the latest technology trends, leadership, and emerging best practices.

Titles in the Wiley CIO series include:

The Agile Architecture Revolution: How Cloud Computing, REST-Based SOA, and Mobile Computing Are Changing Enterprise IT by Jason Bloomberg

Architecting the Cloud: Design Decisions for Cloud Computing Service Models ( SaaS, PaaS, and IaaS) by Michael Kavis

Big Data, Big Analytics: Emerging Business Intelligence and Analytic Trends for Today's Businesses by Michael Minelli, Michele Chambers, and Ambiga Dhiraj

The CEO of Technology: Lead, Reimagine, and Reinvent to Drive Growth and Create Value in Unprecedented Times by Hunter Muller