We talk so much about programming. In an ecosystem, there are not only the source code but also the database. We have to admit that the data is very important. Almost the application we are programming is for supporting to manage data. "Data science, big data" are the hot keywords nowaday. Artificial Intelligence and Data have a tight relation.
A robot needs to learn from data in order to model them into policies. Any big domain we are working on such as Insurance, Logistic, Banking etc, we need data. Storing data is not a problem because we have hardware to handle that. The matter is how we design to manage data that works well with application server.
There are a lot of tools which have emerged in recent years. They solve variety of problems. Single database is no longer fit our system design. For example, we need a database server to manage the searching like Elasticsearch or Solr - separated from main database. We need to use Redis for message queue and many other purposes. Thus, how we make sure all of those things working well together is a matter.
There are a bunch of factors that may influence the design of a data system. However, we can concentrate on three concerns which are important in most software systems:
- Reliability, the system must work correctly, normally even in case of adversity (issues occur at a part of system or the human error).
- Scalability, we design a evolving system so that we have to make sure that we leave a room for evolving. Our system sooner or later will grow. We should be aware of that to make a good design.
- Maintainability, we can't avoid that many developers or designers will work on our system in future. They are not the same people, not have the same thoughts and levels. We must design somehow get easy for understanding and maintenance.
What is the reliability? Do you trust you system? Do you have any evidence to make the team trust in our system? What are the criterion?
- System prevents any unauthorized access.
- It has good enough performance.
- It tolerates the human error in unexpected ways.
- Sure! It works like the user expected.
That is why we say working correctly. Sometimes I heard people said "because the users didn't use the system correctly". It was so funny. The system should continue to work correctly even something went wrong.