Category: 1 SPL Understanding

Wide tables are usually a standard component of the BI system. Many BI projects will first prepare wide tables at the beginning of construction. A wide table is formed by joining up multiple tables that have a certain association relationship. The result set does not conform to the normal forms; and there is a large amount of redundant data. Moreover, as wide tables need to be pre-created, they are not so flexible to use. But why do people very much prefer wide tables even if they have many shortcomings? Because wide tables are FAST! Querying data in the wide table

Global variable In SSC environment, sometimes information needs to be shared between computing tasks (QVM scripts) on multiple QVMs. In this case, SSC global variable (simply written as global variable in the following part) is useful. Managed by QVS, the global variable is available to every script run on each QVM applied through QVS. Qenv function is used to assign value to the global variable: Qenv(“gv1”,1) #gv1 is the variable name Qenv(“gv1”) The function is also used to obtain the global variable: Qenv(“gv1”) Unlike an ordinary SPL global variable, related data will be transmitted between QVS and QVM once the

An application on SSC has two parts: the server side, which consists of QVA and QVM, and the application side, which consists of the application itself (simply written as APP in the following part) and QVS. Both parts involve the use of SPL scripts. The SPL script on the server side is executed on QVM and thus called QVM script. On the application side, another SPL script is written to call the QVM script and it is called APP script. The basic procedure is like this: APP calls the APP script that sends a computation request to QVS; QVS then

Early applications usually only connected to one database, and calculations were also performed in the database, with little or no problem of mixed calculations from multiple data sources. The data sources of modern applications have become very rich, and the same application may also access multiple data sources, such as various SQL and NoSQL databases, text/XLS, WebService/Restful, Kafka, Hadoop…. Mixed computing on multiple data sources is an unavoidable problem that needs to be addressed. Direct hard coding to implement in applications is very cumbersome, and commonly used application development languages such as Java are not good at doing such things.

Customer profiling is very fashionable in current business analysis. Simply put, it means putting various tags on customers, using these tags to define different customer groups (so-called profiles), and then calculating the quantity (and changes) of customers in each customer group. Logically speaking, tags are dimensions or fields of a data table. Tags are fields with relatively simple values, and there are generally two types: binary tags, which have only two values and are usually represented by 0/1, such as marital status or gender. Another type is enumeration tags, with values ranging from a few to a few hundred, which

JSON is a good thing that can carry rich structured data information in a common text format. Many modern technologies prefer to use JSON as a data transmission format, such as Elastic Search, Restful, Kafka, etc. Mongodb, which is more concerned about performance, uses binary JSON. Structured data is often in bulk and often requires recalculation. However, JSON related class libraries are not very convenient to use for calculations. JSONpath is fine to parse JSON, but it doesn’t have much computing power. Simple filtering and aggregation are fine, but it cannot handle slightly complex operations such as grouping and summarization.

It is almost universally accepted in the industry and has been successfully practiced for many years that when the TP database is too bloated, then add an AP database. Is there anything to discuss? The use of AP database can indeed alleviate the computational pressure of the TP database, and the AP database usually has better computational performance and can provide users with a better experience, which is indeed a common practice in the database field. However, it does not mean that this road is incredibly smooth. The first is the cost. The operation and maintenance of databases have never

That’s right, it’s the famous JOIN. JOIN has always been a major challenge in database computing, and the industry has come up with many ways to calculate it. If no optimization is done, it would be to loop through and traverse two associated tables, which is a multiplication level complexity that can be unbearable if the data volume is slightly larger. Mature databases are certainly not so foolish. For the most common equivalence JOIN (with an association condition of equal key values), the HASH JOIN method is usually used, which can reduce the computational complexity by K times (K is

It is the very inconspicuous account de-duplication count, written in SQL as COUNT (DISTINCT…). Account de-duplication count is common and has important business significance in business analysis. The account here may be a user ID, bank account, phone number, license plate number…. The calculation logic is basically the same, which is to calculate how many accounts meet a certain condition from the historical data of a certain period of time. For example, how many cars have been to NewYork last month? How many phones had calls between 2:00 am and 4:00 am last week? How many bank accounts have received

Oracle is a widely used database, and it is not a professional analytical database. When the data volume is large, it often results in poor computational performance, which affects the user experience. Nowadays, there are many new analytical databases that are faster or even far superior in performance than Oracle. So when Oracle performance is insufficient, replacing to these databases seems to solve the problem. The matter is not so simple. Firstly, these new databases generally cannot completely replace Oracle. Oracle is not only used for OLAP, but also for OLTP, and these faster analysis databases do not have the