Category: 3 SPL Technology & Evaluation

1. Test task This task originates from the counting of the age (storage time) of products in the ERP system of a large electrical appliance enterprise. The original data consist of the following pair of tables that are in primary-sub relationship: Store: Field name Data type Field description store String Store ID product Integer Product ID total Integer Current stock Inbound: Field name Data type Field description store String Store ID product Integer Product ID indate Date Inbound date num Integer Inbound number Now we are to calculate the age (the number of days from the inbound date to an

Newly emerging database products often boast of their “fastest” and “faster” computing speeds, but they often fail to achieve satisfactory performance in practical applications. Therefore, conducting operational performance testing in advance is the key to successful database selection. Many users are not familiar with performance testing, which can lead to some misunderstandings. Here are some key considerations and suggestions: 1. We cannot be lazy and take blind trust in international standards. There are international standards for database operation testing in the industry. For example, the TPCH, in which the test cases are carefully designed, simple, and have a certain coverage

I. Test task Funnel analysis is a common statistical requirement in e-commerce business. When a user uses the smart device to shop, the system will establish a connection to form a session. Each session contains a number of events, such as visiting a website, browsing a product page, placing an order. The events of each user occur in a certain order; the later the event occurs, the fewer the number of users involved in the event, just like a funnel. The conversion funnel analysis is to count the number of distinct users of each event first, and then perform further

I. Test task This test is to calculate the distance between two objects based on their positions and then decide whether to establish an association relationship. This is a typical non-equivalence association operation, which cannot be optimized with HASH algorithm. This test originates from the celestial objects (“objects” for short) clustering operation of NAOC, and is conducted based on the simplified operation described below: There are n objects in a photo of objects, and each object has various properties. One object and its properties form a record, and all objects and their properties in a photo form one table.Now we

I. Test task In modern business intelligence, the index analysis page often presents multiple indexes at the same time. These indexes are mostly calculated from the same dataset (such as a wide table). This test is to aggregate (group and aggregate) the desired metric fields by multiple (possibly dozens) different dimensions based on the same filter condition. Based on the wide table described in SPL computing performance test series: associate tables and wide table, we test the performance in calculating one index, two index and three indexes respectively. SPL provides the multipurpose traversal mechanism, which can accomplish multiple grouping and

I. Test task Fact and dimension tables are common in multidimensional analysis. This test is to associate the fact table with multiple and multi-layer dimension tables and then aggregate by dimensions and, aggregate wide table by dimensions. Based on TPCH100G dataset, we design the following association operations between large fact table and multiple dimension tables: II. Technologies to be compared In this test, we only test SPL Enterprise Edition (version 20230528), and select the following two products to make a comparison: III. Test environment One physical server with the following configuration: 2 x Intel3014 CPUs, main frequency 1.7G, 12 cores

In the article Comparison test of Python and SPL in data reading and computing performance, we compare the performance of Python and SPL in data reading and computing. Since modifying dataset is common in daily data processing, we will compare the data processing performance of Python and SPL in this article. Test environment: System: Windows 11Memory: 16GCPU: 8 coresData: TPCH 1G The amount of data involved in this article is not particularly large and can be fully loaded into memory. We use the orderstable which is 1.5 million rows. The size of text file is 172M. The data row is

Test environment:System: CentOS7Virtual machine: VMWare 15.5.1build-15018445Memory: 32GCPU: 4Data: TPCH 1G Data reading There are two types of data sources: text file and database. It should be noted that the data involved in this article is not particularly large in volume and can be fully loaded into memory. The data is stored in an orderstable and the amount of data is 1.5 million rows. Text data The size of text file is 172M. The data row is split by symbol “|”, and presented in the following form: 1|18451|O|193738.97|1996-01-02|5-LOW|Clerk#000000951|0|nstructions sleep furiously among |2|39001|O|41419.39|1996-12-01|1-URGENT|Clerk#000000880|0| foxes. pending accounts at the pending, silent asymptot|3|61657|F|208403.57|1993-10-14|5-LOW|Clerk#000000955|0|sly final

I. Test task TPCH 100G. TPCH is an international standard, the details of which will not be explained in this article. It should be noted that although TPCH includes 22 queries, it still cannot fully reflect the response performance of the SUT (system under test) to actual business, for two main reasons: 1. The queries in TPCH are relatively conventional and do not involve order-related operations, and the stepwise operations are also fairly simple. In practice, however, there exist quires with performance bottleneck, and their complexity is usually much higher than the queries in TPCH, and a large number of

In data analysis, we usually encounter the data that is too big to fit in memory and has to be processed on hard disk. In this article, we’ll compare the calculation abilities of Python and SPL for such order of magnitude of data. As for bigger order of magnitude like PBs, it requires the distributed system to analyze the data, which is out of the scope of this article.