How to Call a Python Program from SPL

How to Call a Python Program from SPL

Though esProc is a powerful computing engine, it is not good at handling machine learning algorithms. Python, however, is excellent in doing that. So esProc offers the YM external library to call a Python program in an esProc SPL program. That’s smart.

We’ll illustrate how to call a Python program from SPL in three aspects:

1. Standards and requirements in Python module development;
2. Interface call using ym_exec;
3. Uses of model building algorithm module.

The diagram shows relationships between the SPL program, the interface and the Python program:

The SPL program calls ym_exec interface to pass in a parameter to Python apply()interface. And apply() calls the Python program to execute and returns the result to SPL.

1.Standards and requirements in Python module development

A. def apply(ls)interface calls and executes a Python program and returns it to SPL program.

B. The list type parameter ls functions in the same way as parameter argv in Java entry interface void main(string argv[]).

C. The return value, which is of DataFrame structure and stored in the list type variable, can be viewed in SPL.

D. Below is a sample program ( of building a Python module:

import pandas as pd
import sys
def apply(lists):
  cols = ["value"]
  ls = []
  for x in lists:

df = pd.DataFrame(ls, columns=cols)
  return lls

if __name__ == "__main__":
  res = apply(sys.argv[1:])

Execution: python "AAA" "BBB" 1000


res=[  value
0    AA
1    BBB
2    1000]

The apply() interface adds the passed-in parameter to the variable list ls, puts ls in the DataFrame structure, and then places the dataframe in the to-be-returned variable list lls. Then we test the apply() interface in Python to make sure it operates well and then we can call the it in the SPL program.

Note: Dataframe is returned in msgpack format. This requires data in same column be of same type; otherwise errors will happen in masgpack serialization and SPL won’t receive the dataframe.

2.Interface call using ym_exec    

Format: ym_exec(pyfile, p1,p2,…)

The esProc interface function calls and executes the py file using passed-in parameters p1 and p2. The number of parameters vary according to those in apply() interface.

This interface needs to work with esProc external library pythonCli. The external library connexts to a Python program through userconfig.xml, whose configuration will be explained later.

A.Install Python:

Download Python 3.0 to install it in, for example, c:\Program Files\raqsoft\yimming\Python37.

B.Install esProc external library:

By default the external library is installed in esProc\extlib\pythonCli. Then select pythonCli on Select external libraries tab.

C.Configuring parameters:

Configure parameters in userconfig.xml under esProc’s external library directory (esProc\extlib\pythonCli\userconfig.xml):

sAppHomeC:\Program Files\raqsoft\yimmingapplication directory
sPythonHomec:\Program Files\raqsoft\yimming\
iPythonScriptPort8512Port number

The application is the Python service-side application:

After all configuration is done, restart esProc to employ the ym_exec() interface.

To call, for example:

2=ym_exec("d:/", false, 12345, 10737418240, 123.45, decimal(1234567890123456), "aaa 123")


6aaa 123

3. Uses of model building algorithm module

To call a Python Partial Least Squares algorithm (PLS, which esProc deosn’t offer) in SPL, first you need to install Yimming External Library. Configuration guide can be found in SPL Smart Modeling and Scoring.

The PLS algorithm contains complex parameters. We specify the invocation format to make it convenient:

ym_exec(pyfile, data, jsonstr)

The SPL program calls and executes pyfile; data is the table sequence for which model is built; the algorithm’s many parameters will be written in JSON strings and represented by parameter jsonstr. Make sure the parameters correspond to those in pyfile’s apply() interface handling to be correctly parsed.

data:Name of a data file over which scoring is to be performed or that has column headers. It includes the column where the target variable (target) settles.

jsonstr: JSON strings. For example:


target, which must not be absent, specifies the column holding the target variable.

SPL script (pls_demo.dfx):

3=file("d:/script/data_test.csv").import@cqt()//Data file
4{target:0,n_components:3,deflation_mode:'regression',mode:'A',norm_y_weights:False}//The first column is the target variable and parameters are written in JSON format
5=ym_exec(A2, A3, A4)

The data file (data_test.csv) where the first column is the target variable:


A sample of coding Python algorithm module (Take file for example)

from scipy.linalg import pinv2
import numpy as np
import pandas as pd
import demjson

#algorithm class pls_demo:
class pls_demo():
. . . . . . .

#interface implementation
def apply(lists):
    if len(lists)<2:
        return None
    data = lists[0] #data parameter
    val =  lists[1] #jsonstr string parameter
    if (type(data).__name__ =="str"):
        data = pd.read_csv(data)

#1. Handle special values in JSON strings
    val = val.lower().replace("false", "'False'")
    val = val.replace("true", "'True'")
    val = val.replace("none", "'None'")
    dic = demjson.decode(val)
    if dic.__contains__('target') ==False:
        print("param target is not set")

#2. Handle parameter target that is either column count or column name
    targ = dic['target']
    if type(targ).__name__ == "int":
        col = data.columns
        colname = col.tolist()[targ]
        colname = targ
    Y = data[colname]               
    X = data.drop(colname, axis=1)

# 3. Handle model building parameters, during which defaults should be set for those without passed-in values
    if dic.__contains__('n_components') :n_components=dic['n_components']
    else: n_components=15
    if dic.__contains__('deflation_mode') :deflation_mode=dic['deflation_mode']
    else: deflation_mode="regression"
    if dic.__contains__('mode'):mode=dic['mode']   
    else: mode="A"

# 4. Load algorithm module
    pls_model = pls_demo(n_components,
    # Training data, Y)
    # Scoring
    y_pred = pls_model.predict(X)

#5. Append return value
    f = ["value"]
    df = pd.DataFrame(y_pred, columns=f)
    return lls

#6. Test
if __name__ == '__main__':
    ls = []
    val = "{target:0,n_components:3,deflation_mode:'regression'," \
      "mode:'a',norm_y_weights:False," \
      "scale:False,algorithm:'nipals'," \


Leave a Reply