pyspark-util

一组PySpark实用函数。

import pyspark_util as psu

data = [(1, 2, 3)]
columns = ['a', 'b', 'c']
df = spark.createDataFrame(data, columns)
prefixed = psu.prefix_columns(df, 'x')
prefixed.show()

# output:
+---+---+---+
|x_a|x_b|x_c|
+---+---+---+
|  1|  2|  3|
+---+---+---+

开发

设置

docker-compose build
docker-compose up -d

代码检查

docker exec psu-cnt ./tools/lint.sh

测试

docker exec psu-cnt ./tools/test.sh