pyspark-profile-udf. Add simple udf profiler for pyspark.

Requirements
1. flameprof
```
pip install flameprof
```

How to use

Rename this project to pyspark
Compress this project to pyspark.zip: zip -r pyspark pyspark And move it to $SPARK_HOME/python/lib
Enable pyspark profile in your application code, and set the profile dump path

spark_session = SparkSession \
      .builder \
      .appName("Python Arrow-in-Spark profile") \
      .config("spark.python.profile", "true") \
      .config("spark.python.profile.dump", profile_dump_path) \
      .getOrCreate()

Name		Name	Last commit message	Last commit date
Latest commit History 27 Commits
ml		ml
mllib		mllib
python/pyspark		python/pyspark
sql		sql
streaming		streaming
testing		testing
tests		tests
.gitignore		.gitignore
README.md		README.md
__init__.py		__init__.py
_globals.py		_globals.py
accumulators.py		accumulators.py
broadcast.py		broadcast.py
cloudpickle.py		cloudpickle.py
conf.py		conf.py
context.py		context.py
daemon.py		daemon.py
files.py		files.py
find_spark_home.py		find_spark_home.py
heapq3.py		heapq3.py
java_gateway.py		java_gateway.py
join.py		join.py
profiler.py		profiler.py
rdd.py		rdd.py
rddsampler.py		rddsampler.py
resourceinformation.py		resourceinformation.py
resultiterable.py		resultiterable.py
serializers.py		serializers.py
shell.py		shell.py
shuffle.py		shuffle.py
statcounter.py		statcounter.py
status.py		status.py
storagelevel.py		storagelevel.py
taskcontext.py		taskcontext.py
traceback_utils.py		traceback_utils.py
util.py		util.py
version.py		version.py
worker.py		worker.py

shengjh/pyspark-profile-udf

Folders and files

Latest commit

History

Repository files navigation

pyspark-profile-udf. Add simple udf profiler for pyspark.

About

Resources

Stars

Watchers

Forks

Languages