Setting Up Spark for PyTest

Snippet shows how to set up and tear down a SparkSession for writing/running tests using pytest in Python. This prevents time from being wasted in setting up and tearing down a new SparkSession for each test!

import pytest
from pyspark.sql import SparkSession


@pytest.fixture(scope='session')
def spark(request):
    spark = (
        SparkSession
            .builder
            .appName('appname')
            .getOrCreate()
    )

    # stops SparkSession at end of testing ONLY
    request.addfinalizer(lambda: spark.stop())

    return spark