Therefore, it is important to know, how the number of partitions is decided based on certain parameters in case of an input RDD or a Dataset. The number of partitions decided in the input RDD/Dataset could affect the efficiency of the entire execution pipeline of the Job. These parameters carry a default value and can also be tweaked by the user. The number of partitions in an input RDD/Dataset (mapped to the data file(s)) is decided based on multiple parameters to achieve optimum parallelism. During the execution of a Spark Job with an input RDD/Dataset in its pipeline, each of the partition of the input RDD/Dataset is computed by reading the data as per the mapping of partition to the data file(s) The computed partition data is then fed to dependent RDDs/Dataset further into the execution pipeline. Users can then perform various transformations/actions on these inputs RDDs/Datasets.Įach of the partitions in an input raw RDD or Dataset is mapped to one or more data files, the mapping is done either on a part of a file or the entire file. These APIs abstract the reading process from data files to an input RDD or a Dataset with a definite number of partitions. To facilitate the reading of data from files, Spark has provided dedicated APIs in the context of both, raw RDDs and Datasets. The majority of Spark applications source input data for their execution pipeline from a set of data files (in various formats).
0 kommentar(er)
