Vyhledávání
Where the Spark loads classes from? It is not guaranteed which classes/libraries/versions will be loaded. System jars from /usr/lib/spark and /usr/lib/hadoop are preferred over that included in fatjars.
Extreme example: you need artifact X in pom in version 3 (declared as NOT provided). This artifact is located in:
- /usr/lib/hadoop/libX in version 2
- /usr/lib/hadoop/another-libY, which is fatjar and contains also your X artifact in version 1
- your fatjar created from pom - in version 3
The X artifact will be laoded from system jars in version 1 or 2 (it depends on classloader) and you will not get the desired version 3! If you want to precede the version from your fatjar, use userClassPathFirst option for spark-submit (see bellow).
Options for spark-submit
--jars
Include jars (jars must be present on driver only and will be distributed to executors)
Runtime config:
spark.driver.userClassPathFirst, spark.executor.userClassPathFirst - prefer classes from your jar