Přejít na menu

Class loading in Apache Spark

Správa článků

Vyhledávání Vyhledávání
20.3.2016 13:01
,
Počet přečtení: 405
Obrázek ke článku Class loading in Apache SparkWhere the Spark loads classes from? 

It is not guaranteed which classes/libraries/versions will be loaded. System jars from /usr/lib/spark and /usr/lib/hadoop are preferred over that included in fatjars.

Extreme example: you need artifact X in pom in version 3 (declared as NOT provided). This artifact is located in:

  • /usr/lib/hadoop/libX in version 2
  • /usr/lib/hadoop/another-libY, which is fatjar and contains also your X artifact in version 1
  • your fatjar created from pom - in version 3

The X artifact will be laoded from system jars in version 1 or 2 (it depends on classloader) and you will not get the desired version 3! If you want to precede the version from your fatjar, use userClassPathFirst option for spark-submit (see bellow).

Options for spark-submit

--jars

Include jars (jars must be present on driver only and will be distributed to executors)

Runtime config:

spark.driver.userClassPathFirst, spark.executor.userClassPathFirst - prefer classes from your jar

OSGI

Vytvořil 20. března 2016 ve 13:03:39 mira. Upravováno 3x, naposledy 26. března 2016 v 19:07:59, mira


Diskuze ke článku

Vložení nového komentáře
*
*
*