Week 11

More on A deeper dive into Spark/Spark tips

Important

• April the 4th 2025 Sophie Germain 1005 13h30-15h30
• Calendar

Lecture : slides

• Spark tips
• Deeper dive

We may come back to several parts of

• Json format
• Spark high level APIs: SQL
• Spark low level APIs: RDD
• Python Data Science Stack
• Dask
• File formats

Notebooks

• Jupyter notebook VIII : wrangling
• webdata

You shall have gone through (on your own)

• Jupyter notebook VII : JSON
• html: JSON
  
• Jupyter notebook VI : Spark SQL
• Jupyter notebook V : Spark RDD
• html: Spark RDD
  
• Jupyter notebook I : tour of Python
• html: tour of Python
  
• Jupyter notebook II : tour of numpy
• html: tour of numpy
  
• Jupyter notebook III: tour of pandas
• html: tour of Pandas

References

• Spark Definitive Guide
• Spark project
• Spark documentation
• Data pipelines
• Next Generation Databases NoSQLand Big Data, Guy Harrison
• Mastering Spark on R

• Parquet
• ‘Arrow’
• Dask Docs
• Dask Examples
• Dask Code
• Dask Blog
• Pandas cheatsheet

Logistics

pyspark

To work the jupyter notebooks, install python 3, and modules related to jupyter: jupyter-cache, jupyter_client, jupyter_core, jupyterlab_widgets (this induces the installation of dependencies).

https://jupyter.org

Download the jupyter notebooks from notebooks listings.

If you do not already have an ENT account, follow instructions on Moodle to get one. You shall need this account to connect to PostGres cluster.

Back to Agenda ⏎