Comments on: Run Apache Spark on AKS with Azure Storage integration

By: NussNuss

NussNuss — Fri, 29 Jul 2022 08:42:44 +0000

Hello Tsuyoshi,
Thank you very much for your fast reply.
Your website is just excellent.
Thank you very much.
Have an excellent day.

LikeLike

By: Tsuyoshi Matsuzaki

Tsuyoshi Matsuzaki — Fri, 29 Jul 2022 08:26:44 +0000

In reply to NussNuss.

In the latest spark, it seems that you should explicitly assign sc (Spark context) as follows.
I have changed my source code in this post, and thank you for your feedback.

sc=spark.sparkContext

LikeLike

By: NussNuss

NussNuss — Thu, 28 Jul 2022 15:52:39 +0000

Hello,
Thank you very much for this excellent tutorial.
It’s perfect.
Anyway, I just have an error when I run the test.py code on my AKS cluster because this line :
model.save(sc, “wasbs://container01@dummyst.blob.core.windows.net/model/flightdelay”)
It seems that sc is unknown.
In fact it’s a Spark Context issue but I don’t know how I can resolve it.
Do you have any idea.
Thank you very much.
Have a great day.

LikeLike

By: grgreg

grgreg — Sat, 13 Nov 2021 23:55:29 +0000

You are still exposing your sas key for azure storage which seems not secured. But overall looks good.

LikeLike

By: Run Distributed Dask on Azure Kubernetes Service – tsmatz

Run Distributed Dask on Azure Kubernetes Service – tsmatz — Mon, 17 May 2021 01:30:42 +0000

[…] my last post, I showed you tutorial for running Apache Spark on managed kubernetes, Azure Kubernetes Service […]

LikeLike

By: Running Spark on Azure Kubernetes Service – Curated SQL

Running Spark on Azure Kubernetes Service – Curated SQL — Wed, 16 Dec 2020 13:10:29 +0000

[…] Tsuyoshi Matsuzaki walks us through running Apache Spark on Azure Kubernetes Service: […]

LikeLike