When you want to see the bottlenecks in your code on Apache Spark, you can use the detailed logs with Spark event logs or REST API.
By the integration with your notebooks and your programming code, sparkMeasure simplifies your works for these logging and analyzing in Apache Spark.
In this post, I briefly show you how to use sparkMeasure.
Tsuyoshi Matsuzaki
Japanese, Hobby runner, and Explainer of computer science
(Japanese name : 松崎 剛)
Optimized Read-Throughput by Azure Cosmos DB Spark Connector
In this post, we see how it efficiently works behind Azure Cosmos DB Spark Connector (azure-cosmosdb-spark). By knowing the mechanism of connector, please optimize the read throughput with Apache Spark and Cosmos DB.
Shor’s Algorithm – Quantum Period Finding (Q#)
For the last post of my quantum programming series, I show you the most famous quantum algorithm, Shor’s algorithm, with Q# programming.
You can also solve integer factorization with polynomial time computation (in the input’s bit size) using this quantum algorithm, though classical one needs exponential time.
Quantum Arithmetic – Adder, Multiplier, and Exponentiation (Q#)
In this post, I show you several arithmetic implementation (circuit), such as addition, subtraction, multiplication, and exponentiation, using Quantum Fourier Transform (QFT).
Quantum Fourier Transform & Quantum Phase Estimation (Q#)
In this post I explain the outline of Quantum Fourier Transform and Quantum Phase Estimation algorithm and see the programming example with Q#.
Q# provides high level operator for both Quantum Fourier Transform and Phase Estimation, but in this post, we implement these algorithms with primitive operators for the purpose of your learning.
Run benchmark for Apache Hive LLAP on Microsoft Azure
In this post, I show you benchmarks for Apache Hive LLAP on Azure HDInsight. You can quickly start and see how LLAP is different with regular Hive (container on Tez) using managed service cluster.
Grover’s Quantum Search Algorithm (Q#)
In this post I explain the outline of Grover’s quantum search algorithm and see the programming example with Q#.
You could find it’s effective algorithm compared with classical brute-force search.
Spark ML Serving with Azure Machine Learning
Using Azure Machine Learning service, you can train the model on the Spark-based distributed platform (Azure Databricks) and serve your trained model (pipeline) on Azure Container Instance (ACI) or Azure Kubernetes Service (AKS).
In this post, I show you this step and background using AML Python SDK.
Quantum Algorithm for Beginners (Q#)
In order to solve the real problem with quantum computing, it’s also important to understand algorithms as well as quantum logic gates.
Here I show primitive programming sample to solve some problem for your very beginning and introductions.
MXNet Distributed Training with Azure ML (Custom Configuration Sample)
In this post, I proceed to more advanced topics by showing you how to set up (customize) your Azure Machine Learning Compute (AmlCompute) for the practical training. In the last part of this post, I’ll show you Apache MXNet distributed training example with Azure Machine Learning service.
tsmatz