Cloud, Big Data, and Business Intelligence are the three buzz words of the decade. Everyone is talking about them. Everyone wants to do it. But no one tells you how to do it. How do you use the cloud to process your big data and build intelligence around it to make business decisions?

There are multiple answers to that question, and in this guide, we tried to answer that question using Microsoft’s cloud solution (Azure) and Microsoft’s BI tool (Power BI) to get you started in the right direction.

Microsoft Azure is one of the leading cloud solutions providers, offering…

Azure SQL Database and SQL Server 2016 (and later) supports an in-built feature for formatting query results as JSON. Then JSON support for Azure SQL data warehouse (currently Azure Synapse Analytics) was announced. Synapse Analytics supports querying and manipulating JSON data. However, there is no out-of-the-box support in Synapse to return SQL query results as JSON.

This article will create an HTTP triggered RESTful Azure Functions app with Node.js. This app will query Azure Synapse Analytics and convert and return the results as JSON.

Caution: Microsoft Azure is a paid service, and following this article can cause financial liability to…

Making Sense of Big Data

Azure Synapse Analytics SQL pool supports various data loading methods. The fastest and most scalable way to load data is through PolyBase. PolyBase is a data virtualization technology that can access external data stored in Hadoop or Azure Data Lake Storage via the T-SQL language.

PolyBase shifts the data loading paradigm from ETL to ELT. The data is first loaded into a staging table followed by the transformation steps and finally loaded into the production tables.

In this article, we load a CSV file from an Azure Data Lake Storage Gen2 account to an Azure Synapse Analytics data warehouse by…

Azure functions offer a great way to store and reference credentials, keys, and other secrets as application settings. Application settings are exposed as environment variables during execution. Application Settings are encrypted at rest and transmitted over an encrypted channel. However, you still run the risk of inadvertently exposing these secrets to unauthorized users.

A more favored approach to alleviate this issue is to store the credentials or keys in the Azure Key Vault as secrets and reference the secrets as environment variables in our Azure functions apps.

This article will set up an Azure functions app to access secrets stored…

Azure Data Factory is a great tool to create and orchestrate ETL and ELT pipelines. The Data Factory's power lies in seamlessly integrating vast sources of data and various compute and store components.

This article looks at how to add a Notebook activity to an Azure Data Factory pipeline to perform data transformations. We will execute a PySpark notebook with Azure Databricks cluster from a Data Factory pipeline while safeguarding Access Token in Azure Key Vault as a secret.

Caution: Microsoft Azure is a paid service, and following this article can cause financial liability to you or your organization.

Please…

Data is the blood of a business. Data comes in varying shapes and sizes, making it a constant challenging task to find the means of processing and consumption, without which it holds no value whatsoever.

This article looks at how to leverage Apache Spark’s parallel analytics capabilities to iteratively cleanse and transform schema drifted CSV files into queryable relational data to store in a data warehouse. We will work in a Spark environment and write code in PySpark to achieve our transformation goal.

Caution: Microsoft Azure is a paid service, and following this article can cause financial liability to you…

Azure Data Lake Storage and Azure Databricks are unarguably the backbones of the Azure cloud-based data analytics systems. Azure Data Lake Storage provides scalable and cost-effective storage, whereas Azure Databricks provides the means to build analytics on that storage.

The analytics procedure begins with mounting the storage to Databricks distributed file system (DBFS). There are several ways to mount Azure Data Lake Store Gen2 to Databricks. Perhaps one of the most secure ways is to delegate the Identity and access management tasks to the Azure AD.

This article looks at how to mount Azure Data Lake Storage to Databricks authenticated…

Azure SQL Servers comes packed with benefits and a twist when you need to access them outside the Azure network. With default settings, the Azure SQL Server firewall denies all access to the server.

This article looks at how to access Azure Synapse Analytics data warehouse from our client computer using SSMS and Databricks without revealing and storing our credentials by leveraging Azure Key Vault-backed Secret Scopes.

Caution: Microsoft Azure is a paid service, and following this article can cause financial liability to you or your organization.

At the time of writing, Azure Key Vault-backed Secret Scopes is in ‘Public…