Data Factory Scheduling

Often questions arise on how to correctly schedule Azure Data Factory pipelines. In this post we will look at some of the possible options and how they can affect your scheduling. The first two options revolve around changing when the datasets will become available for a pipeline to run. A pipeline may be set to…

1

Azure Data Factory – using a Stored Procedure as a destination

Azure Data Factory allows data to move from a multitude of sources to a multitude of destinations. A common usage case is to take on premise data and place it into a cloud database such as Azure SQL Database. Often this is done as a two-stage operation. Firstly, load the data into a data lake…

0

Improving query performance with OPTION (RECOMPILE), Constant Folding and avoiding Parameter Sniffing issues

OPTION (RECOMPILE) when used in the correct manner can give a performance boost though Constant Folding and also help deal with parameter sniffing issues. However, before going into OPTION (RECOMPILE) and how Constant Folding works first a bit of background. Plan Caching Issues The first time a stored procedure is run it will compile into…

4

Index Maintenance – Fragmentation should not be your only concern

Index fragmentation is not the only consideration when constructing an index maintenance plan. This article will look at other considerations that should be taken into account such as: Page fullness FILLFACTOR and PAD_INDEX The impact of Snapshot Isolation The less fragmented an index is, the more performant it will be. Inserting records inside an index…

2

Getting started with Hortonworks 2.4 in Hyper-V

Currently there is no 2.4 Hortonworks sandbox available for Hyper-V. If like me you spend a lot of time travelling with an intermittent network connection and would like to run locally then this article will show you how to set one up: Firstly, an image of the Hortonworks 2.4 sandbox is needed. There are two options…

0

Page Compression and the 4038 Length Limitation

While it is well documented that when applying page compression to a table only in-row data will be compressed, it is not so well known that only strings up a maximum length of 4038 can be compressed with page compression. If you have ever turned on page compression for large blobs of data and wondered…

0

CLRs, Web Services and JSON – Design Decisions

Recently I was working on a CLR that needed to talk to a web service and return a tabular dataset. Within the development and deployment process there were several considerations concerning security and design. This post will documented these to help understand the challenges and options available. Security problems should be considered early as they…

0

An Introduction to U-SQL in Azure Data Lake – Part 2

There has been a certain amount of speculation on what the U in U-SQL stands for. Some have suggested it is because U comes after T in the alphabet. This would suggest U-SQL is a successor to T-SQL which is not the case, you will not be writing your SQL Server stored procedures in U-SQL…

0

Architecting a Big Data Project in Azure

The aim of this post is to give a very brief overview of the technologies in Azure that can comprise a Big Data architecture and to help the reader understand the options out there. With the new data explosion over the last few years new methods and technologies have been developed to deal with data that…

0