New Whitepapers: The Microsoft Data Platform
Thursday, March 17, 2016
After decades of dullness, data is back in vogue. As part of this, we're seeing an increasingly diverse set of data technologies available. Taken as a group, these technologies can be viewed as a platform for working with data.
I've written a set of three papers describing the Microsoft data platform today. Each paper covers the technologies for working with a specific kind of data--operational, analytical, or streaming--and each one is meant to be readable on its own. They're also meant to hang together as a group, which is why each one starts with the same big-picture diagram of this broad set of technologies. That diagram looks like this:
Each paper describes a particular column in this figure, and all three take a scenario-oriented view--they're not deep technology tutorials. The core audience is IT leaders, but I hope they're useful for anybody looking for a broad survey of what Microsoft offers today for working with data.
The papers, all sponsored by Microsoft, are available here:
SOA Lives! APIs and Microservices
Wednesday, February 17, 2016
A dozen years ago, service-oriented architecture (SOA) was all the rage. The idea of exposing application services in a standard way (which at the time meant via SOAP) was so attractive. Why not remake our software to reflect the then-new agreement on how applications should communicate?
But the SOA bubble burst pretty quickly. It turned out that solving the technical problem of communicating between software wasn't enough to solve the real problems. In particular, organizations had a very hard time agreeing on what services applications should expose, how those services should be versioned, and who should pay for what. Much like the software reuse bubble engendered by the advent of objects, and for many of the same reasons, the enterprise dream of universal integration through SOA didn't work out for most organizations.
Yet today, the descendants of SOA live on. Rather than focus on enterprise integration, each of these descendants picked up on a stream of SOA thought and took it further, eventually finding real success. The two most important of these are:
- API management, where cloud-based services provide a standard mechanism for exposing, managing, and controlling access to software of various kinds. The dominant protocol is now REST, not SOAP, but the idea has gone mainstream through offerings from smaller firms (e.g., Apigee) to big ones (e.g., Microsoft and Amazon). In fact, API management has become so important that CA thinks it's worth running ads in the New York Times to explain the idea to non-technical readers.
- Microservices, where applications are built from self-contained chunks of code that interact via interfaces.. Rather than the grand enterprise integration schemes that drove much of the original SOA hype, microservices are primarily about building a single application. This simplifies communication--you can often dispense with authentication, for example--while still providing a way to create manageable, easily deployable application components. Once again, the big vendors are here, providing technologies such as Microsoft's Service Fabric to support this approach.
When a new technology appears, it's always hard to know how best to use it. When SOAP first showed up, it kicked off the original SOA thrust toward enterprise integration. This was certainly a worthy goal, but over time, it's become evident that API management and microservices are the approaches that actually worked. It's also become apparent that the complexity of SOAP and its fellow travelers wasn't required--a RESTful approach (or with microservices, maybe something simpler) was usually good enough.
The startup that was SOA a dozen years ago has pivoted to become the much more successful API management and microservices of today.
New Whitepaper: Introducing Azure Machine Leaning
Wednesday, August 05, 2015
Machine learning has become a big deal. The rise of big data and the massive computing power made possible by cloud computing have made this set of technologies much more useful.
But machine learning isn't especially simple. While the basics are fairly straightforward, they're cloaked in odd terminology, phrases like "training data" and "supervised learning". For data scientists, people with years of specialized training, this isn't a problem. For non-specialists, though, the topic can be off putting.
To perhaps help with this, I've written a Microsoft-sponsored introduction to Azure Machine Learning (ML)
. The paper's subtitle is A Guide for Technical Professionals
, and that's exactly what it is: an introduction to machine learning for ordinary mortals. Azure ML is likely to become a broadly used technology, and so knowing the basics of machine learning is important. The paper's goal is to help you do this, using Azure ML as a concrete example.
New Whitepaper: Introducing Azure Search
Wednesday, April 15, 2015
For most of us, talking about search makes us think of Google (and maybe Bing). But for people who build applications, talking about search should bring something else to mind: the possibility of building a search box directly into a custom application's user interface. It's possible to do this with Google or Bing, but this approach has some limitations. Rather than relying on existing search services, creating a search UI for which you can control the results can have a lot of appeal.
One way to do this is to use Elasticsearch
. A simpler option, though, is to use a managed search service such as Microsoft's recently announced Azure Search. Azure Search isn't designed for end users. Instead, it's accessed by applications via a RESTful interface. The goal is to make it straightforward for developers to add search to the UI of the applications they build.
I've written a Microsoft-sponsored introduction to Azure Search, available here
, that explains why adding search to custom apps makes sense. The paper also walks through the basics of the technology, giving you a big-picture sense of what Azure Search does and how it works.
I don't know about you, but I love search UIs. If every application I use offered at least the option of search, I'd be a happy man. The availability of Azure Search is a step on the road to making this happen.
New White Paper: Understanding NoSQL on Microsoft Azure
Sunday, December 28, 2014
Strictly speaking, this isn't a new whitepaper--it's an update of an earlier paper I wrote on this topic. But Azure's native support for NoSQL has gotten so much broader that the paper
is almost entirely new.
The technologies it covers are:
- DocumentDB, Azure's document store
- Tables, Azure's key/value store
- HBase, Azure's column family store, and
- HDInsight, Azure's implementation of the Hadoop technology family.
As usual, my goal is to provide a big-picture introduction to these technologies. The paper won't provide details on how to use any of them, but I hope it will provide a place to start in deciding whether you need NoSQL and in choosing among the options.
If this sounds interesting to you, the paper is available here
The New Big Picture for Data
Friday, October 31, 2014
It's a heady time for data. We've seen more change in the last few years than in the previous couple of decades. Because of this, we need to think about data in some new ways.
For example, the traditional big-picture view of data technologies looks like this:
In this world view, the operational data that applications use is stored in a
relational database. Over time, that relational data gets loaded into a
relational data warehouse, where it becomes analytical data. Business
intelligence (BI) applications then use that analytical data to help
organizations make better decisions .
But things are changing. Here’s a more accurate big-picture view of
the data world today:
Increasingly, applications are using relational and NoSQL
databases for operational data. Turning this operational data into analytical
data implies having both a relational data warehouse and an unstructured data
lake. BI applications are then able to access both kinds of
data to help their users.
And there’s another new piece: search data. As search
services become more available (Amazon Web Services and Microsoft Azure both
provide them today), building search into every application gets easier. Users love search, and with a managed search service in the cloud,
the barrier to entry is significantly lower. But search data is different from
either operational data or analytical data—it’s a new category. Accordingly,
it’s staking out a new position in the data world.
Data technologies have shaken off decades of relational
torpor; lots of new things are happening. It’s time to look at this world in a