During the last couple of weeks, I’ve been thinking about how to deploy a web project that I’m working on. My preference was to use Azure Websites, so I don’t have to worry about infrastructure and I can focus only on developing my web app. My requirements are: MVC 5 website, Web API 2, SQL server and Elasticsearch cluster. I don’t want to expose this cluster to the public internet, so it has to be accessible only from my Web Server. (Note: this part could be replaced with MongoDB or any other NoSQL variant that you’d prefer to use, the architecture of your project would be similar).

In Azure there are 3 ways to deploy an app:

  1. Azure Websites: I believe that this is one of the most popular products in azure nowadays. You can use Git deployment, staging and production environments, fast deploy among other features, makes azure websites the default option for many people. I thought about this one too, but because I want my apllication to connect to my Elasticsearch cluster, this option doesn’t work for me. More about this in the next section.
  2. Cloud services: This option is similar to websites, you’re the owner of the VM that you’re using, and you have RDP access to each box too. You can also run start-up tasks if you need to. But you still have to worry only about your application code. You don’t need to manage the OS, system updates, patches etc. As you can see you have more “power” than using Websites, you might not need it. So,Why would I want to use Cloud Services instead of Websites? For me, I want to access to the elasticsearch cluster throught the local Azure Network.
  3. Virtual machines / AWS (IAAS): This gives you a Virtual machine with the OS of your choice, and you have to manage it all. You are the sys-admin here. I have to say that using reserved instances through AWS it’s cheaper than Azure. For me, it’s worth to pay the extra and use Cloud Services from Azure. If I was going to use only VMs, I’d probably stay with AWS.

How do I setup Cloud Services to communicate to my Elasticsearch cluster:

Short version:

  1. Create Virtual network.
  2. Create Cloud Service that will contain the Elasticsearch VMs that form the cluster inside that VN.
  3. Create Cloud Service that will contain my app, also inside the VN.

Because all the VMs are inside the same VN, they can communicate without having to open any ports to the public internet and that way I don’t have to add any security proxy to my Elasticsearch cluster. Note:I’m using Azure Cloud Services here, because at the moment you can’t add an Azure Website to a Virtual Network, so I couldn’t communicate my Website – Cluster privately.

Long version:

  1. Create your Virtual Network from Azure Portal.
  2. Crate a Cloud service where you’re going to add your Elasticsearch nodes. To create the first node, I’m using Ubuntu 14.0 LTS, with elastic-cloud-azure plugin. In Azure and AWS, multicast communication doesn’t work, so there are plugins to “discover” the instances in the cluster using their APIs. You can follow the documentation from elasticsearch-cloud-azure to create your VM image. Note on Cloud Service and VMs in Azure: I thought that in Azure, each VM was “wrapped” by a Cloud Service and there was relationship Cloud Service – VM (1 to 1). That’s wrong! You CAN have many VMs inside the same Cloud Service. When you create a Cloud Service, you get a URL / IP to access it. In Azure, when I was creating a Ubuntu box through the portal, by default adds a binding from port 22 (SSH) on the cloud service –> to –> port 22 in the instance. When I was trying to add another VM to the same Cloud Service I was getting an error, but it wasn’t easy to understand what was the problem. The problem was that por 22 in my Cloud Service was already being used by the first VM I created. But I am allowed to bind port 23 on the Cloud Service to port 22 in my instance. That way I could access to my instances like:

first-instance: ssh cloud-service.cloudapp.net -p 22
second-instance: ssh cloud-service.cloudapp.net -p 23

If you access to the list of instances running as Elasticsearch nodes, you’ll see that they have an internal IP like 10.x.x.x, they can be accessed by any VM inside the same Virtual Network.

  1. For the application layer, I created a new Cloud Service, where I deploy my Web Roles / Worker Roles (You can have a Web Role and Worker role in one instance if you need too. If you have a very small website or you want to do a few tests, you can even host your Web Role in a extrasmall instance for $15 a month). If I add this Cloud Service to the Virtual Network that I created for the project, the Web Role / Worker Role can access Elasticsearch to each node in the cluster using the internal IP likehttp://10.x.x.x:9200

That’s all, I’ll stick to Cloud Services for this project so I don’t have to maintain Windows servers / updates / etc.