Skip to content

Posts from the ‘AWS’ Category

20
Nov

Get File from AWS S3 via Powershell

Not battle-tested at all.  Use at your own risk.  Works for me though.  You’ll need an $accessKey and $secretKey variables set outside the scope for this to work.  The $bucketName is the bucket that the file is located in.  The $fileName is the name of the file in said bucket.  The $dest is where you want to write the file locally.  The $logFile does nothing at this point.

function GetS3([string]$bucketName, [string]$fileName, [string]$dest, [string]$logFile)
{
	$AmazonS3 = [Amazon.AWSClientFactory]::CreateAmazonS3Client($accessKey, $secretKey)
	$S3GetRequest = New-Object  Amazon.S3.Model.GetObjectRequest
	$S3GetRequest.BucketName = $bucketName
	$S3GetRequest.Key = $fileName

	$S3Response = $AmazonS3.GetObject($S3GetRequest);
	if($S3Response -eq $null){
	    Write-Error "ERROR: Amazon S3 get requrest failed. Script halted."
	    exit 1
	}

	$stream = $S3Response.ResponseStream;
	$fileStream = new-object system.IO.FileStream($dest, [System.IO.FileMode]::Create, [System.IO.FileAccess]::Write);
	$byteArray = New-Object byte[] 32768
	[int]$bytesRead = 0;
	do
	{
		$bytesRead = $stream.read($byteArray, 0, $byteArray.length);
		$fileStream.write($byteArray, 0, $bytesRead);
	}
	while($bytesRead -gt 0)
	$fileStream.flush();

	$fileStream.close();
	$stream.close()

}
11
Oct

Testing Sql Server AlwaysOn in AWS

This is part deux from the post Configuring Sql Server AlwaysOn in AWS.  I’m calling it “Testing Sql Server AlwaysOn in AWS 2; The Reckoning”.

Keep in mind, this is a multi-subnet configuration.  This plays into the testing below, but it actually works out to our advantage.  I’ve not tested AlwaysOn in a single subnet configuration yet.

Once again, a shout out is deserved to Jeremiah Peschka (blog | twitter), as I’d never have gotten this far without this post.

So, now we have our listener set up, ip’s configured, and we’re ready to start seeing how well the failovers work.  Let’s start by pinging the listener from the primary server:

image

So far so good.  Let’s ping it from the Secondary:

image

Awesome.  Let’s ping it from the DNS server:

image

Looking good.

So, let’s check connections with the native client.  On the desktop of the Primary, right-click and create a file called test.udl and save it.  Next, double-click the file.  It should bring up a dialog like this:

image

Click on the Provider tab, and change the provider to SQL Server Native Client 11.0, click Next and fill out the connection information, entering in your listener name in the “Select or enter a server name” drop-down.  Next, select or type in a database name and click Test Connection.

image image

If you typed everything in correctly, you should see this:

image

You should get the same results on the secondary as well.

Now, let’s manually fail the database over and re-test.  Go on the primary server, right-click on the Availability Group and click Failover…

image

Go through the various steps required to failover the Availability group to the secondary server.  I’ll skip images and descriptions for this process, as this is not the main focus of this post.

So, the Availability Group has been failed over.  Let’s test the primary, secondary & dns server connectivity to the listener via ping again:

Secondary (was primary):
image
No Love (host unreachable)
Primary (was secondary):
image
Love
DNS Server:

image
No Love (timeout)

 

So, at this point only 1 out of 3 servers can connect.  Let’s check our UDL files again on the Secondary (used to be primary):

image

No love.  But wait!  Click on the All tab and find the property named Extended Properties, click Edit Value… and enter in MultiSubnetFailover=Yes.  Now, select the Connection tab and click Test Connection again.

image

 

Yay!

image

The reason this works, is that the provider will look for this property and will aggressively try both IP addresses listed in DNS.  Refer to the section Connecting with MultiSubnetFailover in this article for more details.

image

What I’ve found is the process AWS uses for DNS updating itself to point to the correct host is inconsistent at best.  Sometimes it will update itself after 5 minutes, sometimes 15, sometimes never.  It’s no big deal for clients that use the SQL Server Native Client connection with the MultiSubnetFailover=Yes setting, but for other clients it will definitely present a problem. 

Let’s try one more thing.  Let’s cause an automatic failover.  Let’s go into the AWS console and stop the instance that the current primary is running on.  In my case, the Primary is running on RHDB1. 

image

Go into the AWS EC2 dashboard and stop the instance.

image

Okay, automatic failover.  RHDB2 is now the primary. 

image

Ping from the DNS server resolves, the UDL resolves, pinging the listener resolves.  Just for grins, fire the other server back up and lets shut down the primary via the AWS console again.  Currently, the Primary is RHDB2 and RHDB1 is the secondary. 

image

Let’s kill RHDB2.

image

Automatic failover again:

image

Pinging the listener from RHDB1 works just fine.  The UDL connection also works fine.  Ping from the DNS server, not immediately, but it seemed to resolve after about a minute.

By no means is this a 100% thorough test, but my results are promising.  Automatic failover seems to work just fine in this case.  What doesn’t work reliably is the DNS resolving.  But, if you are using the MultiSubnetFailover when connecting with the sql server native client, it more or less negates the dns not resolving issues. 

10
Oct

Configuring Sql Server AlwaysOn in AWS

This post primarily concentrates on setting up the listener for a SQL Server Availability Group in AWS in a multi-subnet cluster.  Just a warning, the listener does not function in AWS as you would expect on local instances.  Don’t assume this article solves listener issues.  It does not.  I’ll outline what the issues are with the listener in a later post.

I’m assuming you’ve already done the following:

  1. Installed Sql Server 2012 on a couple of EC2 instances located in different subnets.
  2. Created a few test databases (and done full backups) on one of said database EC2 instances.
  3. Created an availability group hosting said databases.

Before I begin, a shout out is deserved to Jeremiah Peschka (blog | twitter), as I’d never have gotten this figured without this post.

Create the listener

In your availability group, create your listener

image

Enter in the Listener DNS Name, enter in the port (I left the default port of 1433 for this demonstration to keep it simple), and select StaticIP for the Network Mode. 

image

Next, at the bottom of the New Availability Group Listener dialog, click the Add button.  This will bring up the Add IP Address dialog.  Enter in your static IP addresses.  If these IP addresses already exist, you’ll get this error message:

The Windows Server Failover Clustering (WSFC) resource control API returned error code 5057.  The WSFC service may not be running or may not be accessible in its current state, or the specified arguments are invalid.  For information about this error code, see "System Error Codes" in the Windows Development documentation.
The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator.
The attempt to create the network name and IP address for the listener failed. The WSFC service may not be running or may be inaccessible in its current state, or the values provided for the network name and IP address may be incorrect. Check the state of the WSFC cluster and validate the network name and IP address with the network administrator. (Microsoft SQL Server, Error: 41009)

 

You’ll want to create new IP addresses for your listener that are not being used by the failover cluster itself.  The IP addresses will be added to DNS automatically, as will be illustrated further down in the post.  For example, currently on my servers I have addresses of 192.168.10.70 & 192.168.10.50 for one node, and 192.168.11.70 & 192.168.11.50 for the second node.  For my IP addresses for the listener, I will add IP addresses of 192.168.10.77 & 192.168.11.77.

Obviously, you have to run this as an account that has the proper permissions to add entries to DNS. In my case, I had god rights.  You’ll probably want to be a bit more restrictive. 

image

Next, if you look in your DNS, you’ll see it added the listener name with both IP’s. 

image

Next, go to your AWS console.  In the console under your EC2 Dashboard under Network & Security, click the Network Interfaces. 

image

Select the network interface that is associated with your AMI and click Manage Private IP Addresses.

image

Now add the IP address that the listener created to said network interface.  Do this for both IP addresses on each of the network interfaces.  Only add the .10 to the AMI in the .10 subnet, and add only the .11 to the AMI in the .11 subnet.  (It won’t allow you to add an 11 to the 10 subnet, but just trying to be clear.  Winking smile)

image

Your listener should now be ready for testing.  This is where the fun really begins.  I’ll split the testing out to another post, as this one is getting pretty big.