Azure PowerShell–The missing links

I attended Campus Days again this year, and doing one of the sessions one of the presenters told everyone you need to buy 3rd party products to take snapshots or copy Storage Blobs from one container/account to another. That doesn’t seem fair. I needed this functionality and have created a PowerShell module that can do all that, and a lot more, so I decided to share that, with the world.

I think its important to understand the reason I wrote this before I get flamed with comments and emails. When I began playing around with azure, doing a proof of concept setup together with a guy from Microsoft Denmark, I started creating a PowerShell module to manage it, but quickly came across a module that someone else had written than already did most of what I needed, so abandoned the project again. Several months later, when I was asked to start doing some more work with Azure, especially Persistent VM’s I found that Microsoft had adopted the PowerShell module that I had once used, and in that process removed several key features that I needed (but was once there) I also noticed the module was painfully slow to work with when handling Persistent VM’s, so I started writing my own. Half way though that process I got bored with it, and started mixing Microsoft's PowerShell with my own, just focusing on the things they miss. So you will find a lot of useful stuff, and some redundant stuff. In a real scenario you would probably end up using both (for instance, I've only written half of the code to create new VM’s, and that sure is is an important command, right ? ;-) )

Download the module and extract it somewhere, like c:\wa  Open PowerShell and load the module with

# If you just downloaded it, unblock files
Unblock-File C:\wa\Release\*
import-module C:\WA\Release\wa.psd1

First of all, I hate the way Microsoft handles working with different Subscriptions. (granted some of mine ended up working the same way in the end, but still ). So you load the module and connect to one of your subscriptions with

# Find your Management Certificate, or create one and upload it to azure
dir cert:\CurrentUser\my

# Tell WA module about the subscription and cert and give it a name
Select-Subscription -Name Pay-As-You-Go -SubscriptionId 1f6b36b5-d2ab-49e6-8674-2155a3ea4a55 -CertificateThumbprint AAF4E3017B3DE26E93601E1573FDAF44A3AA15E8

The module loads certificates from localmachine and localuser store, but most will have it in the user\my store. Name can be what ever makes sense to you, I normally use the name Azure also used within the management portal. This gets saved in HKLU\software\wa Registry and from now on, every time you start a new PowerShell prompt you can just use

Select-Subscription -Name Pay-As-You-Go 
# or list all, known subscription with
Get-Subscription
# This will also query azure about statistics data, so you can see if your about to hit your Core, HostedService or Storage Account limits.

If you only want to manage an storage account, you can ignore the above and connect directly to it, using

Select-StorageAccount -AccountName skadefro -AccountKey blahBlagBlah1234==

But if you loaded an subscription, you can let the module handle all that for you and just use the Storage account’s Service Name.

Select-StorageAccount -ServiceName Skadefro

Ok, So what can you do, that Microsoft’s version is missing. Well, managing Storage/Blob/Queue’s was the first show stopper for me, so now you can Create/modify/delete tables, select, insert, modify,delete rows from tables. You can add/update/remove blob containers, update ACL on containers and blobs. Add/Upload/download/snapshot/signature/lease/unlock/remove/copy blobs, both as Page and Block blobs. ( uploading is insanely fast with Page blob, and if you really want to tweak your VM’s, making sure page size fits the size you format your disks with makes a huge difference. )

The reason I’m sharing all this, is several people asked about snapshot’s and copying content from one storage account to another ( hell, even my favorite azure application Azure Explore, doesn’t support copying blobs from one container to another ) ill give a few examples of that.

First, snapshots. very simple

$storageacountname = 'wingu2'
$containername = 'public'
$filename = 'test.txt'
# select an StorageAccount if you got more than one.
# if you only have 1, that will always be selected
Select-StorageAccount -ServiceName wingu2
# Lets upload a text file
# we should place that in a container
$container = Get-Container $containername
if(!$container) { $container = New-Container $containername }
# Lets make it public, so we can show cross subscription/account 
# copying later 
if( (Get-ContainerACL $containername) -ne 'container'){ Set-ContainerACL $containername container }

# Create a test file, and upload it
if ( (Test-Path $filename) -eq $true) { Remove-Item $filename -confirm:$false }
$file = New-Item -ItemType file $filename
add-content $file '1. Hello world'
New-Blob $containername $filename
$blob = Get-Blob $containername $filename

#Now, lets take a snapshot
$snapshot = $blob | New-Snapshot

#Lets update the file
add-content $file '2. Hello world'
New-Blob $containername $filename

# This will return 1 result
Get-Blob $containername $filename

# This will return 2 result
Get-Blob $containername $filename -Snapshots

# Remove the snapshot, supply snapshot time or pipe the Object
#$snapshot | Remove-Blob

# Removing the file will also fail, since we have a snapshot
Get-Blob $containername $filename | Remove-Blob

# But we can force, deleting all snapshot with 
Get-Blob $containername $filename | Remove-Blob -deletesnapshots

Next up, is copy blobs across containers or even storage accounts. If you want to move everything from one account to another, here's a crud but efficient way

Select-Subscription Pay-As-You-Go
Select-StorageAccount -ServiceName wingu2
$containers = @{}
foreach($container in Get-Container){
	$containers += @{$container.Name=Get-Blob $container.Name}
	# Could even update it to public and close it again after with
	# Set-ContainerACL $container.Name container 
}

Select-Subscription skadefro
Select-StorageAccount -ServiceName copytest
foreach($container in $containers.GetEnumerator()){ 
	$c = get-container $container.Name
	if(!$c){ $c = New-Container $container.Name }
	
	foreach($blob in $container.Value){
		Write-Host ("Copying '" + $blob.Name + "' to container '" + $c.Name + "'")
		Copy-Blob -Uri $blob.Uri -Destinationuri ($c.Uri.ToString() + '/' + $blob.Name)
	}
}

When I showed this to a friend, he quickly asked, what if I don’t want to make it public ? Well, we can use shared signature’s for that, so here’s another way, without requiring the containers to be public.

Select-Subscription Pay-As-You-Go
Select-StorageAccount -ServiceName wingu2
$containers = @{}
foreach($container in Get-Container){
	$blobs = @{}
	foreach($blob in (Get-Blob $container.Name)){
		# By default, this cmdlet creates a 55 minute readonly adhoc signature
		$url = $blob | Get-BlobSignature 
		$blobs += @{$blob.Name=$url}
	}
	$containers += @{$container.Name=$blobs}
}

Select-Subscription skadefro
Select-StorageAccount -ServiceName copytest
foreach($container in $containers.GetEnumerator()){ 
	$c = get-container $container.Name
	if(!$c){ $c = New-Container $container.Name }
	
	foreach($blob in $container.Value.GetEnumerator()){ 
		Write-Host ("Copying '" + $blob.Name + "' to container '" + $c.Name + "'")
		Copy-Blob -Uri $blob.Value -Destinationuri ($c.Uri.ToString() + '/' + $blob.Name)
	}
}

Another feature that is missing from Microsoft's azure module is the ability to work with tables and queues. So now you can use your favorite PowerShell prompt and work with them there (if that rocks your boat)

Select-StorageAccount -ServiceName wingu2
$tablename = 'testtable'
$table = Get-Table $tablename
if(!$table){ 
	New-Table $tablename 
} else {
	Remove-Table $tablename 
	New-Table $tablename 
}

# add using hashtable
New-Row $tablename  -Columns @{ ID=1; Name='John Doe' }

# add using PSObject
$row = New-Object PSObject -Property @{ ID=2; Name='Jane Doe'; upn='jd@identity.local' }
New-Row $tablename $row

# add overriding RowKey
New-Row $tablename  -Columns @{ ID=3; Name='Little Doe'; City='Aarhus' }

Get-Row $tablename 

# We can search by Row Key and/or Partition key. Or just use PowerShell
$row = Get-Row $tablename | ?{$_.ID -eq 3}
# Add Age to the object
$row | Add-Member -Name 'Age' -Value 33 -MemberType Noteproperty
# Update row
$row | Set-Row $tablename

# Create new, based on another 
$row.ID = 4
$row.Name = 'Bill Doe'
$row.upn = 'bill@identity.local'
$row | New-Row $tablename

Get-Row $tablename 

Lastly, I short list of all the commands. Source code will be uploaded once I cleaned it up a bit, and added some help texts and such. Drop me an email if your interested before I'm done (its not highest on my priority list at the moment). Should probably also add some logic to handle async blob copy’s etc. you know.

Copy-WAStorageBlob
Get-WADeployment
Get-WADisk
Get-WAHostedService
Get-WALocation
Get-WAOperationStatus
Get-WAOS
Get-WAOSFamily
Get-WAOSImage
Get-WAServiceCertificate
Get-WAStorageAccount
Get-WAStorageAccountKeys
Get-WAStorageBlob
Get-WAStorageBlobLease
Get-WAStorageBlobSignature
Get-WAStorageContainer
Get-WAStorageContainerACL
Get-WAStorageContainerSignature
Get-WAStorageRow
Get-WAStorageTable
Get-WASubscription
Get-WASubscriptionCertificate
Get-WAVM
Get-WAAffinityGroup
New-WADeployment
New-WADisk
New-WAHostedService
New-WAOSImage
New-WAServiceCertificate
New-WAStorageAccount
New-WAStorageAccountKeys
New-WAStorageBlob
New-WAStorageBlobLease
New-WAStorageBlobSnapshot
New-WAStorageContainer
New-WAStorageRow
New-WAStorageTable
New-WASubscriptionCertificate
New-WAAffinityGroup
Remove-WADeployment
Remove-WADisk
Remove-WAHostedService
Remove-WAOSImage
Remove-WAServiceCertificate
Remove-WAStorageAccount
Remove-WAStorageBlob
Remove-WAStorageBlobLease
Remove-WAStorageContainer
Remove-WAStorageRow
Remove-WAStorageTable
Remove-WASubscriptionCertificate
Remove-WAAffinityGroup
Select-WAStorageAccount
Select-WASubscription
Set-WADeployment
Set-WADeploymentStatus
Set-WADisk
Set-WAHostedService
Set-WAOSImage
Set-WAStorageAccount
Set-WAStorageBlob
Set-WAStorageContainerACL
Set-WAStorageRow 

# aliases registed though module manifest file in wa.ps1
New-Alias -name Select-Subscription -value Select-WASubscription
New-Alias -name Get-Subscription -value Get-WASubscription
New-Alias -name Get-OperationStatus -value Get-WAOperationStatus
#New-Alias -name Get-Location -value Get-WALocation

New-Alias -name Get-Deployment -value Get-WADeployment
New-Alias -name New-Deployment -value New-WADeployment
New-Alias -name Set-Deployment -value Set-WADeployment
New-Alias -name Set-DeploymentStatus -value Set-WADeploymentStatus
New-Alias -name Remove-Deployment -value Remove-WADeployment

New-Alias -name Get-AffinityGroup -value Get-WAAffinityGroup
New-Alias -name New-AffinityGroup -value New-WAAffinityGroup
New-Alias -name Set-AffinityGroup -value Set-WAAffinityGroup
New-Alias -name Remove-AffinityGroup -value Remove-WAAffinityGroup

New-Alias -name Get-Disk -value Get-WADisk
New-Alias -name New-Disk -value New-WADisk
New-Alias -name Remove-Disk -value Remove-WADisk
New-Alias -name Set-Disk -value Set-WADisk

New-Alias -name Get-OSImage -value Get-WAOSImage
New-Alias -name New-OSImage -value New-WAOSImage
New-Alias -name Set-OSImage -value Set-WAOSImage
New-Alias -name Remove-OSImage -value Remove-WAOSImage

New-Alias -name Get-VM -value Get-WAVM


New-Alias -name Get-ServiceCertificate -value Get-WAServiceCertificate
New-Alias -name New-ServiceCertificate -value New-WAServiceCertificate
New-Alias -name Set-ServiceCertificate -value Set-WAServiceCertificate
New-Alias -name Remove-ServiceCertificate -value Remove-WAServiceCertificate

New-Alias -name Get-Certificate -value Get-WASubscriptionCertificate
New-Alias -name New-Certificate -value New-WASubscriptionCertificate
New-Alias -name Remove-Certificate -value Remove-WASubscriptionCertificate

New-Alias -name Get-HostedService -value Get-WAHostedService
New-Alias -name New-HostedService -value New-WAHostedService
New-Alias -name Remove-HostedService -value Remove-WAHostedService
New-Alias -name Set-HostedService -value Set-WAHostedService

New-Alias -name Get-StorageAccount -value Get-WAStorageAccount
New-Alias -name Get-StorageAccountKeys -value Get-WAStorageAccountKeys
New-Alias -name New-StorageAccount -value New-WAStorageAccount
New-Alias -name New-StorageAccountKeys -value New-WAStorageAccountKeys
New-Alias -name Remove-StorageAccount -value Remove-WAStorageAccount
New-Alias -name Select-StorageAccount -value Select-WAStorageAccount
New-Alias -name Set-StorageAccount -value Set-WAStorageAccount

New-Alias -name Get-Blob -value Get-WAStorageBlob
New-Alias -name Get-BlobLease -value Get-WAStorageBlobLease
New-Alias -name Get-BlobSignature -value Get-WAStorageBlobSignature
New-Alias -name New-Blob -value New-WAStorageBlob
New-Alias -name New-BlobLease -value New-WAStorageBlobLease
New-Alias -name New-Snapshot -value New-WAStorageBlobSnapshot
New-Alias -name Copy-Blob -value Copy-WAStorageBlob
New-Alias -name Remove-Blob -value Remove-WAStorageBlob
New-Alias -name Remove-BlobLease -value Remove-WAStorageBlobLease
New-Alias -name Set-Blob -value Set-WAStorageBlob

New-Alias -name Get-Container -value Get-WAStorageContainer
New-Alias -name Get-ContainerACL -value Get-WAStorageContainerACL
New-Alias -name Get-ContainerSignature -value Get-WAStorageContainerSignature
New-Alias -name New-Container -value New-WAStorageContainer
New-Alias -name Remove-Container -value Remove-WAStorageContainer
New-Alias -name Set-Container -value Set-WAStorageContainer
New-Alias -name Set-ContainerACL -value Set-WAStorageContainerACL

New-Alias -name Get-Row -value Get-WAStorageRow
New-Alias -name New-Row -value New-WAStorageRow
New-Alias –name Set-Row –value Set-WAStorageRow


New-Alias -name Remove-Row -value Remove-WAStorageRow

New-Alias -name Get-Table -value Get-WAStorageTable
New-Alias -name New-Table -value New-WAStorageTable
New-Alias -name Remove-Table -value Remove-WAStorageTable