How to clone a Azure VM

Standard

run on your sample-vm

waagent -deprovision+user

run on your azure-cli env

$rgName = "VMTestGroup"
$template = "Template-test.json"
$vmName = "VMTest"
$vhdName = "VHDTest"
 
azure vm deallocate -g $rgName -n $vmName
azure vm generalize $rgName -n $vmName
azure vm capture $rgName $vmName $vhdName -t $template
 
# the $template should looks like this. and you have to change 'newvmname' before use.
...
         "storageProfile": {
          "dataDisks": [
            {
              "caching": "ReadOnly",
              "vhd": {
                "uri": "https://yourdiskname.blob.core.windows.net/vhds/dataDisk-0.newvmname.vhd"
              },
              "image": {
                "uri": "https://yourdiskname.blob.core.windows.net/system/Microsoft.Compute/Images/vhds/yourcapturedvmname-dataDisk-0.ff60129b-...3cf59bf9315a.vhd"
              },
              "createOption": "FromImage",
              "name": "yourcapturedvmname-dataDisk-0.ff60129b-4ec5-4dcd-ae97-3cf59bf9315a.vhd",
              "lun": 0
            }
          ],
          "osDisk": {
            "caching": "ReadWrite",
            "vhd": {
              "uri": "https://yourdiskname.blob.core.windows.net/vhds/osDisk.newvmname.vhd"
            },
            "image": {
              "uri": "https://yourdiskname.blob.core.windows.net/system/Microsoft.Compute/Images/vhds/yourcapturedvmname-osDisk.ff60129b-...3cf59bf9315a.vhd"
            },
            "createOption": "FromImage",
            "name": "yourcapturedvmname-osDisk.ff60129b-4ec5-4dcd-ae97-3cf59bf9315a.vhd",
            "osType": "Linux"
          }
        },
...
 
 
azure group deployment create $rgName MyDeployment -f Template-test-modified.json
    info:    Executing command group deployment create
    info:    Supply values for the following parameters
    vmName: NewVmName
    adminUserName: username
    adminPassword: password
    networkInterfaceId: /subscriptions/61719d1b-...ab74b6f77865/resourceGroups/VMTestGroup/providers/Microsoft.Network/networkInterfaces/YourNetworkInterfaceName
 
#If you do not have an existing NetworkInterface, you need create first. 
azure network nic create $rgName YourNetworkInterfaceName -k default -m YourSubnetVnetName  -l "westus2"

Elasticsearch cluster on Azure

Standard
#3 VM on Azure
#node1 10.0.0.3
#node2 10.0.0.4
#node3 10.0.0.5
 
apt-get update;
apt-get install default-jdk;
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -;
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list;
apt-get update && sudo apt-get install elasticsearch;
update-rc.d elasticsearch defaults 95 10;
 
apt-get install lvm2
fdisk /dev/sdc
pvcreate /dev/sdc1
pvdisplay 
vgcreate VolGroup00 /dev/sdc1
vgdisplay 
lvcreate -L 500GB -n lvData VolGroup00
lvdisplay 
mkfs -t ext4 /dev/VolGroup00/lvData
mkdir /data
mount /dev/VolGroup00/lvData /data/
blkid
  /dev/mapper/VolGroup00-lvData: UUID="b65c5a78-e078-4ca8-8119-2de94a414002" TYPE="ext4" 
cat /etc/fstab
  UUID=b65c5a78-e078-4ca8-8119-2de94a414002  /data   auto    defaults,nobootwait,nosuid,noexec,noatime,nodiratime    0 0
 
cat /etc/elasticsearch/elasticsearch.yml 
network.host: 0.0.0.0
cluster.name: es-cluster
node.name: node?
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.3"]
index.number_of_shards: 3
index.number_of_replicas: 1 
path.data: /data
 
mkdir /data/es-cluster
chown elasticsearch.elasticsearch /data/es-cluster
 
cat /usr/share/elasticsearch/bin/elasticsearch.in.sh
...
if [ "x$ES_MIN_MEM" = "x" ]; then
    ES_MIN_MEM=?g
fi
if [ "x$ES_MAX_MEM" = "x" ]; then
    ES_MAX_MEM=??g
fi
...
 
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head
/usr/share/elasticsearch/bin/plugin install analysis-smartcn
/usr/share/elasticsearch/bin/plugin install analysis-kuromoji
/usr/share/elasticsearch/bin/plugin list
 
/etc/init.d/elasticsearch start
移动(move)
把分片从一节点移动到另一个节点,可以指定索引名和分片号。
 
取消(cancel)
取消分配一个分片,可以指定索引名和分片号。
node参数可以指定在那个节点取消正在分配的分片。
allow_primary参数支持取消分配主分片。
 
分配(allocate)
分配一个未分配的分片到指定节点,可以指定索引名和分片号。
node参数指定分配到那个节点。
allow_primary参数可以强制分配主分片,不过这样可能导致数据丢失。
 
curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
    "commands" : [ {
        "move" : 
            {
              "index" : "索引名称", "shard" : 分片号, 
              "from_node" : "节点名称A", "to_node" : "节点名称B"
            }
        },
       "cancel" : 
            {
              "index" : "索引名称", "shard" : 分片号, "node" : "节点名称"
            }
        },
        {
          "allocate" : {
              "index" : "索引名称", "shard" : 分片号, "node" : "节点名称"
          }
        }
    ]
}'
 
curl -XPOST localhost:9200/_aliases -d '
{
    "actions": [
        { "remove": {
            "alias": "别名",
            "index": "索引名A"
        }},
        { "add": {
            "alias": "别名",
            "index": "索引名B"
        }}
    ]
}
'
 
curl localhost:9200/_nodes/节点名称/plugins?pretty=true
 
curl -s localhost:9200/_cat/shards
 
 
Elasticsearch版本升级
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html
1.
curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}'
 
2.
curl -XPOST http://localhost:9200/_flush/synced
 
3.
apt-get update;
apt-get --only-upgrade install elasticsearch
 
4.
/usr/share/elasticsearch/bin/plugin remove analysis-kuromoji;
/usr/share/elasticsearch/bin/plugin remove analysis-smartcn;
/usr/share/elasticsearch/bin/plugin remove analysis-icu;
/usr/share/elasticsearch/bin/plugin remove mobz/elasticsearch-head;
 
/usr/share/elasticsearch/bin/plugin install analysis-kuromoji;
/usr/share/elasticsearch/bin/plugin install analysis-smartcn;
/usr/share/elasticsearch/bin/plugin install analysis-icu;
/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head;
 
5.
curl -XGET http://localhost:9200/_cat/nodes
 
6.
curl -XPUT 'http://localhost:9200/_cluster/settings' -d '{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  }
}'
 
7.
curl -XGET http://localhost:9200/_cat/health

Deleting files in Azure Blob Storage with Azure CLI

Standard
azure login
azure storage account list
azure storage account set YourAccountName
export AZURE_STORAGE_CONNECTION_STRING='DefaultEndpointsProtocol=https;AccountName=YourAccountName;AccountKey=myDSCnRVxxxxxNCwcFkkxxxxxx4nbhV10unY2yFRHNGb8VHZLjg77F9WA=='
azure storage container show YourContainer
azure storage container set YourContainer
azure storage blob show YourContainer BlobName
azure storage blob delete YourContainer BlobName

You can address a blob in your storage account using the following URL format:
http://storage-account-name.blob.core.windows.net/container-name/blob-name

For example, here is a URL that addresses one of the blobs in the diagram above:
http://sally.blob.core.windows.net/movies/MOV1.AVI

用Azure命令行配load balancer

Standard

https://azure.microsoft.com/en-us/documentation/articles/load-balancer-get-started-internet-arm-cli/
https://azure.microsoft.com/en-us/documentation/articles/load-balancer-get-started-ilb-arm-cli/
只是参照上面两篇敲一遍练练手.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
azure login
azure config mode arm
azure account set <Subscription ID>
 
azure group create ResourceGroupsName eastus2
 
azure network vnet create ResourceGroupsName ResourceGroupsName-Vnet eastus2 -a 10.0.0.0/16
azure network vnet subnet create ResourceGroupsName ResourceGroupsName-Vnet ResourceGroupsName-VnetSubnet -a 10.0.0.0/24
azure network public-ip create -g ResourceGroupsName -n ResourceGroupsName-PublicIP -l eastus2 -d rgn-esg -a static -i 4
azure network lb create ResourceGroupsName ResourceGroupsName-LB eastus2
azure network lb frontend-ip create ResourceGroupsName ResourceGroupsName-LB ResourceGroupsName-FrontendPool -i ResourceGroupsName-PublicIP
azure network lb address-pool create ResourceGroupsName ResourceGroupsName-LB ResourceGroupsName-BackendPool
 
azure network lb inbound-nat-rule create -g ResourceGroupsName -l ResourceGroupsName-LB -n ssh1 -p tcp -f 62201 -b 22
azure network lb inbound-nat-rule create -g ResourceGroupsName -l ResourceGroupsName-LB -n ssh2 -p tcp -f 62202 -b 22
 
azure network lb inbound-nat-rule create -g ResourceGroupsName -l ResourceGroupsName-LB -n ssh3 -p tcp -f 62203 -b 22
 
azure network lb rule create ResourceGroupsName ResourceGroupsName-LB lbrule -p tcp -f 9200 -b 9200 -t ResourceGroupsName-FrontendPool -o ResourceGroupsName-BackendPool
azure network lb probe create -g ResourceGroupsName -l ResourceGroupsName-LB -n healthprobe -p "http" -o 9200 -f / -i 15 -c 4
 
azure network lb show ResourceGroupsName ResourceGroupsName-LB
 
azure network nic create -g ResourceGroupsName -n lb-nic1-be --subnet-name ResourceGroupsName-VnetSubnet --subnet-vnet-name ResourceGroupsName-Vnet -d "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/ResourceGroupsName/providers/Microsoft.Network/loadBalancers/ResourceGroupsName-LB/backendAddressPools/ResourceGroupsName-BackendPool" -e "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/ResourceGroupsName/providers/Microsoft.Network/loadBalancers/ResourceGroupsName-LB/inboundNatRules/ssh1" eastus2
azure network nic create -g ResourceGroupsName -n lb-nic2-be --subnet-name ResourceGroupsName-VnetSubnet --subnet-vnet-name ResourceGroupsName-Vnet -d "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/ResourceGroupsName/providers/Microsoft.Network/loadBalancers/ResourceGroupsName-LB/backendAddressPools/ResourceGroupsName-BackendPool" -e "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/ResourceGroupsName/providers/Microsoft.Network/loadBalancers/ResourceGroupsName-LB/inboundNatRules/ssh2" eastus2
 
azure vm create --resource-group ResourceGroupsName --name es1 --location eastus2 --vnet-name ResourceGroupsName-Vnet --vnet-subnet-name ResourceGroupsName-VnetSubnet --nic-name lb-nic1-be --availset-name ResourceGroupsName-AvSet --storage-account-name rgnesgdata -z Standard_D3_V2 --os-type Linux --image-urn Canonical:UbuntuServer:14.04.3-LTS:14.04.201512032
azure vm create --resource-group ResourceGroupsName --name es2 --location eastus2 --vnet-name ResourceGroupsName-Vnet --vnet-subnet-name ResourceGroupsName-VnetSubnet --nic-name lb-nic2-be --availset-name ResourceGroupsName-AvSet --storage-account-name rgnesgdata -z Standard_D3_V2 --os-type Linux --image-urn Canonical:UbuntuServer:14.04.3-LTS:14.04.201512032

How to use AWS ElasticCache on Azure

Standard
[ec2-user@ip-174-129-100-10 ~]$ cat /etc/rc.local
echo "redis-name";
ADDRESS=`nslookup redis-name.7exo1h.0001.use1.cache.amazonaws.com | grep "Address:" | tail -n 1 | awk '{print $2}'`;
echo $ADDRESS;
iptables -t nat -A PREROUTING -i eth0 -p tcp -m tcp --dport 26379 -j DNAT --to-destination $ADDRESS:6379;
iptables -t nat -A POSTROUTING -j MASQUERADE;
sysctl net.ipv4.ip_forward=1;
 
[ec2-user@ip-174-129-100-10 ~]$ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:26379 to:10.153.181.100:6379
 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
 
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
 
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  anywhere             anywhere
azure@azure:/home/azure# cat /etc/rc.local
# redis-name
iptables -t nat -I OUTPUT -p tcp -d redis-name.7exo1h.0001.use1.cache.amazonaws.com --dport 6379  -j DNAT --to-destination 174.129.100.10:26379
 
azure@azure:/home/azure# iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
 
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             ec2-50-17-68-100.compute-1.amazonaws.com  tcp dpt:6379 to:174.129.100.10:26379
 
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

A lookup table of the Azure and AWS

Standard
Microsoft Azure Amazon Web Services (AWS)
Available Regions Azure Regions AWS Global Infrastructure
Compute Services Virtual Machines (VMs) Elastic Compute Cloud (EC2)
Cloud Services
Azure Websites and Apps
Amazon Elastic Beanstalk
Azure Visual Studio Online None
Container Support Docker Virtual Machine Extension (how to) EC2 Container Service (Preview)
Scaling Options Azure Autoscale (how to) Auto Scaling
Analytics/Hadoop Options HDInsight (Hadoop) Elastic MapReduce (EMR)
Government Services Azure Government AWS GovCloud
App/Desktop Services Azure RemoteApp Amazon WorkSpaces
Amazon AppStream
Storage Options Azure Storage (Blobs, Tables, Queues, Files) Amazon Simplge Storage (S3)
Block Storage Azure Blob Storage (how to) Amazon Elastic Block Storage (EBS)
Hybrid Cloud Storage StorSimple None
Backup Options Azure Backup Amazon Glacier
Storage Services Azure Import Export (how to) Amazon Import / Export
Azure File Storage (how to) AWS Storage Gateway
Azure Site Recovery None
Content Delivery Network (CDN ) Azure CDN Amazon CloudFront
Database Options Azure SQL Database Amazon Relational Database Service (RDS)
Amazon Redshift
NoSQL Database Options Azure DocumentDB Amazon Dynamo DB

Azure Managed Cache (Redis Cache) Amazon Elastic Cache
Data Orchestration Azure Data Factory AWS Data Pipeline
Networking Options Azure Virtual Network Amazon VPC
Azure ExpressRoute AWS Direct Connect
Azure Traffic Manager Amazon Route 53
Load Balancing Load Balancing for Azure (how to) Elastic  Load Balancing
Administration & Security Azure Active Directory AWS Directory Service
AWS Identity and Access Management (IAM)
Multi-Factor Authentication Azure Multi-Factor Authentication AWS Multi-Factor Authentication
Monitoring Azure Operational Insights Amazon CloudTrail
Azure Application Insights Amazon CloudWatch
Azure Event Hubs None
Azure Notification Hubs Amazon Simple Notification Service (SNS)
Azure Key Vault (Preview) AWS Key Management Service
Compliance Azure Trust Center AWS CLoudHSM
Management Services & Options Azure Resource Manager Amazon CloudFormation
API Management Azure API Management None
Automation Azure Automation AWS OpsWorks
Azure Batch
Azure Service Bus
Amazon Simple Queue Service (SQS)
Amazon Simple Workflow (SWF)
None AWS CodeDeploy
Azure Scheduler None
Azure Search Amazon CloudSearch
Analytics Azure Stream Analytics Amazon Kinesis
Email Services Azure BizTalk Services Amazon Simple Email Services (SES)
Media Services Azure Media Services Amazon Elastic Transcoder
Amazon Mobile Analytics
Amazon Cognitor
Other Services & Integrations Azure Machine Learning (Preview) None
None AWS Lambda (Preview)
None AWS Config (Preview)