Is this a BUG REPORT or FEATURE REQUEST?:

Bug Report
/kind bug

What happened: I created a PVC using storageclass. The Pod which used the PVC fails to start. I see the disk was attached successfully to the node but the mount operation has failed.

Events:
  FirstSeen	LastSeen	Count	From			SubObjectPath	Type		Reason			Message
  ---------	--------	-----	----			-------------	--------	------			-------
  1m		1m		1	{default-scheduler }			Normal		Scheduled		Successfully assigned thindiskpod to node3
  1m		1m		1	{kubelet node3}				Normal		SuccessfulMountVolume	MountVolume.SetUp succeeded for volume "default-token-5h5x6" 
  1m		1m		5	{kubelet node3}				Warning		FailedMount		MountVolume.MountDevice failed for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" : executable file not found in $PATH

What you expected to happen: The PV to mount, format and the Pod to start

How to reproduce it (as minimally and precisely as possible): Use the latest master builds for kubernetes deployments.

Anything else we need to know?: N/a

Environment: vSphere

I see the following errors on the kubelet where the node failed to format the disk and eventually mount was failed.

{"log":"I0816 20:56:15.900236 659 operation_generator.go:417] MountVolume.WaitForAttach succeeded for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" (UniqueName: "kubernetes.io/vsphere-volume/[vsanDatastore] kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk") pod "thindiskpod" (UID: "52fbf7a2-82c5-11e7-a497-0050569c48bf") \n","stream":"stderr","time":"2017-08-16T20:56:15.901072134Z"}

{"log":"I0816 20:56:15.902508 659 mount_linux.go:413] Checking for issues with fsck on disk: /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5\n","stream":"stderr","time":"2017-08-16T20:56:15.902695897Z"}

{"log":"I0816 20:56:15.921616 659 mount_linux.go:427] fsck error fsck from util-linux 2.25.2\n","stream":"stderr","time":"2017-08-16T20:56:15.921725608Z"}

{"log":"fsck: error 2 (No such file or directory) while executing fsck.ext2 for /dev/sdb\n","stream":"stderr","time":"2017-08-16T20:56:15.921752327Z"}

{"log":"I0816 20:56:15.921652 659 mount_linux.go:432] Attempting to mount disk: ext4 /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[vsanDatastore] kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\n","stream":"stderr","time":"2017-08-16T20:56:15.921774238Z"}

{"log":"I0816 20:56:15.921686 659 mount_linux.go:160] Mounting cmd (mount) with arguments ([-t ext4 -o defaults /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[vsanDatastore] kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk])\n","stream":"stderr","time":"2017-08-16T20:56:15.921948786Z"}

{"log":"E0816 20:56:15.929506 659 mount_linux.go:165] Mount failed: exit status 32\n","stream":"stderr","time":"2017-08-16T20:56:15.929619467Z"}

{"log":"Mounting command: mount\n","stream":"stderr","time":"2017-08-16T20:56:15.92963496Z"}

{"log":"Mounting arguments: -t ext4 -o defaults /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5 /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[vsanDatastore] kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk\n","stream":"stderr","time":"2017-08-16T20:56:15.929639337Z"}

{"log":"Output: mount: wrong fs type, bad option, bad superblock on /dev/sdb,\n","stream":"stderr","time":"2017-08-16T20:56:15.929642498Z"}

{"log":" missing codepage or helper program, or other error\n","stream":"stderr","time":"2017-08-16T20:56:15.929645312Z"}

{"log":"\n","stream":"stderr","time":"2017-08-16T20:56:15.929647822Z"}

{"log":" In some cases useful info is found in syslog - try\n","stream":"stderr","time":"2017-08-16T20:56:15.929650562Z"}

{"log":" dmesg | tail or so.\n","stream":"stderr","time":"2017-08-16T20:56:15.929653095Z"}

{"log":"\n","stream":"stderr","time":"2017-08-16T20:56:15.929655488Z"}

{"log":"I0816 20:56:15.931335 659 mount_linux.go:480] Attempting to determine if disk "/dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5" is formatted using lsblk with args: ([-n -o FSTYPE /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5])\n","stream":"stderr","time":"2017-08-16T20:56:15.931446377Z"}

{"log":"I0816 20:56:15.938198 659 mount_linux.go:483] Output: "\n"\n","stream":"stderr","time":"2017-08-16T20:56:15.938284852Z"}

{"log":"I0816 20:56:15.938236 659 mount_linux.go:452] Disk "/dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5" appears to be unformatted, attempting to format as type: "ext4" with options: [-F /dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5]\n","stream":"stderr","time":"2017-08-16T20:56:15.938393307Z"}

{"log":"E0816 20:56:15.939026 659 mount_linux.go:460] format of disk "/dev/disk/by-id/wwn-0x6000c29d3c96f77096950581483fe3a5" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[vsanDatastore] kubevols/kubernetes-dynamic-pvc-4dc51e93-82c5-11e7-a497-0050569c48bf.vmdk") options:(["defaults"])error:(executable file not found in $PATH)\n","stream":"stderr","time":"2017-08-16T20:56:15.939126395Z"}

{"log":"I0816 20:56:15.939915 659 server.go:214] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"thindiskpod", UID:"52fbf7a2-82c5-11e7-a497-0050569c48bf", APIVersion:"v1", ResourceVersion:"5203", FieldPath:""}): type: 'Warning' reason: 'FailedMount' MountVolume.MountDevice failed for volume "pvc-4dc51e93-82c5-11e7-a497-0050569c48bf" : executable file not found in $PATH\n","stream":"stderr","time":"2017-08-16T20:56:15.94001794Z"}

@tusharnt @divyenpatel @luomiao @rohitjogvmw