vs2017-win2016 as Agent Specification but it was removed 2022-03-15. There were some points that I needed to look into it, so I wrote down those in this article.
Bash script fails for all commands in gradle
There are 3 bash.exe in the environment (Azure pipeline). I checked the path of bash.exe in Bash Script by where bash command. The result is as follows.
C:\Program Files\Git\bin\bash.exe
C:\Windows\System32\bash.exe
C:\Program Files\Git\usr\bin\bash.exeGit-bash is what we use on our development machines too, so it looks correct. We define Gradle tasks and they are used in the pipeline. It is something like this.
task build(type: Exec) {
description 'Builds all modules with lerna'
dependsOn 'lerna_bootstrap'
workingDir 'src'
commandLine 'bash', '-c', 'lerna run build'
}However, it fails with exit code 1. Even if the command is the simplest command “ls“, it fails. I tried the following commands.
task yuto(type: Exec) {
description 'test'
workingDir 'src'
commandLine 'bash', '-c', 'ls'
}
task yuto2(type: Exec) {
description 'test'
workingDir 'src'
commandLine 'C:\Program Files\Git\bin\bash.exe', '-c', 'ls'
}
task yuto3(type: Exec) {
description 'test'
workingDir 'src'
commandLine 'C:\Windows\System32\bash.exe', '-c', 'ls'
}
task yuto4(type: Exec) {
description 'test'
workingDir 'src'
commandLine 'C:\Program Files\Git\usr\bin\bash.exe', '-c', 'ls'
}yuto and yuto3 tasks failed with exit code 1. It turned out that C:\Windows\System32\bash.exe was executed in Gradle even though C:\Program Files\Git\bin\bash.exe is the highest priority. It worked in vs2017-win2016 environment, so it seems something changed.
I executed where bash command in Bash Script but it might be valid only in the script itself in the new environment windows-2022.
By specifying the absolute path to the desired bash.exe, it succeeds. However, one developer uses a Virtual Machine for his dev-env and the path was different. Using the absolute path was not so nice and it doesn’t fit the developer. Therefore, I changed it to PowerShell instead.
task build(type: Exec) {
description 'Builds all modules with lerna'
dependsOn 'lerna_bootstrap'
workingDir 'src'
commandLine 'powershell', '-c', 'lerna run build'
}It also works.
How to get an Environment Variable on Powershell in gradle task
Changing bash to PowerShell works for most tasks but some of them need an adjustment because the way to get an environment variable is different. The original one is something like this.
task publish(type: Exec) {
description 'Publish artifacts'
dependsOn 'build', 'test'
environment NPM_REGISTRY: project.property('NPM_REGISTRY')
workingDir 'src'
commandLine 'powershell', '-c', "lerna publish \"${version}-${build_timestamp}\" --registry \"$NPM_REGISTRY\" --exact --no-git-tag-version --no-push --yes --force-publish=* --no-git-reset"It’s not difficult. It publishes artifacts after build and test steps but it contains an environment variable $NPM_REGISTRY. It works on bash script but it doesn’t on Powershell. Normally, we can get an env variable with this.
$env:ENV_NAMEBut I didn’t know how to write it in the command at first. I tried the following commands but all of them failed.
commandLine 'powershell', '-c', " echo $env:NPM_REGISTRY"
commandLine 'powershell', '-c', " echo ${env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo ${$env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo ${$env:NPM_REGISTRY}"
commandLine 'powershell', '-c', " echo " + env:NPM_REGISTRY + ""
commandLine 'powershell', '-c', " echo " + $env:NPM_REGISTRY + ""To make it work, $ mark needs to be added to the command when PowerShell is executed. It means the symbol must be escape like this.
commandLine 'powershell', '-c', " echo $env:NPM_REGISTRY"I applied it to the actual command.
task publish(type: Exec) {
description 'Publish artifacts'
dependsOn 'build', 'test'
environment NPM_REGISTRY: project.property('NPM_REGISTRY')
workingDir 'src'
commandLine 'powershell', '-c', "lerna publish ${version}-${build_timestamp} --registry $env:NPM_REGISTRY --exact --no-git-tag-version --no-push --yes --force-publish=* --no-git-reset"
}MSBuild.exe not found error in gradle
Our project has different languages and we need to build them all in different ways. MSBuild.exe is used to build C# code but it fails because of MSBuild.exe not found error.
Azure pipeline offers MSBuild task but it was not possible to use it because MSBuild.exe is called in one of bash scripts called by Gradle task.
I looked for the MSBuild.exe file first with the following commands in the Azure pipeline.
echo "-----ls----"
ls
echo "------Program Files---"
ls "/c/Program Files"
echo "-----vs----"
ls "/c/Program Files/Microsoft Visual Studio/"
echo "-----vs2022----"
ls "/c/Program Files/Microsoft Visual Studio/2022/"
echo "------enterprise---"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/"
echo "-----msbuild----"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/"
echo "-----Current----"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/"
echo "------bin---"
ls "/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin"
echo "---X86"
ls "/c/Program Files (x86)/"
echo "---Installer"
ls "/c/Program Files (x86)/Microsoft Visual Studio/Installer"
echo "---shared"
ls "/c/Program Files (x86)/Microsoft Visual Studio/Shared"There might be another better way to do a similar thing but I don’t have enough access right. So I investigated it in this way.
I added the desired path to MSBuild.exe to PATH variable.
PATH="/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin/":$PATHAfter this, MSBuild.exe can be found by a bash script called by a Gradle task.
node-gyp does not find Visual Studio version
When we used vs2017-win2016 environment, it works without any problem but I got the following error on windows-2022.
lerna ERR! npm run build stderr:
gyp ERR! find VS
gyp ERR! find VS msvs_version not set from command line or npm config
gyp ERR! find VS VCINSTALLDIR not set, not running in VS Command Prompt
gyp ERR! find VS unknown version "undefined" found at "C:\Program Files\Microsoft Visual Studio22\Enterprise"
gyp ERR! find VS could not find a version of Visual Studio 2017 or newer to use
gyp ERR! find VS looking for Visual Studio 2015
gyp ERR! find VS - not found
gyp ERR! find VS not looking for VS2013 as it is only supported up to Node.js 8
gyp ERR! find VS
gyp ERR! find VS **************************************************************
gyp ERR! find VS You need to install the latest version of Visual Studio
gyp ERR! find VS including the "Desktop development with C++" workload.
gyp ERR! find VS For more information consult the documentation at:
gyp ERR! find VS https://github.com/nodejs/node-gyp#on-windows
gyp ERR! find VS **************************************************************
gyp ERR! find VS
gyp ERR! configure error
gyp ERR! stack Error: Could not find any Visual Studio installation to use
gyp ERR! stack at VisualStudioFinder.fail (C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:121:47)
gyp ERR! stack at C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:74:16
gyp ERR! stack at VisualStudioFinder.findVisualStudio2013 (C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:351:14)
gyp ERR! stack at C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:70:14
gyp ERR! stack at C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\find-visualstudio.js:372:16
gyp ERR! stack at C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\util.js:54:7
gyp ERR! stack at C:\hostedtoolcache\windows\node.16.1\x86\node_modules\npm\node_modules\node-gyp\lib\util.js:33:16
gyp ERR! stack at ChildProcess.exithandler (child_process.js:310:5)
gyp ERR! stack at ChildProcess.emit (events.js:311:20)
gyp ERR! stack at maybeClose (internal/child_process.js:1021:16)
gyp ERR! System Windows_NT 10.0.20348
gyp ERR! command "C:\\hostedtoolcache\\windows\\node\.16.1\\x86\\node.exe" "C:\\hostedtoolcache\\windows\\node\.16.1\\x86\\node_modules\\npm\\node_modules\\node-gyp\\bin\\node-gyp.js" "rebuild"
gyp ERR! cwd D:\a\s\src\modules\module-name
gyp ERR! node -v v12.16.1
gyp ERR! node-gyp -v v5.0.5
gyp ERR! not ok
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! module-name@x.y.z build: `sh build.sh`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the module-name@x.y.z build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.node-gyp version is old, so I needed to update it first. I’ve read Updating-npm-bundled-node-gyp and applied it.
I added the following commands to update node-gyp in the right place.
cd "/c/hostedtoolcache/windows/node/12.16.1/x86/node_modules/npm/node_modules/npm-lifecycle"
npm install node-gyp@9.0.0I don’t remember well but the path to MSBuild.exe might need to be added to the PATH env.
PATH="/c/Program Files/Microsoft Visual Studio/2022/Enterprise/MSBuild/Current/Bin/":$PATHRedirect operation leads a different result
If a redirect operation is used in the command, the results can be different. See the following two commands.
commandLine 'bash', '-c', 'echo node.exe app.js > run.bat'
commandLine 'powershell', '-c', 'echo node.exe app.js > run.bat'Both commands are the same but the execution script is different. It looks it generates exactly the same file but actually not.
The result of the bash command
node.exe app.jsThe result of PowerShell command
node.exe
app.jsIf using PowerShell, the line is split into two lines. For PowerShell, it’s necessary to add double quotations there.
echo "node.exe app.js" > run.batThen, the result becomes the same in PowerShell but it still didn’t work in the Gradle task. When I executed the run.bat, it failed with the following error.
D:\temp>.\run.bat
D:\temp>■e
'■e' is not recognized as an internal or external command,
operable program or batch file.The characters were garbled. I found the encoding different
- bash: Unix(LF), UTF-8
- PowerShell: Windows (CR LF), UTF-16 LE BOM
I guess BOM causes the garbled character above. BOM stands for Byte Order Mark that is used to know in which file format the file is written.
BOM is added by default in Powershell. I somehow needed to remove it. I tried the following.
echo "node.exe app.js" | Out-File -encoding utf8 run.bat
echo "node.exe app.js" | Out-File -encoding ascii run.batThe first one creates a file with BOM even though it doesn’t add BOM in the encoding.
The second one creates exactly what I want. It doesn’t work if the content contains characters that ASCII can’t handle but it is no problem in my case.
But it doesn’t work in Gradle script!! None of the following commands didn’t work.
commandLine 'powershell', '-c', 'echo "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'echo \"node.exe app.js\" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'echo `"node.exe app.js`" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Host "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output "node.exe app.js" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output \"node.exe app.js\" | Out-File -encoding ascii -FilePath run.bat'
commandLine 'powershell', '-c', 'Write-Output `"node.exe app.js`" | Out-File -encoding ascii -FilePath run.bat'I changed from PowerShell to cmd to make it work.
commandLine 'cmd', '/c', 'echo node.exe app.js > run.bat'This actually works as expected but I found a better way with Gradle.
task create_run_bat() {
description 'Generate a batch script'
file("src/app/dist/run.bat").text = "node.exe app.js"
}It looks a proper way for Gradle.
End
I could finally manage to adjust the Azure pipeline for the new environment. I’ve touched Azure pipeline before but it was very small things to change. I struggled with many things this time but got more familiar with Azure pipeline.
I hope this article helps other developers.
Comments