Execute BES Download Cacher (BESDownloadCacher 5.8.6)
Log In or Register to download the BES file, and more.

1 Votes

Versioning - This is the latest version.

1Execute BES Download Cacher (BESDownloadCacher 5.8.6)11/4/2016 8:56:15 AM
2Execute BES Download Cacher (BESDownloadCacher 5.8.6)11/4/2016 11:35:39 AM
3Execute BES Download Cacher (BESDownloadCacher 5.8.6)11/7/2016 9:06:15 AM
4Execute BES Download Cacher (BESDownloadCacher 5.8.6)11/7/2016 9:09:47 AM

Description

This is intended to be an improved usage of the BES Download Cacher.  In its current form, the BES Download Cacher will read a list of BES Sites, and determine all of the download URLs listed in the site.  Then, it can compare the download URL list to the sha1 cache folder of the BES Server, and retrieve any downloads that are not already precached.

However, the logic it uses to determine whether the files are precached is slow; statusing every file in the sha1 cache, in a loop repeated for each download URL found in the site(s).  This can take days to run to completion.

This Task has several actions. 

The first Action will download and execute the BESDownloadCacher tool to wwwrootbes\BESDownloadCacher.  It will then run the Download Cacher tool to list the URLs available for download - but it will not compare to the contents of the sha1 folder, it only generates lists of download URLs.  Each download URL list file will be stored at wwwrootbes\BESDownloadCacher\, have a filename starting with "downloadlist-", ending with ".txt", and containing the name of the efxm file for the given site; example "downloadlist-OS Deployment and Bare Metal Imaging.efxm.txt".  The list of sites to cache is contained in a parameter that can be updated per organizational needs.

The second Action will take the previously-generated list of download URLs, and compare them to the contents of the sha1 folder using a more efficient comparison logic.  It will generate a list of downloads that are not already precached and save that result to wwwrootbes\BESDownloadCacher\result-download-list.txt.  This is an optional task to preview the download list.

The third Action will compare the download urls lists generated by the first Action to the sha1 folder, then use curl to download any files that are not already precached and save them to the sha1 folder.

 


Property Details

ID22471
StatusAlpha - Code that was just developed
TitleExecute BES Download Cacher (BESDownloadCacher 5.8.6)
DomainBESC
SourceInternal
Source Release Date11/2/2016 12:00:00 AM
KeywordsBESDownloadCacher, sha1, precache, cache, download
Is TaskTrue
Added by on 11/7/2016 9:09:47 AM
Last Modified by on 11/7/2016 9:09:47 AM
Counters 5154 Views / 17 Downloads
User Rating 1 star 2 star 3 star 4 star 5 star * Average over 0 ratings. ** Log In or Register to add your rating.

Relevance

Used in 94 fixlets and 5 analyses   * Results in a true/false
Show indented relevance
exists main gather service
isWindows (Relevance 1172)
Used in 1117 fixlets and 524 analyses   * Results in a true/false
Show indented relevance
windows of operating system
Used in 21 fixlets and 3 analyses   * Results in a true/false
Show indented relevance
if exists property "in proxy agent context" then not in proxy agent context else true

Actions

Action 1

Action Link Click here to Generate Download URL List
Script Type BigFix Action Script
begin prefetch block
    add prefetch item name=BESDownloadCacher.exe sha1=df3507a5b0ec26e0efe8f6d9106c16a9dc8c62cd size=4440160 url=https://www.ibm.com/developerworks/community/wikis/form/anonymous/api/wiki/90553c0b-42eb-4df0-9556-d3c2e0ac4c52/page/c71910e2-8052-4751-853b-5a45bfc97266/attachment/266b00e4-f8ab-468c-8899-58aa36cda3df/media/BESDownloadCacher-5.8.6.exe sha256=ec3d6b3c0ae5c0019d238408128425c8f78acdf5ed5c186447b38e2fa0663534
    collect prefetch items
end prefetch block

utility __Download\BESDownloadCacher.exe

// Customize this to match the list of sites you wish to download
parameter "efxmlist"="BES Support.efxm;Enterprise Security.efxm;Updates for Windows Applications.efxm;OS Deployment and Bare Metal Imaging.efxm"

// Set up the BESDownloadCacher directory beneath wwwrootbes and copy BESDownloadCacher.exe to it
// We don't want to run out of the client folder, especially the client downloads folder, because
// the BESDownloadCacher is going to continue running long after this Action stops. Use "runhidden" rather than "waithidden" so
// the BES Client isn't tied up waiting for it to finish.
if {not exists folder "BESDownloadCacher" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}
    folder create "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher"
endif

// Delete any existing BESDownloadCacher (ensure we're using the correct version)
delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\BESDownloadCacher.exe"
copy __Download\BESDownloadCacher.exe "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\BESDownloadCacher.exe"

// Build a list of download URLs that are present in the site files
// First delete any existing download listings, then execute BESDownloadCacher for each site,
// outputting the download list to a text file that can be read later.
delete __appendfile
appendfile @echo Starting %DATE% %TIME% %COMPUTERNAME%
appendfile pushd "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher"
appendfile del /q downloadlist-*.txt
appendfile del /q BESDownloadcacher.log
appendfile {concatenation "%0d%0a" of ("%22" & pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\BESDownloadCacher\BESDownloadcacher.exe%22 -s -l -m %22" & pathname of client folder of site "actionsite" & "\" & it & "%22 -w %22" & pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\BESDownloadCacher\downloadlist-" & it & ".txt%22") of substrings separated by ";" of parameter "efxmlist"}
appendfile @echo Finished %DATE% %TIME% %COMPUTERNAME%
delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\GetDownloadList.cmd"
move __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\GetDownloadList.cmd"

// runhidden / waithidden have problems with multiple quoted command lines, so build a wrapper script
delete __appendfile
appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\GetDownloadList.cmd" > "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\GetDownloadList.log" 2>&1
delete RunDownloadCacher.cmd
move __appendfile RunDownloadCacher.cmd
runhidden cmd /c RunDownloadCacher.cmd
Success Criteria

This action will be considered successful when the applicability relevance evaluates to false.

Action 2

Action Link Click here to generate a list of what should be downloaded, without downloading the files
Script Type BigFix Action Script
// Build a listing of files that *would* be downloaded - they are in the list of site download URLs but not present in the sha1 folder

delete __appendfile
if {exists processes whose (pathname of image file of it = pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\BESDownloadCacher\BESDownloadCacher.exe")}
// Can't do much processing if BESDownloadCacher.exe is still running; update the batch and log files to reflect that
    appendfile Skipping due to active BESDownloadCacher.exe process {now}

else
// This first form gives unique values of downloard URL, sha1, size, etc.
//appendfile {concatenation "%0d%0a" of (" name=" & item 1 of it & " sha1=" & item 1 of it & " size=" & item 2 of it & " url=" & item 0 of it) of (if not exists file (pathname of folder "bfmirror\downloads\sha1" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\" & item 1 of it) then it else nothing) of (preceding text of first " " of following text of first "url=" of it, preceding text of first " " of following text of first "sha1=" of it, preceding text of first " " of following text of first "size=" of it) of unique values of following texts of firsts " " of following texts of firsts "actionid=" of lines whose (it does not contain "%7b" and it does not contain "http://127.0.0.1" and it does not contain "http://localhost" and it does not contain "MANUAL_BES_CACHING_REQUIRED" and it does not contain "<" and it does not contain ">") of files whose (name of it starts with "downloadlist-" and name of it ends with ".txt") of folders "BESDownloadcacher" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}
// This second form gives the original line from the download lists, including Site Name, Fixlet ID, and Action ID; but may have duplicate downloads
    appendfile {concatenation "%0d%0a" of (if not exists file (pathname of folder "bfmirror\downloads\sha1" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\" & item 0 of it) then item 1 of it else nothing) of ( preceding texts of firsts " " of following texts of firsts "sha1=" of it, it ) of lines whose (it does not contain "%7b" and it does not contain "http://127.0.0.1" and it does not contain "http://localhost" and it does not contain "MANUAL_BES_CACHING_REQUIRED" and it does not contain "<" and it does not contain ">") of files whose (name of it starts with "downloadlist-" and name of it ends with ".txt") of folders "BESDownloadcacher" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}
endif

delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\result-download-list.txt"
move __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\result-download-list.txt"
Success Criteria

This action will be considered successful when the applicability relevance evaluates to false.

Action 3

Action Link Click here to use cURL to download the files
Script Type BigFix Action Script
// Enter your action script here

// useful curl command line options listed at http://www.thegeekstuff.com/2012/04/curl-examples/
// basic usage: curl -o MDT.zip -L http://software.bigfix.com/download/osd/MDTBundleCreator-3.9.04.zip


begin prefetch block

// - This unzip is a compiled verion of unzip v5.52 fom http://info-zip.org/
// - https://www.virustotal.com/en/file/8d9b5190aace52a1db1ac73a65ee9999c329157c8e88f61a772433323d6b7a4a/analysis
//add prefetch item name=unzip.exe sha1=e1652b058195db3f5f754b7ab430652ae04a50b8 size=167936 url=http://software.bigfix.com/download/redist/unzip-5.52.exe

// VirusTotal report - https://www.virustotal.com/en/file/fd872950049dddcca106cf6d3fd4c48356d68669dbee702b8d83a96c001708c0/analysis/
//add prefetch item name=curl.zip sha1=413f14f96e5fc0723a55d8ddbaed5a8193fba1e5 size=1168899 url=http://www.paehl.com/open_source/?download=curl_750_3_ssl.zip sha256=fd872950049dddcca106cf6d3fd4c48356d68669dbee702b8d83a96c001708c0

// VirusTotal report - https://www.virustotal.com/en/file/93fc9fac65f4a5c2923949ab47609f2bd4d97a6f6ba4b39ae30e15c88f7a9202/analysis
    add prefetch item name=curl.exe sha1=1b857aaef905c24c4063a26c3d5aad5435d0e9cd size=1673216 url=https://raw.githubusercontent.com/strawgate/Patching-Tools/master/Downloads/cURL/curl_7_51_0_openssl_nghttp2_x86.exe sha256=93fc9fac65f4a5c2923949ab47609f2bd4d97a6f6ba4b39ae30e15c88f7a9202

// Certificate bundle for curl.exe based on Mozilla Foundation
    add prefetch item name=ca-bundle.crt sha1=0875794bc3b7d31d65a69c702ae0452696921874 size=261644 url=https://raw.githubusercontent.com/strawgate/Patching-Tools/master/Downloads/cURL/ca-bundle_20160907.crt sha256=e39faef9a13c44ddc75f7c0d0d6977fb2bc707ae1d08d47469b9c5ac8f260e40

collect prefetch items
end prefetch block

// ADD TO UTILITY CACHE:
utility __Download\curl.exe
utility __Download\ca-bundle.crt

delete __appendfile

if {exists processes whose (pathname of image file of it = pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\BESDownloadCacher\BESDownloadCacher.exe")}
// Can't do much processing if BESDownloadCacher.exe is still running; update the batch and log files to reflect that
    appendfile @echo Skipping download list resolution due to active BESDownloadCacher.exe process {now}
    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.cmd"
    copy __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.cmd"

    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.cmd"
    copy __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.cmd"
    
    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.out"
    copy __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.out"

    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.out"
    copy __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.out"

else
// Build the list of files to download - these are present in the downloadlist-site listing but not present in sha1
    appendfile mkdir "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\Downloads"
    appendfile echo Beginning downloads...
    appendfile {concatenation "%0d%0a" of ("%22" & (pathname of file "curl.exe" of download folder | "__Download\curl.exe") & "%22 -L -o %22" & pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\BESDownloadCacher\Downloads\" & item 1 of it & "%22 %22" & item 0 of it & "%22") of (if not exists file (pathname of folder "bfmirror\downloads\sha1" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "\" & item 1 of it) then it else nothing) of (preceding text of first " " of following text of first "url=" of it, preceding text of first " " of following text of first "sha1=" of it, preceding text of first " " of following text of first "size=" of it) of unique values of following texts of firsts " " of following texts of firsts "actionid=" of lines whose (it does not contain "%7b" and it does not contain "http://127.0.0.1" and it does not contain "http://localhost" and it does not contain "MANUAL_BES_CACHING_REQUIRED" and it does not contain "<" and it does not contain ">") of files whose (name of it starts with "downloadlist-" and name of it ends with ".txt") of folders "BESDownloadcacher" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}

// Download the files, temporarily to the BESDownloadCacher folder
    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.cmd"
    move __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.cmd"

// waithidden has issues with multiple quoted command line params. Using a path with spaces for the .cmd, and for the output file, so make a new wrapper script
// to execute the real batch file and save the output
    delete __appendfile
    appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.cmd" > "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\get-downloads.out" 2>&1
    delete RunDownloads.cmd
    move __appendfile RunDownloads.cmd
    waithidden cmd /c RunDownloads.cmd

// If curl had a bad response (like 404), that response ends up in the file.
// Validate that the file we actually downloaded matches the sha1 we expect, and if so move it to the proper sha1 folder
// When we downloaded with curl, we coerced the output filename to match the sha1 we expect; so if filename = sha1, we downloaded the correct file
// After moving the "good" downloads to sha1, delete the BESDownloadCacher\Downloads folder; it would still contain the "bad" downloads
    delete __appendfile
    appendfile @echo Checking that downloaded files match their expected sha1 values and copying them to sha1 cache
    if {exists folder "Downloads" whose (exists files of it) of folders "BESDownloadCacher" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) }
        appendfile {concatenation "%0d%0a" of ("copy /y %22" & it & "%22 %22" & pathname of folder "bfmirror\downloads\sha1" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client) & "%22") of pathnames of files whose (name of it = sha1 of it) of folder "BESDownloadCacher\Downloads" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}
    endif
    appendfile @echo Finished copying to sha1, clearing downloads folder
    appendfile @echo Files downloaded:
    appendfile dir "{pathname of folder "BESDownloadCacher\Downloads" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}"
    appendfile @echo Cleaning up downloaded files
    appendfile rd /s /q "{pathname of folder "BESDownloadCacher\Downloads" of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}"

    delete "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.cmd"
    move __appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.cmd"
    delete __appendfile

    appendfile "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.cmd" > "{pathname of folder (value of setting "_BESRelay_HTTPServer_ServerRootPath" of client)}\BESDownloadCacher\CopyToSha1.out" 2>&1
    move __appendfile CopySha1.cmd
    waithidden cmd /c CopySha1.cmd
endif
Success Criteria

This action will be considered successful when the applicability relevance evaluates to false.


Sharing

Social Media:
Share this page on Yammer

Comments

Log In or Register to leave comments!
jgstew -
Ideally this would be 3 separate actions with relevance to detect if it had run successfully within the past day for each step so that you could have better error handling and relevant vs non relevant feedback.
JasonWalker -
Latest version corrects an issue where the RunDownloads.cmd is not deleted before attempting to replace it.