[Bf-blender-cvs] [975ca919391] master: Steam Release: Script creation of Steam build files
Nathan Letwory
noreply at git.blender.org
Wed Dec 16 11:15:41 CET 2020
Commit: 975ca9193913c034f864121597e044d6eb043a5f
Author: Nathan Letwory
Date: Wed Dec 16 11:12:51 2020 +0100
Branches: master
https://developer.blender.org/rB975ca9193913c034f864121597e044d6eb043a5f
Steam Release: Script creation of Steam build files
Script tool for automation of Steam build files for tasks like {T77348}
This script automates creation of the Steam files: download of the archives,
extraction of the archives, preparation of the build scripts (VDF files), actual
building of the Steam game files.
Requirements
============
* MacOS machine - Tested on Catalina 10.15.6. Extracting contents from the DMG
archive did not work Windows nor on Linux using 7-zip. All DMG archives tested
failed to be extracted. As such only MacOS is known to work.
* Steam SDK downloaded from SteamWorks - The `steamcmd` is used to generate the
Steam game files. The path to the `steamcmd` is what is actually needed.
* SteamWorks credentials - Needed to log in using `steamcmd`.
* Login to SteamWorks with the `steamcmd` from the command-line at least once -
Needded to ensure the user is properly logged in. On a new machine the user
will have to go through two-factor authentication.
* App ID and Depot IDs - Needed to create the VDF files.
* Python 3.x - 3.7 was tested.
* Base URL - for downloading the archives.
Reviewed By: Jeroen Bakker
Differential Revision: https://developer.blender.org/D8429
===================================================================
A release/steam/README.md
A release/steam/blender_app_build.vdf.template
A release/steam/create_steam_builds.py
A release/steam/depot_build_linux.vdf.template
A release/steam/depot_build_macos.vdf.template
A release/steam/depot_build_win.vdf.template
===================================================================
diff --git a/release/steam/README.md b/release/steam/README.md
new file mode 100644
index 00000000000..05eda799c3f
--- /dev/null
+++ b/release/steam/README.md
@@ -0,0 +1,70 @@
+Creating Steam builds for Blender
+=================================
+
+This script automates creation of the Steam files: download of the archives,
+extraction of the archives, preparation of the build scripts (VDF files), actual
+building of the Steam game files.
+
+Requirements
+============
+
+* MacOS machine - Tested on Catalina 10.15.6. Extracting contents from the DMG
+ archive did not work Windows nor on Linux using 7-zip. All DMG archives tested
+ failed to be extracted. As such only MacOS is known to work.
+* Steam SDK downloaded from SteamWorks - The `steamcmd` is used to generate the
+ Steam game files. The path to the `steamcmd` is what is actually needed.
+* SteamWorks credentials - Needed to log in using `steamcmd`.
+* Login to SteamWorks with the `steamcmd` from the command-line at least once -
+ Needded to ensure the user is properly logged in. On a new machine the user
+ will have to go through two-factor authentication.
+* App ID and Depot IDs - Needed to create the VDF files.
+* Python 3.x - 3.7 was tested.
+* Base URL - for downloading the archives.
+
+Usage
+=====
+
+```bash
+$ export STEAMUSER=SteamUserName
+$ export STEAMPW=SteamUserPW
+$ export BASEURL=https://download.blender.org/release/Blender2.83/
+$ export VERSION=2.83.3
+$ export APPID=appidnr
+$ export WINID=winidnr
+$ export LINID=linuxidnr
+$ export MACOSID=macosidnr
+
+# log in to SteamWorks from command-line at least once
+
+$ ../sdk/tools/ContentBuilder/builder_osx/steamcmd +login $STEAMUSER $STEAMPW
+
+# once that has been done we can now actually start our tool
+
+$ python3.7 create_steam_builds.py --baseurl $BASEURL --version $VERSION --appid $APPID --winid $WINID --linuxid $LINID --macosid $MACOSID --steamuser $STEAMUSER --steampw $STEAMPW --steamcmd ../sdk/tools/ContentBuilder/builder_osx/steamcmd
+```
+
+All arguments in the above example are required.
+
+At the start the tool will login using `steamcmd`. This is necessary to let the
+Steam SDK update itself if necessary.
+
+There are a few optional arguments:
+
+* `--dryrun`: If set building the game files will not actually happen. A set of
+ log files and a preview manifest per depot will be created in the output folder.
+ This can be used to double-check everything works as expected.
+* `--skipdl`: If set will skip downloading of the archives. The tool expects the
+ archives to already exist in the correct content location.
+* `--skipextract`: If set will skip extraction of all archives. The tool expects
+ the archives to already have been correctly extracted in the content location.
+
+Run the tool with `-h` for detailed information on each argument.
+
+The content and output folders are generated through appending the version
+without dots to the words `content` and `output` respectively, e.g. `content2833`
+and `output2833`. These folders are created next to the tool.
+
+From all `.template` files the Steam build scripts will be generated also in the
+same directory as the tool. The files will have the extension `.vdf`.
+
+In case of errors the tool will have a non-zero return code.
\ No newline at end of file
diff --git a/release/steam/blender_app_build.vdf.template b/release/steam/blender_app_build.vdf.template
new file mode 100644
index 00000000000..9e2d0625d72
--- /dev/null
+++ b/release/steam/blender_app_build.vdf.template
@@ -0,0 +1,17 @@
+"appbuild"
+{
+ "appid" "[APPID]"
+ "desc" "Blender [VERSION]" // description for this build
+ "buildoutput" "./[OUTPUT]" // build output folder for .log, .csm & .csd files, relative to location of this file
+ "contentroot" "./[CONTENT]" // root content folder, relative to location of this file
+ "setlive" "" // branch to set live after successful build, non if empty
+ "preview" "[DRYRUN]" // 1 to enable preview builds, 0 to commit build to steampipe
+ "local" "" // set to flie path of local content server
+
+ "depots"
+ {
+ "[WINID]" "depot_build_win.vdf"
+ "[LINUXID]" "depot_build_linux.vdf"
+ "[MACOSID]" "depot_build_macos.vdf"
+ }
+}
diff --git a/release/steam/create_steam_builds.py b/release/steam/create_steam_builds.py
new file mode 100644
index 00000000000..2ecd0c347f7
--- /dev/null
+++ b/release/steam/create_steam_builds.py
@@ -0,0 +1,397 @@
+#!/usr/bin/env python3
+
+import argparse
+import pathlib
+import requests
+import shutil
+import subprocess
+from typing import Callable, Iterator, List, Tuple
+
+# supported archive and platform endings, used to create actual archive names
+archive_endings = ["windows64.zip", "linux64.tar.xz", "macOS.dmg"]
+
+
+def add_optional_argument(option: str, help: str) -> None:
+ global parser
+ """Add an optional argument
+
+ Args:
+ option (str): Option to add
+ help (str): Help description for the argument
+ """
+ parser.add_argument(option, help=help, action='store_const', const=1)
+
+
+def blender_archives(version: str) -> Iterator[str]:
+ """Generator for Blender archives for version.
+
+ Yields for items in archive_endings an archive name in the form of
+ blender-{version}-{ending}.
+
+ Args:
+ version (str): Version string of the form 2.83.2
+
+
+ Yields:
+ Iterator[str]: Name in the form of blender-{version}-{ending}
+ """
+ global archive_endings
+
+ for ending in archive_endings:
+ yield f"blender-{version}-{ending}"
+
+
+def get_archive_type(archive_type: str, version: str) -> str:
+ """Return the archive of given type and version.
+
+ Args:
+ archive_type (str): extension for archive type to check for
+ version (str): Version string in the form 2.83.2
+
+ Raises:
+ Exception: Execption when archive type isn't found
+
+ Returns:
+ str: archive name for given type
+ """
+
+ for archive in blender_archives(version):
+ if archive.endswith(archive_type):
+ return archive
+ raise Exception("Unknown archive type")
+
+
+def execute_command(cmd: List[str], name: str, errcode: int, cwd=".", capture_output=True) -> str:
+ """Execute the given command.
+
+ Returns the process stdout upon success if any.
+
+ On error print message the command with name that has failed. Print stdout
+ and stderr of the process if any, and then exit with given error code.
+
+ Args:
+ cmd (List[str]): Command in list format, each argument as their own item
+ name (str): Name of command to use when printing to command-line
+ errcode (int): Error code to use in case of exit()
+ cwd (str, optional): Folder to use as current work directory for command
+ execution. Defaults to ".".
+ capture_output (bool, optional): Whether to capture command output or not.
+ Defaults to True.
+
+ Returns:
+ str: stdout if any, or empty string
+ """
+ cmd_process = subprocess.run(
+ cmd, capture_output=capture_output, encoding="UTF-8", cwd=cwd)
+ if cmd_process.returncode == 0:
+ if cmd_process.stdout:
+ return cmd_process.stdout
+ else:
+ return ""
+ else:
+ print(f"ERROR: {name} failed.")
+ if cmd_process.stdout:
+ print(cmd_process.stdout)
+ if cmd_process.stderr:
+ print(cmd_process.stderr)
+ exit(errcode)
+ return ""
+
+
+def download_archives(base_url: str, archives: Callable[[str], Iterator[str]], version: str, dst_dir: pathlib.Path):
+ """Download archives from the given base_url.
+
+ Archives is a generator for Blender archive names based on version.
+
+ Archive names are appended to the base_url to load from, and appended to
+ dst_dir to save to.
+
+ Args:
+ base_url (str): Base URL to load archives from
+ archives (Callable[[str], Iterator[str]]): Generator for Blender archive
+ names based on version
+ version (str): Version string in the form of 2.83.2
+ dst_dir (pathlib.Path): Download destination
+ """
+
+ if base_url[-1] != '/':
+ base_url = base_url + '/'
+
+ for archive in archives(version):
+ download_url = f"{base_url}{archive}"
+ target_file = dst_dir.joinpath(archive)
+ download_file(download_url, target_file)
+
+
+def download_file(from_url: str, to_file: pathlib.Path) -> None:
+ """Download from_url as to_file.
+
+ Actual downloading will be skipped if --skipdl is given on the command-line.
+
+ Args:
+ from_url (str): Full URL to resource to download
+ to_file (pathlib.Path): Full path to save downloaded resource as
+ """
+ global args
+
+ if not args.skipdl or not to_file.exists():
+ print(f"Downloading {from_url}")
+ with open(to_file, "wb") as download_zip:
+ response = requests.get(from_url)
+ if response.status_code != requests.codes.ok:
+ print(f"ERROR: failed to download {from_url} (status code: {response.status_code})")
+ exit(1313)
+ download_zip.write(response.content)
+ else:
+ print(f"Downloading {from_url} skipped")
+ print(" ... OK")
+
+
+def copy_contents_from_dmg_to_path(dmg_file: pathlib.Path, dst: pathlib.Path) -> None:
+ """Copy the contents of the given DMG file to the destination folder.
+
+ Args:
+ dmg_file (pathlib.Path): Full path to DMG archive to extract from
+ dst (pathlib.Path): Full path to destination to extract to
+ """
+ hdiutil_attach = ["hdiutil",
+ "attach",
+ "-readonly",
+ f"{dmg_file}"
+ ]
+ attached = execute_command(hdiutil_attach, "hdiutil attach", 1)
+
+ # Last line of output is what we want, it is of the form
+ # /dev/somedisk Apple_HFS /Volumes/Blender
+ # We want to retain the mount point, and the folder the mount is
+ # created on. The mounted disk we need for detaching, the folder we
+ # need to be able to copy the contents to
@@ Diff output truncated at 10240 characters. @@
More information about the Bf-blender-cvs
mailing list