devappserver / sdk 470+ / environment can only contain strings error FIX

[maclek]

I'm running python 311 app engine standard with devappserver on windows.

Upgraded to the latest sdk, upgrade to the latest to latest @NoCommandLine patch (you rock!)

Getting an error "environment can only contain strings" when launching waitress.

The issue for me is I have environment variables in my app.yaml. These are in the environment dict as bytes not a strings so whatever is loading app.yaml isn't converting things correctly. I spent some time in application_configuration.py but couldn't work it out.

app.yaml

env_variables:
  ENV_NAME: env_value

My solution was to fix the symptom into safe_process.py by adding this around line 83:

    if shell:
      args = ' '.join(args)

    # start local patch
    # the environment strings can contain bytes which Popen
    # doesn't handle. Convert them to strings here
    if env:
      env_keys = env.keys()
      for k in list(env.keys()):
        if isinstance(k, bytes):
          v = env[k]
          del env[k]
          env[k.decode()] = v.decode()
    # end local patch

    p = subprocess.Popen(args, env=env, cwd=cwd, stdout=stdout, stderr=stderr,
                         stdin=subprocess.PIPE, startupinfo=startupinfo,
                         shell=shell)

It's pretty gross but it works for me.

[NoCommandLine]

Hi,

Thanks for using the Patch and I'm happy you found it useful.

In instance_factory.py of version 427+, there's this line which handled that error.

I dropped it in version 470+ because I no longer encountered that error (that gcloud version dropped all links to Python 2) and others haven't run into it too. Can you tell me what version of the gcloud CLI you're running?

    ......NoCommandLine ......
https://nocommandline.com

        Analytics & GUI for 
App Engine & Datastore Emulator

[maclek]

Google Cloud SDK 475.0.0
app-engine-python 1.9.113
app-engine-python-extras 1.9.106
bq 2.1.4
cloud-datastore-emulator 2.3.1
core 2024.05.03
gcloud-crc32c 1.0.0
gsutil 5.27

Do you still have environment vars in your app.yaml? 😀

We also found that we can use vscode to debug this version of the sdk using the subprocess true launch setting. We need to call waitress using python directly as opposed to the exe. We also had to set shell to false in safe_subprocess.py around line 80.

    if args[0] == 'waitress-serve':
      args[0] = 'waitress'
      
    # see if this is the entrypoint call
    is_entrypoint = args[0] in ['waitress', 'gunicorn']

    if is_entrypoint:
      # if we're running the waitress process then run it through
      # python so the debugger understands this is a python subprocess
      # also add -Xfrozen_modules=off to make debugpy happy
      site_packages = site.getsitepackages()[-1]
      args[:1] = [sys.executable, '-Xfrozen_modules=off', f'{site_packages}/{args[0]}']
      # if we're running the waitress process don't use the shell
      # because debugpy won't see this as a python subprocess
      shell = False

    if shell:
      args = ' '.join(args)

[maclek]

Here's our launch.json

{
    // Use IntelliSense to learn about possible attributes.
    // Hover to view descriptions of existing attributes.
    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
    "version": "0.2.0",
    "configurations": [
        {
            "name": "GAE",
            "type": "debugpy",
            "request": "launch",
            "program": "${userHome}/AppData/Local/Google/Cloud SDK/google-cloud-sdk/bin/dev_appserver.py",
            "args": [
                "--python_virtualenv_path", "venv", "--storage_path", "dev_ds", "src"
            ],
            "cwd": "${workspaceFolder}/..",
            "subProcess": true,
            "django": true,
            "justMyCode": true
        }
    ]
}

[NoCommandLine]

---

@maclek wrote:

Do you still have environment vars in your app.yaml?

---

I didn't specifically check for env variables. Will work on a fix for that.

If you have a github account, can you raise an issue and include reproducible steps (for this and your other issues)? I want to see if I can reduce the number of files you're having to customize (looks you've now customized 4 files instead of the 2 in our patch) since the more files you customize, the more work you have to do when you upgrade your CLI

    ......NoCommandLine ......
https://nocommandline.com

        Analytics & GUI for 
App Engine & Datastore Emulator