service-to-service with forbidden status returned

Hi everyone,

I have an issue that I have seen posted before but I cannot seem to get figured out.

I have 2 cloud run services that I would like to be able to make requests from A to B. B is supposed to be internal only. Both need access to SQL. If I have B set to Ingress Control All everything works fine.

Cloud Run settings.

A.

  • Networking - Ingress Control: All
  • Security – Authentication: Allow unauthenticated invocations.

B.

  • Networking – Ingress Control: Internal
  • Security – Authentication: Require Authentication

On the Cloud Run services page for service B which is the internal one I added the service account that service A is using as a Cloud Run Invoker.

I have VPC network setup and both are using the same one. I have Private Google Access turned on.

They are both set to the same region.

I tried making some changes to the VPC firewall but that didn’t seem to make a difference. At the moment I just have all the filters disabled.

For IAM setup I tried originally having only one principal but then made 2. Though they both have the same roles.

  • Cloud Logging Service Agent
  • Cloud Run Invoker
  • Cloud Run Service Agent
  • Cloud SQL Service Agent
  • Serverless VPC Access Service Agent

Service A code was made with NextJS and Service B code is Python. The URL format is “https://my-cloud-run-service.run.app” without a “/” at the end. But I also tried with it.

Here is the code I use on NextJS to make the call to Python.

const serviceAPI = async ({ url, options, targetAudience }: IServiceApi) => {
  console.log('serviceAPI options', options);
  console.log('serviceAPI targetAudience1', targetAudience);
  console.log('serviceAPI isDev', isDev);
  let res = null;
  if (!options) {
    options = {};
  }
  try {
    if (isDev === false) {
      if (!targetAudience) {
        targetAudience = new URL(url).origin;
        // targetAudience = new URL(url).origin + '/';
        // targetAudience = url;
      }
      console.log('serviceAPI targetAudience2', targetAudience);
      if (targetAudience) {
        try {
          const auth = new GoogleAuth();
          const client = await auth.getIdTokenClient(targetAudience);
          if (client) {
            const clientHeaders = await client.getRequestHeaders();
            console.log('serviceAPI clientHeaders', clientHeaders);
            if (!options.headers) {
              options.headers = {};
            }
            if (!options.method) {
              options['method'] = 'GET';
            }
            if (!options.timeout) {
              options['timeout'] = 3000;
            }
            if (!options.headers['Content-Type']) {
              options.headers['Content-Type'] = 'application/json';
            }
            options.headers['Authorization'] = clientHeaders['Authorization'];
          }
        } catch (err: any) {
          throw Error('could not create an identity token: ' + err.message);
        }
      }
    }
    console.log('serviceAPI options', options);
    try {
      res = await fetch(url, options);
      console.log('serviceAPI res', res);
      return await res.json();
    } catch (err: any) {
      throw Error('failed to fetch data: ' + err.message);
    }
  } catch (err: any) {
    console.log('serviceAPI Error:', err);
  }
};

Here are some of the console.log entries that I thought might be relevant.

serviceAPI targetAudience2 https://....run.app

serviceAPI options

{
	headers: {
		'Content-Type': 'application/json',
		Authorization: 'Bearer eyJh....-xWKQ'
	},
	method: 'GET',
	timeout: 3000
}

serviceAPI res Response

{
	aborted: false,
	rangeRequested: false,
	timingAllowPassed: true,
	requestIncludesCredentials: true,
	type: 'default',
	status: 403,
	timingInfo: {
		startTime: 98527.608099,
		redirectStartTime: 0,
		redirectEndTime: 0,
		postRedirectStartTime: 98527.608099,
		finalServiceWorkerStartTime: 0,
		finalNetworkResponseStartTime: 0,
		finalNetworkRequestStartTime: 0,
		endTime: 0,
		encodedBodySize: 235,
		decodedBodySize: 235,
		finalConnectionTimingInfo: null
},
	cacheState: '',
	statusText: 'Forbidden',
	headersList: HeadersList {
		[Symbol(headers map)]: [Map],
		[Symbol(headers map sorted)]: null
	},
	urlList: [ [URL] ],
	body: { stream: undefined }
},

I’m just working on a portfolio website and wanted to try having Python as some kind of microservice. Though this is not the optimal way to do it this is more for experimentation. While trying to figure this out I have flipped a lot of switches back and forth and probably have settings and other things that aren’t actually needed. I’ll take any criticism given.

If any other information is needed that I forgot please let me know and I will try to provide it.

Just trying to figure out what’s going on.

Any help is much appreciated.

Thanks

0 3 159
3 REPLIES 3

Here is a Stackoverflow article that seems to be close to your story ...

https://stackoverflow.com/questions/66341476/google-cloud-run-communication-between-internal-service...

I notice that they are using a different technique to obtain the authentication token.

Thanks for the link. I checked it out and the suggestion it had was to make sure I have a serverless VPC connector setup. I did already have one setup. I tried changing the VPC setting from “Route all traffic through the VPC connector” to “Route only requests to private Ips through the VPC connector”. I also changed the service B Cloud Run Invoker to include allUsers just to see if it would work. Unfortunately, it has not. I am still getting the statusText Forbidden in the logs on service A.

I see in the Stackoverflow question the person had Header: {"Metadata-Flavor", "Google"}. I haven’t seen that before. Is it something I missed?

I have tried looking in the VPC logs as well to see if there was some error or warning or something mentioning the request but have not been able to find anything.

Thanks for the suggestion though.

I figured out what the issue was. Definitely my mistake. In my code I had

if (!options.headers['Content-Type']) {
	options.headers['Content-Type'] = 'application/json';
}

Which was a mistake. Removing that fixed the problem.

Thanks again for the help